01 · The challenge
One business question. Many disconnected signals.
Store teams needed a reliable way to see whether products were actually on shelf, and whether shelf conditions matched the planned retail execution. The signals came from very different places: a cloud retail data platform, on-prem inventory snapshots, a data warehouse, planogram files, and product images.
Each feed had its own source and schedule (daily store inventory, weekly store-to-planogram ties, item end dates, cover images, and planogram PSA files), and some steps were still manual. On top of that, the processed data had to reach a second Databricks workspace in a different Azure region without making another copy.
02 · What I did
What I owned and built.
- Designed and documented the integration architecture across all five feeds, from source extraction through the vendor hand-off and the metric return path.
- Investigated Delta Sharing between workspaces in two Azure regions. I found that the metastore region mismatch blocks native sharing, tested an open-sharing workaround, and recommended a shared ADLS path pointing at the post-merge Silver/Gold tables, with the egress-cost and migration trade-offs documented.
- Optimized historical on-hand extraction queries and caught an unexpected 136 million rows that needed investigating before extraction.
- Set up a branch-based change process in the ADF development environment.
- Worked with a retail partner on logging store exceptions, which gave the business reason codes for gaps between physical in-stock % and on-shelf availability %.
03 · Architecture
How it fits together.
Inventory · Planograms · Product · Images
Blob / ADLS
Bronze → Silver → Gold
04 · Key decisions
The judgment behind the build.
Why incremental processing?
Daily retail volumes are big enough that repeated full reloads waste compute, run longer, and widen the blast radius when something fails. Incremental patterns keep each run focused on new or changed business data.
Why separate business-effective date from system date?
Files arrive late. A pipeline keyed only to today’s system date can attach data to the wrong business period. Deriving the effective date from the source file keeps the business meaning intact when delivery timing slips.
Why isolate retailer-specific logic?
Retailers differ in structure, cadence, identifiers, and exception rules. Keeping those differences at the edges stops the reusable core from turning into a pile of hard-coded branches.
Why a shared ADLS path instead of a copy?
Native sharing was blocked by the region mismatch. Pointing the second workspace at the post-merge tables avoids duplicate storage and means consumers only ever read clean, processed data.
05 · Impact
What changed because of the work.
- Gives major retail partners a consistent, repeatable data feed into the shelf-intelligence platform.
- The sharing recommendation avoids duplicate storage, and the second workspace reads only processed data.
- Reason-code logging gives the business a way to explain store-level performance gaps.
06 · What’s next
Where it goes from here.
Automate the cover-image feed (currently desktop RPA) and implement the shared ADLS approach. At 10× scale I would move the highest-frequency feeds to event-driven ingestion, add data contracts with automated schema checks, onboard new retailers through metadata and configuration, and put freshness and quality metrics into a single operational dashboard with alerting.
This case study is a sanitized reconstruction. Organization names, proprietary code, data, and internal schemas are intentionally left out.