01 · The challenge
People needed answers without waiting on an analyst.
Business users relied on legacy static reports, and anything custom meant waiting for an analyst. The business needed a modern self-service tool that a large user base could run many times a day across billing, units, cost, MSRP, POS, return rates, and sell-through on a retail week-ending calendar.
There was one non-negotiable: the numbers had to match the BI reports leadership already trusted.
02 · What I did
What I owned and built.
- Designed the platform end to end: ADF extraction of on-prem data, Blob landing, incremental Databricks ingestion, Silver/Gold modeling, and job orchestration.
- Built the multi-page app with backend Databricks job integration, run history, and a scheduled cleanup job that purges old tracking records and export files.
- Built Excel exports whose formatting is driven by column name (percent, currency, units), and fixed a concurrency bug by giving each run its own staging path.
- Shipped the features users asked for: cascading agency-to-channel filters, an optional week-by-week pivot written in native Spark, a selection summary tab, warehouse-club store locations, an on-sale date filter, and store-level returns with chain cutover logic.
- Led chain-level validation against the BI reports. I fixed a join that was dropping rows after an account correction and traced POS unit variances to specific chains.
- Set up CI/CD with Asset Bundles in Azure DevOps, including a post-deploy CLI step to work around unsupported OAuth scopes.
03 · Architecture
How it fits together.
Bronze → Silver
Run history · cleanup
Filters · pivots · summaries
04 · Key decisions
The judgment behind the build.
Why run queries as backend jobs?
Heavy Spark work runs on job compute, not in the app process. That keeps the UI responsive for many concurrent users and gives every run an auditable history.
Why a staging path per run?
Concurrent users were colliding on shared export locations. Isolating each run’s staging path removed the race entirely, which beat trying to coordinate it.
Why reconcile to BI before rollout?
If the app disagrees with the reports leadership uses, nobody adopts it. Chain-level reconciliation turned “is this right?” into “this matches.”
Why column-name-driven formatting?
New fields get the right percent, currency, or unit formatting automatically, so adding a metric doesn’t mean touching export code.
05 · Impact
What changed because of the work.
- Users run their own reporting instead of waiting on an analyst.
- Reconciliation to BI means people can trust that the app’s numbers match leadership’s reports.
- Automated, repeatable deployments make releases safer and faster.
06 · What’s next
Where it goes from here.
UI polish and branding, job-status polling, and a metadata-driven attribute catalog that can support hundreds of optional fields. I’m also extending the platform with ERP sales-order-line data.
This case study is a sanitized reconstruction. Organization names, proprietary code, data, and internal schemas are intentionally left out.