01 · The challenge
Five emails. One file. Don’t load it five times.
Triggering the pipeline for every notification risked repeated loads, duplicate processing, and no clear answer to “is today’s data in?”
02 · What I did
What I owned and built.
- Parsed incoming report messages to pull out the business activity date and retailer identifier.
- Checked a date-specific blob control marker before triggering downstream ingestion.
- Added create-if-absent control-file behavior so duplicate or repeated notifications are handled safely.
- Integrated the flow with an Azure Data Factory pipeline and investigated polling behavior during testing.
03 · Architecture
How it fits together.
Create-if-absent per activity date
Only once per date
04 · Key decisions
The judgment behind the build.
Why a control marker?
It turns “did we already load this?” into a cheap, explicit check, and it doubles as a record of what was processed.
Why key on the business activity date?
Notifications arrive whenever they arrive. The activity date is what the data is actually about.
Why create-if-absent?
Two notifications landing at once shouldn’t both win. An atomic create gives one clear owner of the run.
05 · Impact
What changed because of the work.
- Adds an explicit processing-state check before ingestion launches.
- Makes duplicate avoidance and activity-date tracking visible in the integration flow.
06 · What’s next
Where it goes from here.
Validate concurrency, failed-run retry behavior, and marker timing before broader rollout.
This case study is a sanitized reconstruction. Organization names, proprietary code, data, and internal schemas are intentionally left out.