Data EngineeringOpenAdvanced

Make a flaky pipeline idempotent

The source sends yesterday's file twice, once with a new column. Build a pipeline that survives both without a human.

Time needed: 2 weeks
Scenarios: 6 failure modes

The brief

You inherit a working-on-Tuesday pipeline and a source system that misbehaves on purpose. Duplicate deliveries, a silently added column, and one late-arriving partition. The pipeline must be re-runnable at any point without corrupting the warehouse, and it must fail loudly when it should.

What you submit

An orchestrated pipeline that is safe to re-run at any point
Data-quality checks that fail the run on contract violations
A short runbook a stranger could operate at 9am

How to participate

Register free: create your account and tell us a little about yourself.
Clone the starter repo with the misbehaving source simulator.
Build, test the replay scenarios, and write the runbook.
Submit the repo; mentors run every failure scenario against it.

How it’s judged

Idempotency under replay
Quality checks that actually fire
Runbook clarity

What you walk away with

Feedback for every entry · Top 3 win a full flagship seat

Resources

Everything you need, in one zipThe dataset, the full brief and the guides for this challenge.Register free to unlock

Want the full track behind this brief?

Data Engineering — projects across the modern stack.

Explore Data Engineering →