Gate a consumer pipeline on upstream data readiness instead of a clock
A consumer build runs at a fixed time and, on runs where an upstream load is late, builds on partial data without anyone noticing.
- Orchestration
- Any domain
Problem area
5 RecipesOrchestration and reliability
Recipes for scheduling, gating and monitoring pipelines so runs start when their inputs are ready and failures are visible.
Convert a scheduled data job to another platform without changing its required behavior
A scheduled load-and-build job has to move to another platform, and nobody has written down what it must keep doing.
- Orchestration
- Any domain
Author a Fabric Data Pipeline that sequences ingestion, dbt, and downstream refresh as committed code
The nightly loads, the dbt build and the report refresh are run by hand or live only as a pipeline clicked together in the workspace.
- Orchestration
- Any domain
Schedule a dlt load and a dbt build on MotherDuck as a committed MotherDuck Flight
The MotherDuck load and the dbt build that follows it run from a cron job on someone's laptop.
- Orchestration
- Any domain
Declare source freshness SLAs and make stale data fail before downstream work
A load stopped and nothing failed; downstream models rebuilt on old data and the reports still looked current.
- Data quality
- Any domain