Convert an ADF mapping data flow into dlt and dbt and prove parity
Order or customer reporting still comes from an Azure Data Factory mapping data flow whose logic lives only on the ADF canvas, with no tests or review.
- Transformation
Problem area
10 RecipesIngest and reconcile source data
Recipes for landing source data in owned pipelines and governed tables, with explicit record disposition, safe reruns and reconciliation.
Land a SaaS or REST API into bronze with an incremental cursor and a schema contract
An API source has to load into bronze incrementally instead of reloading everything on every run.
- Ingestion
Build current-state tables from exported database change events
A batch export of database change events is available and the current state of its tables has to be rebuilt without reading the source database again.
- Ingestion
Reconcile equivalent data across two live systems before cutover
A table or model has been rebuilt on another platform or database while the original is still running, and the owner needs evidence the two hold the same data before cutting over.
- Data quality
Load a database table set into bronze with a bounded incremental strategy and landed-data proof
An operational database's tables have to land in bronze, and reloading every table in full on each run is too slow or too heavy on the source.
- Ingestion
Load a recurring file drop into bronze with file-completeness and schema controls
A partner or vendor drops files into a folder on a schedule, and bronze has to pick up each delivery without a rerun loading it twice.
- Ingestion
Promote a spreadsheet or CSV to a governed, tested table
A spreadsheet that reports and models depend on is edited by hand, and a moved, renamed or retyped column keeps breaking them quietly.
- Ingestion
Replace a Fivetran or Airbyte connector with a dlt pipeline and reconcile the result
A managed connector's bill changed shape, or it broke and the vendor fix has been pending for weeks, and the team wants the load in code it owns.
- Ingestion
Convert a Spark ingestion notebook into a dlt pipeline and reconcile what it lands
A Spark notebook lands raw files into bronze, and the team wants that load as tested pipeline code without keeping a cluster for it.
- Ingestion
Land call-detail batches and aggregate them into subscriber-day usage with manifest completeness
Monthly subscriber usage differs by who ran it, because nobody can say which call-detail batches had landed or whether each arrived complete.
- Ingestion
- Telecom