Convert an ADF mapping data flow into dlt and dbt and prove parity
Order or customer reporting still comes from an Azure Data Factory mapping data flow whose logic lives only on the ADF canvas, with no tests or review.
- Transformation
- Any domain
Domain
52 RecipesAny domain
Recipes for data engineers whose outcome no single business function owns, such as ingestion, dbt engineering, orchestration, identity and migration.
Re-engineer something you own 13
Backfill a bounded date range after a logic change without a full refresh
A business rule in an incremental model was corrected, and only a known date range of published history should be restated.
- Transformation
- Any domain
Convert a full-refresh dbt model to incremental and show the runtime delta
A dbt model is correct but a full refresh is too slow or too expensive to run routinely, and gets slower as history grows.
- Transformation
- Any domain
Refactor dbt models without changing consumer output
A model has grown into one long query with logic packed inline, and nobody dares change it because dashboards and reports depend on its output.
- Transformation
- Any domain
Convert a legacy SQL script or view into a dbt model and prove parity
A report or table is still produced by a hand-run SQL script or a database view outside dbt, with no tests, lineage or review.
- Transformation
- Any domain
Replace a Fivetran or Airbyte connector with a dlt pipeline and reconcile the result
A managed connector's bill changed shape, or it broke and the vendor fix has been pending for weeks, and the team wants the load in code it owns.
- Ingestion
- Any domain
Convert a materialized view into a dbt model and preserve its required refresh semantics
Warehouse logic lives in a materialized view that nobody can test, review or trace, and the team wants it in the dbt project.
- Transformation
- Any domain
Convert a notebook transformation into a dbt model and prove parity
A table or report is still produced by running a notebook by hand, pulling warehouse data into dataframes, with no tests, lineage or review.
- Transformation
- Any domain
Convert a scalar SQL function into a dbt macro and prove equivalence
Models or views call a scalar function created directly in the warehouse, so dbt cannot see the dependency and changes to it are not reviewed or tested.
- Transformation
- Any domain
Convert a hand-written history routine into a dbt snapshot and prove history parity
A Type 2 dimension is kept by a hand-written MERGE procedure or trigger that nobody wants to maintain.
- Transformation
- Any domain
Convert a Spark ingestion notebook into a dlt pipeline and reconcile what it lands
A Spark notebook lands raw files into bronze, and the team wants that load as tested pipeline code without keeping a cluster for it.
- Ingestion
- Any domain
Convert a stored procedure into a dbt model and prove parity
A nightly load is a stored procedure nobody fully understands, and it inserts, updates, deletes or merges rows in a table people depend on.
- Transformation
- Any domain
Add a breaking data contract change as a new version beside the old one and revise its named consumers
A required change to a published relation's grain, column meaning or shape cannot keep the old interface.
- Transformation
- Any domain
Build something that is not there yet 33
Land a SaaS or REST API into bronze with an incremental cursor and a schema contract
An API source has to load into bronze incrementally instead of reloading everything on every run.
- Ingestion
- Any domain
Build AML monitoring features at account-day grain
A transaction-monitoring model is fed hand-built features nobody can reproduce, so two runs on the same data disagree.
- Transformation
- Any domain
Build current-state tables from exported database change events
A batch export of database change events is available and the current state of its tables has to be rebuilt without reading the source database again.
- Ingestion
- Any domain
Preserve business-effective history and what was known at each reporting cutoff
A backdated correction must change today's reading of the past, yet a report issued earlier has to stay reproducible as it was published.
- Transformation
- Any domain
Build a business-event fact with auditable duplicate handling
Transaction feeds overlap, and source deliveries may be counted as separate business events.
- Transformation
- Any domain
Unify consent and communication preferences with auditable latest-wins logic
A customer who opted out in one system still gets messages from another, because each source says something different and whichever job ran last wins.
- Governance and cost
- Any domain
Build retention cohorts and a churn feature table
Marketing, finance and product each quote a different churn rate because each defines an active and a lost customer differently.
- Transformation
- Any domain
Score customers with RFM, maintain segments incrementally and explain every move
Marketing segment lists come from a spreadsheet or BI export nobody can reproduce, and the same customer lands in different segments depending on who pulled the list.
- Transformation
- Any domain
Load a database table set into bronze with a bounded incremental strategy and landed-data proof
An operational database's tables have to land in bronze, and reloading every table in full on each run is too slow or too heavy on the source.
- Ingestion
- Any domain
Add dimension history with verified historical fact classification
Customer, asset or vendor attributes change, but past events need a defensible historical classification.
- Transformation
- Any domain
Build historical event amounts from effective-dated business rules
Historical fees or surcharges change when a current rate is applied to past events.
- Transformation
- Any domain
Build auditable event-to-resource attribution
Revenue or productivity must be reported by employee, asset or team, but responsibility for each event is unclear.
- Transformation
- Any domain
Author a Fabric Data Pipeline that sequences ingestion, dbt, and downstream refresh as committed code
The nightly loads, the dbt build and the report refresh are run by hand or live only as a pipeline clicked together in the workspace.
- Orchestration
- Any domain
Author a Direct Lake semantic model as code over governed gold tables
The gold marts exist and the Power BI team is about to rebuild the star schema by hand in every report.
- Semantic layer
- Any domain
Load a recurring file drop into bronze with file-completeness and schema controls
A partner or vendor drops files into a folder on a schedule, and bronze has to pick up each delivery without a rerun loading it twice.
- Ingestion
- Any domain
Promote a spreadsheet or CSV to a governed, tested table
A spreadsheet that reports and models depend on is edited by hand, and a moved, renamed or retyped column keeps breaking them quietly.
- Ingestion
- Any domain
Build a thirty-day readmission cohort with explicit exclusions
The quality team rebuilds a readmission rate from a spreadsheet each quarter, and nobody can say who was left out of the rate or why.
- Transformation
- Any domain
Build nightly occupancy, ADR and RevPAR by property with closed-night restatements
Occupancy, ADR and RevPAR differ between reports because each one counts stays, comps, out-of-order rooms and cancellations its own way.
- Transformation
- Any domain
Deduplicate customers into a golden record with a source crosswalk
The same customer exists in several systems under different ids, so customer counts and per-customer revenue are inflated.
- Transformation
- Any domain
Standardize names and addresses and mint deterministic match keys
The same customer is written differently across CRM, billing and support, so joins on name or address miss.
- Transformation
- Any domain
Author an incremental dbt model with its contract and prove idempotency
A new dbt model must be brought up to date incrementally on each run, and nobody can show that a rerun or a re-delivery of records already applied leaves it unchanged.
- Transformation
- Any domain
Build clinical-trial enrollment and site-performance models with every screened subject accounted for
Clinical operations rebuilds enrollment and site performance in a spreadsheet each month, and nobody can say who was left out of the enrolled count or why a site looks slow.
- Transformation
- Any domain
Build a metric with explicit eligibility and reconcilable rollups
A new percentage, rate or average needs a defensible eligible population and missing-value policy.
- Transformation
- Any domain
Define governed metrics as a MetricFlow semantic model over a gold mart
Revenue or average order value means different things in different marts and dashboards, and nobody can say which number is right.
- Semantic layer
- Any domain
Schedule a dlt load and a dbt build on MotherDuck as a committed MotherDuck Flight
The MotherDuck load and the dbt build that follows it run from a cron job on someone's laptop.
- Orchestration
- Any domain
Build state durations by entity, location and reporting period
Waiting, idle, available or busy time must be measured by site or zone and hour from operational event histories.
- Transformation
- Any domain
Tag personal-data columns, mask them by kind for a lower-privilege audience and prove the masked outputs
Analysts need tables that carry names, addresses, emails, phone numbers or ID numbers they must not see.
- Governance and cost
- Any domain
Build a training dataset that uses only information available at prediction time
A model scores well in training but worse in production because training rows used facts, corrections or labels a real prediction could not have seen.
- Transformation
- Any domain
Sessionize event streams and build reproducible funnels from batch exports
Product analytics numbers cannot be reproduced outside the vendor's screen, and nobody can say why two conversion figures differ.
- Transformation
- Any domain
Gate a consumer pipeline on upstream data readiness instead of a clock
A consumer build runs at a fixed time and, on runs where an upstream load is late, builds on partial data without anyone noticing.
- Orchestration
- Any domain
Build one shared calendar and fiscal-period date table
Two reports put the same day in different weeks, months, quarters or fiscal periods because each mart builds its own date logic.
- Transformation
- Any domain
Declare source freshness SLAs and make stale data fail before downstream work
A load stopped and nothing failed; downstream models rebuilt on old data and the reports still looked current.
- Data quality
- Any domain
Land call-detail batches and aggregate them into subscriber-day usage with manifest completeness
Monthly subscriber usage differs by who ran it, because nobody can say which call-detail batches had landed or whether each arrived complete.
- Ingestion
- Any domain
Prove a change is safe before you ship 1
Assess an existing dbt change for safe merge without modifying it
An existing dbt change builds successfully but reviewers still need evidence that its business answers remain valid.
- Data quality
- Any domain
Work across more than one platform 5
Assess a Synapse dedicated pool or SSIS estate for a Fabric rebuild and recommend whether to go
A Synapse dedicated SQL pool or a set of SSIS packages is due to move to Fabric, and nobody can say what is in it, what each piece would take or what stops it.
- Re-engineering
- Any domain
Port a dbt macro to each approved warehouse without changing its callers
Part of a dbt project is moving to another warehouse, and a macro its models call only compiles, or only returns correct values, on the old one.
- Transformation
- Any domain
Reconcile equivalent data across two live systems before cutover
A table or model has been rebuilt on another platform or database while the original is still running, and the owner needs evidence the two hold the same data before cutting over.
- Data quality
- Any domain
Convert a scheduled data job to another platform without changing its required behavior
A scheduled load-and-build job has to move to another platform, and nobody has written down what it must keep doing.
- Orchestration
- Any domain
Move a dbt project to a new destination and check parity before cutover
A dbt project has to run on a different warehouse or lakehouse, and the dashboards and reports that read it cannot show a different number afterwards.
- Transformation
- Any domain