Build something that is not there yet
Land a SaaS or REST API into bronze with an incremental cursor and a schema contract
- An API source has to load into bronze incrementally instead of reloading everything on every run.
- A pipeline breaks whenever the source API adds a field or changes a field's type.
- The team has to know which of an API's records landed in bronze, which were refused and why, and which it can show were missed.
Build a dlt pipeline that lands a SaaS or REST API into bronze incrementally under an approved cursor, enforces an approved schema contract for new fields and changed types, keeps refused records visible, and accounts for each load's records by outcome and by extraction, completely where the API offers an independent, enumerable view of what it holds.
Use this Recipe
Use the Vibedata Recipe `api-to-bronze-incremental-contract` at https://github.com/accelerate-data/vibedata-cookbooks/blob/main/recipes/api-to-bronze-incremental-contract/recipe.md Read the Recipe and execute it in the context of the current Intent.
Recipe id api-to-bronze-incremental-contract · Canonical definition
Verified by
What has to be observably true before this Recipe is finished.
- Every approved resource lands in bronze under its approved write disposition and, where it has one, business key, and each landed row carries the approved source identifiers and provenance.
- From the approved start, each load lands only the records the approved cursor rule selects, including any approved lookback and handling of records with no cursor, and a rerun over an unchanged source lands or changes nothing beyond what the approved re-send and refusal rules do with re-reads.
- A record that shares a cursor value with the stored mark or another record, or writes its cursor with a different offset or format, is selected, skipped or re-read exactly as the approved cursor rule and lookback say.
- Late, re-sent and corrected records are handled as the approved rules say, and a record the cursor rule cannot reach is reported as missed wherever an independent view of the source shows it.
- Deletes reach bronze as the approved policy says, whether flagged, removed or recorded separately, and the result states which deletes the load cannot see.
- A new field and a changed field type, including on a field the API added after the first load, are handled as the approved contract says, whether the field is added or dropped, the record is refused, the value is kept apart or the load fails.
- Every refused record stays visible where the approved rule puts it, with its reason and approved identifying detail; an unchanged re-read keeps its refused outcome, is marked re-read and is not counted as a second refused record; whether the cursor moves past it follows the approved rule.
- For each completed load and resource, every relevant record the source shows has exactly one outcome, landed, refused or missed, and the outcomes reconcile to the records shown, by content and any business key.
- Each record a load reads is marked newly read or re-read under the approved re-read rule, and the records read reconcile to what the approved rules selected, by content and any business key.
- Where the API offers an independent, enumerable view of its records, such as a full listing, an audit endpoint or a snapshot, every relevant record it holds is accounted for; otherwise the records visible through the API reconcile and the result states that unseen misses cannot be proven absent.
- The result states what the load cannot capture, such as deletes the approved policy does not detect, records later than any approved lookback, or corrections that leave the cursor unchanged.
Recipe promptThe task specification the agent reads. Reference only — it is not what you copy.
Build a dlt pipeline in the current Intent that lands <resource_list> from the <source_name> API into bronze. Inherit the Intent's repository, platform, Domain, sources and approved requirements, and resolve only the semantics they leave open. Load each resource incrementally under the approved cursor, any approved key and the approved write disposition, enforce the approved schema contract for new fields and changed types, keep every refused record visible with its reason, and account for each load's records by outcome (landed, refused or missed) and by extraction (newly read or re-read), completely where the API offers an independent, enumerable view of the records it holds; otherwise account for the records visible through the API and state that unseen misses cannot be proven absent. Report what the load cannot capture.
Agent guidanceHow the agent approaches the work, and what it will not do.
Inherit the Intent's repository, platform, Domain, sources and approved requirements. Confirm what the API exposes first: its resources and pagination, each resource's identity, which field marks a change and whether every change moves it, how deletes appear, and whether the API can filter on the cursor. Where the approved requirements leave them open, propose for approval the write disposition, key, cursor and starting point from that evidence, and state what each choice relies on and what it would miss, such as deletes, corrections that keep their cursor, or records that arrive late. Land each resource with the approved schema contract, keep refused records visible, and carry the approved provenance on every row. Find out whether the API offers an independent, enumerable view of its records, such as a full listing, an audit endpoint or a snapshot. Run the loads and give each relevant record the source shows one outcome and each record read an extraction status under the approved re-read rule; where such a view exists, reconcile each load against it by content and any business key, and where it does not, reconcile what the API exposes and say that unseen misses cannot be proven absent.
Composes
- capturing-requirements
- add-or-update-source
- discovering-source-schema
- generating-dlt-pipeline
- dlt-unit-testing
- running-dlt-in-sandbox
- ingestion-data-testing
- verifying
Asks first
Semantic decisions the Intent cannot supply. Never context Studio already holds.
- If unresolved, which API resources are in scope, and are child resources needed?
- If unresolved by the API's evidence, which write disposition, business key, cursor field and starting point each resource uses?
- If unresolved, how a record that arrives with a cursor at or behind what was already loaded is handled: picked up within a bounded lookback (how wide) or missed and reported?
- If unresolved, how the API shows a deleted record, how bronze should show it, and whether records deleted outright must be detected?
- If unresolved, what bronze keeps when a record is re-read, re-sent or corrected: one row per key with the latest copy, every copy, or the copies flagged, and how records are matched where a resource has no key?
- If unresolved, what the contract does when the API adds a field (add it, drop it, refuse the record or fail the load), and when an existing field, including one added later, changes type (refuse the record, keep the value apart or fail the load)?
- If unresolved, where refused records go and with what reason and payload, whether the cursor moves past them, and how a record with no cursor or no key is treated?
- If unresolved, which source identifiers and provenance every landed row must carry, whether the reconciliation must list individual keys as well as counts per load, and whether a re-read means the same key or the same record version?
Guardrails
- Do not choose or apply a cursor, lookback, delete, identity or schema-contract rule without approval.
- Do not coerce a value the contract refuses into a landed column, or overwrite a landed row with it.
- Do not fall back to a full reload to hide a cursor problem, or re-read records the stored cursor has passed beyond the approved lookback, unless that reload is approved.
- Do not claim deletes, late records or corrections are captured when neither the chosen cursor nor an approved detection can see them.
- Do not force business deduplication or transformation into bronze beyond the approved key.
- Do not treat a count match as reconciliation; compare content and any business key.
- Do not claim that no record was missed unless an independent, enumerable view of the source's records shows it.