Re-engineer a legacy data package one Intent at a time

Take a package with two procedures and a materialized view. The view sits on the output of both procedures. One package, three objects, two dependency edges, and a decision about how much work to hand over at once.

The method is to re-engineer it one Intent per object. The alternative is one big-bang job, and the case for it is stronger than it looks. Each Intent carries that object's dbt model and the tests the team has agreed to. Its requirement is complete before the agent starts building.

A package is a set of objects with their own tests

An object here is a unit of work with its own inputs, outputs and acceptance test. File types do not define it. A procedure that loads a table is an object. A view that aggregates two procedures is an object if you can test its totals without rebuilding the procedures.

An object can be a procedure, a view, a table or a model. The label matters less than the test. If the object has one output and one way to say it is correct, it can stand on its own.

You can find the boundary by asking what correct looks like for the object alone. If you can state that, the object can carry its own Intent. If you cannot, it belongs with the object it depends on.

The incremental idea is older than agentic data engineering. Martin Fowler described the strangler fig in 2004: replace a legacy system one slice at a time [3]. What changes with an agent is the size of the slice it can carry and verify.

The package can be large. The unit you hand over stays small enough to specify, review and verify.

One Intent per object makes the requirement completable

An Intent is the unit of work, the branch and the audit trail. Its requirement records the ask, the scope and the acceptance conditions. The agent reads it, decides what has to happen for that object, and stops for review before it builds.

Our Intent flow already puts the requirement in front of a reviewer before design and planning start [1]. The tests are agreed at that gate and recorded in the requirement. Each Intent gets its own branch and an isolated workspace on the data platform [2], so a failed object does not touch the one that already passed.

The object boundary is what makes the requirement completable. A whole package has too many unknowns to specify in one document, so the agent fills the gaps by guessing. An object has a smaller surface: its inputs, its output grain and its tests. The requirement can name all three.

Complete here means the reviewer can approve the requirement without asking what the object reads, what it produces or how correctness is measured. The agent can start without inventing an acceptance test. The requirement can also name what is out of scope, so the next Intent does not inherit it by accident.

The agent profiles the source, proposes the dbt model, writes the tests and runs them in the Intent's workspace. The requirement gate is where the team agrees that the object and the test are the right ones. That is a smaller decision than approving a whole package, and it is the decision the build depends on.

That workspace holds the data the Intent runs against [2], so the object can be tested before the rest of the package is rebuilt. That is the difference between a requirement that is complete and a scope that is merely long.

The dependency order lives between Intents

The strongest objection to this split is the one about dependencies. If each object gets its own Intent, where does the package-level order live?

It lives in the order of the Intents, and in the dependency map the team already has to draw. In this package, the materialized view depends on both procedures. That is the package-level edge the map has to carry. If the procedures are independent, two Intents can run in parallel and the view waits; if one procedure feeds the other, the map orders them.

The map can be a table with one row per object and a list of the objects it reads. The row order is the Intent order. A cycle is a sign that the boundary is wrong, because two objects that need each other are one object for this purpose.

The map is one artifact, reviewed once. The Intents are the work, reviewed one at a time. The split keeps the order visible; a single giant scope hides it.

The opposing view says the object boundary is an implementation detail. It is, until you ask what the reviewer can check on their own. At that point the boundary is the unit of delegation.

More reviews, each one small enough to read

The objection is right about the count. Three objects mean three Intents, three branches and three reviews. A single big-bang job means one review.

What matters is how much of each review a person can check. A review that covers a whole package asks the reviewer to hold the procedures, the view, the joins and the tests in their head at once. A review that covers one object asks them to check one requirement against one diff. One giant diff makes the reviewer choose between reading it and trusting it, and at some size the review becomes a signature.

The count rises, and the cost per change falls, because each review is smaller. The productivity claim for Vibedata is that the same team ships more, measured as time to PR and time per PR. The gain depends on changing what the team focuses on and how it builds, and review compounds as each accepted change shrinks the next review. We have not measured the saving on a package like this.

A big-bang job hides the failure until the end

A single big-bang job gives one giant diff and one acceptance test at the end. A wrong join can hide in it until the whole package runs. A wrong join in one procedure can also invalidate the view's totals, and the package-level test tells you the totals are wrong without saying which procedure caused it.

A batch of generated models can look like progress because it produces many files. It can produce a model for every table and still leave the view without an acceptance test. The file count grows; the test count does not.

The object boundary gives each change a test it can fail on its own. The failure arrives at the smallest diff that can carry it.

How to slice a package without losing the order

1. Inventory the package by object. Name each procedure, view and table that has its own output.
2. Draw the dependency edges. An object that reads another object's output comes after it.
3. Write the acceptance test before the requirement. If you cannot state what correct looks like for the object alone, merge it with its dependency.
4. Open one Intent per object in dependency order. Put the dbt model and the agreed tests in the requirement.
5. Merge each Intent when its tests pass. The next Intent starts from the merged state and inherits the accepted change.

The shape is one package, three object Intents, three PRs in order.

The test is whether the object can prove itself

The object boundary is a test you can run before you open an Intent. Can you state the object's inputs, output grain and acceptance test without describing the rest of the package? If yes, it can carry its own Intent.

If no, keep it with the object it depends on. Count the tests. The object boundary is also the review boundary: if the reviewer cannot check the object alone, the Intent is too big.

The package can be large. The unit of delegation stays small enough to specify and review.

The method does not make a bad package small. A package with one tangled object has one Intent, and that Intent is still large. The boundary test tells you where the cut is; it does not promise the cut is free.

Which object in your legacy package can prove itself today? Start there, and let the next Intent depend on the merged result.

References

1. Agentic data engineering is delegation, not automation, the requirement, design and plan gates.
2. The AI-native data engineer is real, one Intent, one isolated workspace.
3. Martin Fowler, StranglerFigApplication, 2004 (replace a legacy system incrementally, one slice at a time).