Every wave of data engineering moved the work. The agent wave pays the teams that change how they build.

On 11 March 2026, Databricks launched Genie Code, and Ali Ghodsi, its co-founder and CEO, handed the industry its sentence of the year: "We're moving from a world where data professionals are assisted by AI to one where AI agents do the work, guided by humans."

Half the room heard a productivity story. The other half heard a headcount story and began rehearsing it for the next budget meeting. The first half is right, and the last three words decide how much of that productivity it gets.

"Guided by humans" is where the job goes. Trace how data engineering has changed hands since the mainframe and the pattern is almost boringly consistent. Every wave moved a responsibility somewhere else, and whenever a wave took work off the team, the team spent the time on more delivery. The agent wave does the same. It moves construction to your data engineering agent, and it leaves you holding the three things that were always the hard part: saying what correct means, deciding what evidence proves it, and owning the failures nobody automated. How well your team does those three decides how many more pipelines it ships.

The history is the best evidence we have about what happens next, so it gets the first half of this post. The second half is about managers, whose job changes more than anyone's.

Five waves, five handovers

The first wave was reporting bound to an application. IBM announced IMS for its mainframes in 1968, and for a long time afterwards a report was something a person who understood the source application extracted, reconciled and maintained. The knowledge lived in one head. The management problem was what happened when that head went on holiday.

The second wave was the warehouse. Ralph Kimball introduced dimensional modelling to the industry in 1996 with The Data Warehouse Toolkit. The work grew from extracting records to making separate systems agree on what a customer is. Engineers needed shared models. Managers needed departments to agree who owned a disputed definition and who accepted the result. Business meaning and governance were already a full-time argument in the nineties, whatever this year's keynotes imply.

The third wave was big data, and it went the other way. Hadoop's own design guide lists its first assumption plainly: "Hardware failure is the norm rather than the exception." That sentence moved a platform onto the data team's back. A team that wanted distributed processing now also ran the cluster. Anyone who tells you the history of data is a smooth ramp of simplification has never been paged at three in the morning by a NameNode.

The fourth wave handed the platform back. On 2 April 2009, AWS announced Elastic MapReduce with a line that reads like a eulogy for wave three: "You don’t have to rack it, network it, or administer it." The hardware moved to the provider. The workload stayed put. Databricks' own operational-excellence guidance, updated in September 2026, still tells customers to stand up a platform operations team, source control, automated deployment and monitoring. The provider took the racks. You kept the pipelines.

The fifth wave was analytics engineering. In March 2019, Claire Carroll wrote down a job that already existed: an analytics engineer "spends their time transforming, testing, deploying, and documenting data." Business logic became versioned code with tests and continuous integration. This is the bridge to everything after it. Meaning that used to sit in one analyst's head became something another person, or another program, could read.

Five waves, and five changes of who does what. Not one of them retired the data team. When a wave took work away, as managed cloud took the racks, the team spent the time shipping more.

The sixth wave comes in two halves

Agents arrive in two configurations. They ask different things of the people in charge, so it pays to know which one you are looking at.

In the first, agents build pipelines for people. Construction changes and consumption does not. Take a monthly customer-revenue dashboard for finance. The business owner approves the revenue definition, including what happens to refunds and currencies. The engineer records that as a specification someone can check and supplies independent reconciliation checks. Your data engineering agent builds the pipeline, the checks run against the proposed release, and the engineer handles the exceptions and authorises the release. Finance opens the dashboard it has always opened. The finance team never notices a wave went past, which is rather the point.

In the second, agents build pipelines for other agents. A revenue-analysis agent investigating a regional decline needs a customer attribute the governed model does not carry, so it requests a change. An agent proposes one. Release checks assess the contract, compatibility and the agreed acceptance evidence, and any change to what revenue means goes to a named human owner. After an authorised release, the analysis agent uses the new version. The immediate consumer is now a program, even if a person eventually reads its conclusion.

The consumption half already exists in the wild. In January 2026, OpenAI published an account of its in-house data agent, built on a platform that serves more than 3,500 internal users across 600 petabytes and 70,000 datasets. Read it as a specification for what a consuming agent needs. It has six layers of context. Its access is "strictly pass-through," so users "can only query tables they already have permission to access." It is evaluated continuously against golden SQL, because, in the authors' words, "Without a tight feedback loop, regressions are inevitable and invisible." Their third lesson learned is titled "Meaning Lives in Code." Wave five put the meaning there. In wave six, a program reads it with no human in between.

I should be straight about the grade of evidence. The first five waves rest on primary sources. The sixth rests on what we can already see in production. That is enough to act on and too early to call settled. Genie Code's announcement is a vendor's description of its own product, and OpenAI's account covers one company and the consumption side only, not the full loop in which agents build what other agents consume. You can see parts of the model in production today. An industry-wide reorganisation you cannot, yet.

Manual effort per pipeline, wave by wave. Illustrative, not measured: each later wave absorbs part of the work before it, and what the agent waves leave with people is defining correct and setting the contracts. From Vibedata research, "The waves of data engineering", 13 October 2026.

What the engineer keeps

Take construction out of the job and look at what remains. It is the part that was always hardest to hire for.

The engineer turns a request into a specification someone can check. They get agreement on business meaning from the people who own it, instead of inventing a definition of revenue at four in the afternoon because an agent needs an answer. They define the evidence that would show a result is correct, and that evidence has to be independent: a test that repeats the code's wrong assumption proves only that the code is consistently wrong. They review changes they did not write. And they own the failures outside the agent's authority. In the second configuration those are the nasty ones, where every pipeline test passes and the consuming agent still used the output to answer a question nobody asked.

Writing code stays in the job. Delegation adds a second job on top of it: judging work you did not personally write, at the speed it now arrives.

For the Data Reliability Engineer, this is the familiar job made explicit. "What is expected" and "what was built and why" stop being knowledge chased across Slack threads and become the artefacts every release is checked against. Or they never get written, and the agent ships at machine speed against a specification nobody wrote down. Agents do not remove that choice. They make it urgent.

What the manager keeps, which is more

The agent wave changes a manager's job more than an engineer's.

Managers used to organise specialist delivery. Now they organise a governed mix of human and automated work, and four decisions stop being abstract.

1. Ownership. Who approves business meaning, accepts the release, owns the service, and answers for harm downstream?
2. Delegation. Which tools and actions may agents use, and when? Which changes need a person to decide? Permission to read data is not permission to change a pipeline, publish a new definition or take a business action, and a consuming agent that requests work must not be able to commission unlimited pipelines.
3. Evidence. Which checks permit release, and how does the team find and correct the errors that slip past them?
4. Productivity and competence. Did time to PR and time per PR fall once specification, review and rework are counted? Does the team keep the skill to diagnose a failure when agents do the routine construction, including the juniors who once learned the trade by doing exactly that construction?

Every one of these existed before agents, in some form. What changes is how explicitly each must be enforced. A person improvising inside a vague mandate slows down when unsure. An agent improvising inside one does not slow down at all.

More generated code means more review. A manager who funds agents and not review capacity has bought a faster way to produce work nobody has checked.

Change how you build, and review less of it each time

Productivity does not arrive with the licence. A team that hands construction to an agent and keeps every other habit gets the agent's speed on the build and its own speed on everything else, and everything else is most of the job. Two numbers show whether the change is working: time to PR, the elapsed time from request to merged pull request, and time per PR, the human effort it takes to get one change accepted, counting specification, review and rework.

Review comes first, because review is where the gain goes to die. Agent output arrives faster than anyone can read it, and the tempting response is to rebuild every change in your head before approving it. Sometimes you must. A reviewer who does it for every change has moved the bottleneck from the build to the review and called the result governance.

The way out is to make each accepted change shrink the next review. Dan Shipper and Kieran Klaassen at Every call this compound engineering, and their rule is that "each unit of engineering work should make subsequent units easier—not harder." When a reviewer catches the agent mishandling refunds, the fix is a corrected pipeline, a definition the agent reads next time, and a check that runs without anyone remembering to run it. Their advice to a reviewer who distrusts the output is blunt: "If you don't trust the results, fix the system, instead of compensating by doing everything yourself." That is one software team describing its own practice. Data needs one addition: fixing the system includes evidence from running the change against real data, because a pipeline can pass every code review and still double-count revenue.

Four more changes feed that cycle.

• Write correct down before the agent starts. A checkable specification and agreed acceptance evidence cost time up front and save the rework when the agent guesses what revenue means.
• Give the agent context it can read. Business meaning, contracts and past decisions belong in the repository, where they outlast any one session. Every question the agent can answer from the repository is a question it does not stop to ask you.
• Build each change in isolation against real data. When every change gets its own environment, nobody queues for a development copy, and a failure shows up on the branch, before production.
• Set the write boundary. Decide which actions the agent may take alone and which need a person. A team that knows where the boundary sits can stop reviewing everything at the same depth.

Each of these leaves the next review smaller. The agent builds, a person checks, and what the person learns goes back into the system, so there is less to check next time. Time per PR falls with each cycle.

The same team, more pipelines

The headcount story assumes a fixed amount of work and asks how few people it takes. I have yet to meet a data team with a short backlog. Managed cloud took the racks away in 2009, and ten years later the industry was naming a new data job, analytics engineering, instead of retiring the old one.

The agent wave makes the same offer. Lower construction effort buys more pipelines, wider coverage or deeper review. Which mix your organisation buys is a decision, made by a manager, and no model makes it for them.

Construction and consumption also move independently. The finance dashboard in the first configuration survives the agent wave untouched. Agents change how it gets built long before anyone proposes changing how it gets read.

What the evidence supports is a test. Can your team delegate more work and still explain, verify and correct the result? A team that passes ships more pipelines with the people it has. A team that fails ships the same amount, wrong, faster.

Measure accepted outcomes

If you are running this transition, do not judge it by generated code, pipeline count or agent activity. Those are the vanity metrics of the sixth wave, and they will look magnificent right up to the month-end reconciliation.

Measure comparable work before and after delegation. Track time to PR and time per PR, and count all of the human effort in the second: specification, review, incident handling and rework, as well as build time. Both should fall, and keep falling as the cycle compounds. Check that accepted delivery went up without correctness, security, reliability or cost going the wrong way. For agent consumers, check interpretation separately from pipeline correctness, because the tests can all pass while the consuming agent answers the wrong question.

We are building Vibedata on that bet. Your data engineering agent does the construction inside isolation, and its work is accepted on evidence from independent execution against real data, never on the agent's own report of it. What your team teaches it goes back into the repository for the next change. The specification, the acceptance standard and the hard failures stay with your team. Every wave since 1968 suggests that is where they belong.

So here is a question for your next planning cycle. Of the pipelines your team shipped last quarter, how many came with a written definition of correct, and evidence that someone other than their author checked it? That number is the ceiling on how much you can safely hand to an agent.

References

• IBM, "Information Management Systems", IBM History
• Kimball Group, "Dimensional Modeling Techniques"
• Apache Hadoop, "HDFS Architecture Guide", version 1.2.1
• Jeff Barr, AWS, "Announcing Amazon Elastic MapReduce", 2 April 2009
• Databricks, "Best practices for operational excellence", updated 11 September 2026
• Claire Carroll, dbt Labs, "What is analytics engineering?", 26 March 2019
• Databricks, "Databricks Launches Genie Code: Bringing Agentic Engineering to Data Work", 11 March 2026
• Dan Shipper and Kieran Klaassen, Every, "Compound Engineering: How Every Codes With Agents", 11 December 2025
• Bonnie Xu, Aravind Suresh and Emma Tang, OpenAI, "Inside OpenAI's in-house data agent", 29 January 2026