The final data stack needs your agent

On 25 September, David Jayatillake published The Final Data Stack and told his readers to stop subscribing, because the blog was done.

It is the clearest statement yet of the world we are building Vibedata for: the agent does the data work, so the data team keeps the tools its agent can drive and cancels the rest. That leaves every data team with one question: how much of your stack can your data engineering agent understand, change and verify?

That question already has a name. ADER, our Autonomous Data Engineering Readiness model, scores a data estate on thirteen dimensions that decide whether an agent can change it safely and get the answer right. Eight cover safety and are numbered M1 to M8; five cover correctness and are numbered S1 to S5. The fourth safety dimension, M4, is stack fit: how much of your existing stack a given agent's tools can understand, change and verify. This post is about why David's argument lands on it, and what we think the final stack still needs before an agent can run on it without a human watching.

Where David is right

David has earned the right to make this call, and from both sides of the purchase order. He ran data teams for most of a decade, as Head of BI & Analytics at Elevate Credit and then Senior Director of Data at Lyst, where he moved the company from Redshift to Snowflake in eight weeks ahead of Black Friday and ran analytics on Snowflake, dbt and Looker. He has signed for the warehouses and BI seats his post now tells you to cut. Then he built for the other side of the table: co-founder and CEO of Delphi Labs, VP of AI at Cube, and now co-founder and CTO of Quarri. He also invests in data tools, Lightdash and Streamkap among them, and says so plainly: "I say this all as someone who stands to lose money if I'm right."

His argument, in four moves.

The budget is coming for the tools. Large companies can spend up to 0.5% of operating expenditure on BI tools alone. At a billion in opex, that is five million, which "could easily cover the whole cost of their analytics team". Next year the CFO stops asking and starts telling. The savings will come from software or headcount, because those are the two big costs of a data team.

Tokens are the wrong place to save. "Intelligence keeps getting cheaper." Cutting token spend as a data team is "cutting off your nose to spite your face". The test is return per token, and for data work the return is easy to find.

The agent does the work, so the UI stops mattering. This is the line the rest of the post hangs on:

"We aren't going to be using any data tools by hand any more. [...] So if the agent is going to use it, the pretty UI and nice product design cease to matter. What matters is whether the API can cover the full breadth of actions you need to take and content you want to deliver and that's all."

The agent has to be yours. Addressed to BI vendors, and true for everyone else:

"I don't want to use your agent, I want to use mine... the one that is plugged into every other thing I use already, blessed by Infosec."

We agree with all four. We would go further on the third. Once the agent does the work, a tool is worth what the agent can do with it. A dashboard nobody opens by hand is a query and a layout. A pipeline tool with a beautiful canvas and a thin API is, to an agent, a thin API. The ranking of tools changes from "which one do my people like using" to "which one can my agent drive end to end".

That ranking is M4. It is the one dimension of the thirteen that depends on which agent you assess. The other twelve say what any agent needs from any estate: code that built each table, a test that fails when something breaks, a declared grain. M4 says what this agent can do with your estate, and a gap in it marks a limit of the agent.

David's post is an M4 argument from the platform side. Databricks wins his assessment because, in his words, it put "every single object that can be created or managed in Databricks available by API and CLI". The platform raised its own stack fit for every agent at once.

What the one-stop shop leaves out

We agree with the direction. We part company with the ending.

David's final stack is one platform plus a general-purpose agent. "Databricks is truly a one stop shop for all your data stack needs, all nicely integrated and easy for your agent to use." His advice to everyone else building agents: "Stop trying to build your own agent."

Two things sit between that picture and the estates we see.

Most estates are several platforms, and M4 is scored against all of them. The one-stop shop is a destination. The estate a data team runs today holds a warehouse from one decade, a lakehouse from the next, a SaaS source nobody wants to touch, and a folder of stored procedures that pays the bills. David's own ending admits the consolidation is still in play: BI vendors are about to offer hosted query engines and "eat their compute business back". In that world, stack fit is a property of the agent across the estate. An agent that is excellent on one platform and absent on the other three has an M4 score the size of one platform.

API breadth gives the agent reach, and safe change needs more. An API that exposes every object lets an agent touch every object. That is the start of the problem. A general coding agent pointed at a data platform through a CLI still lacks four things data work needs, and they come as a set:

• Isolation. A wrong hypothesis has to be disposable. Code has branches. Data needs the same: a place where the agent can build, run and break things against real data without touching production.
• Guardrails. Something other than the agent's own judgement has to decide what ships. At agent speed, human review of every change is the bottleneck, so the gate has to be automated: tests, data diffs and review checks on every change before it reaches production.
• Context. A coding agent reads the repository. A data engineering agent also has to read the data: profile the tables, find the grain, check the join, see what the numbers look like before it decides what to build.
• Cross-platform reach. The same skills, the same checks and the same way of working, on whichever platform the change lands.

The fair reply is that platforms will build these in. Some will, for their own platform, and that is good news: every platform that exposes more of itself to agents raises stack fit for every agent, ours included. It also puts each gap inside one platform's walls. A team whose estate spans three platforms gets three sets of guardrails, three kinds of isolation and three partial views of context, and its agent has to reconcile them. We would rather the agent carry one way of working across all three.

No UI supplies any of the four, and they are what would make David's "two days of occasionally pressing enter" safe to leave running overnight. They barely show in a demo, and they decide whether the agent's third change of the day double-counts revenue in production.

So we would edit one sentence of the post. BI vendors should stop bolting an agent onto a UI; on that, David is right, and the Salesforce example makes the case. The data team still needs a data engineering agent. It needs that agent to be its own.

What "mine" has to mean

David's definition of "mine" is the best spec we have read for what we are building: plugged into every other thing you use, blessed by Infosec, improving constantly. Here is how Vibedata meets each clause.

Blessed by Infosec. Vibedata runs on your own Kubernetes or on local Docker, inside the same network perimeter as your data platform. There is no Vibedata-operated multi-tenant service holding your workloads. The code the agent writes lands in your GitHub and runs on your platform, so nothing is captive to us. Your security team reviews one deployment in your tenant, on your terms.

Plugged into everything you use. Work arrives where the team already works. Studio is the primary surface, and Slack, Teams and other channels reach the same piece of work. CI gates what reaches production, and GitHub Issues carry detect-to-fix work between build and operate.

Improving constantly, on your tokens. David is right that intelligence keeps getting cheaper, and we built for it. Vibedata is model-tiered: frontier models where judgement carries the load (specification, design, review), cheaper tiers where the work is well specified (generation, mechanical checks), all on endpoints you provision. The gate decides what ships, never a model. That is what lets a team ride the price curve down without riding quality down with it, which is exactly the return-per-token discipline David asks for.

Reaching across your stack. This is where M4 comes back. Vibedata standardises on a small open-source stack, dlt for ingestion, dbt for transformation and Elementary for data quality, because open standards travel across platforms and a narrow tool surface makes agents reliable. The same skills and checks run on DuckDB, Microsoft Fabric and MotherDuck today. Design partner open: Databricks, Redshift. Coming next: BigQuery, Oracle, Postgres, Snowflake. When we assess an estate on M4, we say which parts of it Vibedata can understand, change and verify, and which parts it cannot yet. The scope is written down so the score can be compared over time, and a gap is our limit to close.

Carrying the four gaps as the product. Each change runs in an isolated workspace against real data, gets checked by CI before production, and draws its context from profiling your tables as well as reading your code. That is what we spend our engineering on. It is the part of "mine" a CLI cannot supply.

Where this leaves the data team

David's post ends on the vendors. The more useful ending is for the data team, because he is right that the squeeze is coming and right that it lands on software or headcount.

The team that goes into next year's budget round well has three things to show. A shorter tool list, because the tools that survive are the ones its agent can drive. A token bill it can defend on return, because the expensive models do the judgement and the cheap ones do the volume. And a stack-fit map: which parts of the estate its own agent can already understand, change and verify, which parts it cannot, and what it will take to close the gap.

The third one is the argument for keeping the team. A data team with an agent that can safely change most of the estate is one of the "single pizza data teams" David predicts, managing "vast amounts of work". A data team whose agent can reach one platform out of four is still paying people to do the other three by hand.

So here is the question to take into the planning cycle. List every platform, pipeline and scheduled job your team owns. For each one, can your agent understand it, change it and prove the change is right, without a human in the loop for the proof?

The rows where the answer is no are your real budget risk. They are also the work we are building Vibedata to do.