I just sat in a room full of data engineers the other week who were worrying about AI automating them out of work the same way auto manufacturing in Detroit was upended half a century ago.
All AI. All the time. That’s what technology professionals are talking about.
Data scientists, data engineers, and data architects are right to sound an alarm at that. Using AI to solve and automate data problems all the way at the beginning of the pipeline is an obvious use case of agentic engineering in data. Shifting AI left for automation.
That looms as a threat to data engineering positions who own the pipeline underlying the architecture and deliverables. It’s a discussion that we can no longer avoid. In all fields, AI is looming, bringing with it new risks and bigger change.
Introducing AI there can be dangerous, and that’s a conversation all its own. You hear horror stories about AI initiatives that failed—and what failed them.
Agentic frameworks stall because the retrieval layer can’t be trusted. RAG pipelines work in demo then fall apart in production. Problems that should have been solved upstream are solved by building governance tools downstream.
The conversation comes back to one thing. The data wasn’t ready.
Don’t neglect the data layer
A Cloudera and Harvard Business Review study from March 2026 found that only 7% of enterprises consider their data completely ready for AI, and over a quarter said it wasn’t ready at all. Another data point: In Informatica’s 2025 CDO Insights survey, 43% of organizations named data quality and readiness as their top obstacle to AI success. Not model performance. Not tooling. Data.
So why does this keep happening?
Organizations are treating AI as a technology procurement decision. Buy the platform, hire the engineers, deploy the models. But the foundation underneath those initiatives—the data layer—is missing.
The data wasn’t governed. The lineage wasn’t tracked. The pipeline was built for reporting, not for model consumption.
The engineers in that room could easily be part of the solution. Because nobody owned the quality problem. And when the model surfaced a confident, wrong answer, nobody could trace it back to find out why.
That’s not an AI problem. That’s a data problem that AI made visible.
Readiness starts before the model
Data that feeds AI systems needs to be made consistent and owned. Not owned in the sense of having a name in a RACI chart. Owned in the sense that an engineer or data professional is accountable when it degrades. Lineage matters because AI outputs are only as auditable as the data behind them. Quality matters because model performance in production is directly correlated with what goes in.
These aren’t new principles. They’re established data engineering practices. They just haven’t been treated as AI deployment fundamentals. That needs to change.
Data readiness closes the gap between AI ambition and AI outcomes. McKinsey’s 2025 State of AI survey found that organizations investing in their data foundations first were likely to see real financial returns from AI. Without solutions like data contracts between producers and consumers, automated quality monitoring at the pipeline level, and governance frameworks that treat AI as a first-class data consumer rather than an afterthought, your AI spend will be wasted.
Thinking back to my convo with those engineers a few weeks ago, the engineers in that room worried about being automated out of work. Data engineers who understand pipelines, lineage, and quality at depth aren’t facing obsolescence. In fact, there’s a good chance they’ll soon see demand for their services spike, as organizations realize their AI initiatives aren’t failing because they hired the wrong AI engineers. They likely failed because those organizations didn’t invest in the data infrastructure and engineers.
The data engineering job isn’t going away. It’s changing shape as it solves a problem we’re all facing and talking about.
For data engineers, AI readiness is a table stakes deliverable now. That means owning the data that feeds AI systems, and building governance frameworks around what AI actually consumes. AI engineers, for their part, have to stop treating the data layer as someone else’s problem. When an agentic framework stalls or a RAG pipeline falls apart in production, the instinct is to look at the model or the retrieval architecture. The data is usually where the answer is. It behooves these two disciplines to share a definition of “done” that includes the data being ready before the model is deployed rather than after it fails.
The AI problem, for most organizations, is a data problem that can be solved by data engineers and data professionals. The sooner that lands in the boardroom, the better the odds that the next initiative doesn’t end up in the abandoned 42%.