Data engineering is a hard job — waking up to failed runs that take hours to investigate, addressing a constant backlog of asks and needs from business teams, and answering to finance teams that want to know why the cost of all these pipelines keeps going up. And in many ways, AI is making it harder, bringing in more and varying types of data, and letting far more people across the company use that data through coding and chat agents that can write SQL for them. Most data engineering teams are trying to keep up by adopting their own coding agents, but still, they seem to be falling further and further behind.
An MIT Technology Review report on "Redefining Data Engineering in the Age of AI" found that while 8 in 10 organizations have deployed AI-based data engineering tools, data engineers are also managing more complexity with the biggest challenges reported as ensuring data security and privacy (55%).
We need a more fundamental shift. Instead of looking to AI to make existing processes run faster, it's time to rethink the processes themselves and move beyond manually building and maintaining pipelines. Data teams will get more time to focus on the output of data engineering, data products, rather than the process of transforming that raw data step by step. It will become critical for teams to ensure data is ready to consume by both humans and AI with the context and governance intact.
The future of data engineering will hinge on creating actual, usable data products that can fuel business decisions, and having truly autonomous agents build, maintain and optimize everything needed to keep those products up to date with humans as the key decision makers and arbiters.
These agents will pay attention to new products needed, problems and failures with existing ones, and search for opportunities to improve, clean up and optimize.
Humans will of course be critical, but their role gets elevated, defining the most important business problems, guiding the agents on the right overall architecture and making sure that the context of the business is appropriately captured for the agents consuming the data products.
Jobs “professionalized” by AI are growing twice as fast as jobs “democratized” by AI, with 42% faster wage growth since 2021.
Source: PwC AI Jobs Barometer 2026
Journey toward Autonomous Data Engineering #
This is a big shift, and teams won't be able to get there overnight, nor should they. A better approach is to decide that this is the right goal, and then move toward it incrementally and thoughtfully. At each step, teams will learn more about what the agents and models are capable of, and will build trust in how much can be safely automated, and where you need additional human-in-the-loop (HITL) checks, tooling or processes. Many companies are already underway: Every team that has adopted a software-defined lifecycle, version control, declarative pipelines or AI-assisted development has taken a step toward the future.
The Autonomous Data Engineering maturity curve
| Stage 1: The Foundation | In this stage, data engineers build and maintain pipelines manually. Every schema change, orchestration decision and failure response requires human effort. Teams that treat Stage 1 as the place to get their engineering fundamentals right will have more success in later stages. Movement to more modern data engineering practices, such as declarative pipelines, can generate improved efficiency. Travelpass, for instance, made its data engineering efforts more accessible, which helped the team scale and deliver data to business units over 350% more efficiently. | | Stage 2: Copilots | This is where AI enters the editor. Autocomplete and inline code generation reduce the friction of transformation and configuration, but engineers still own every decision. The fundamental workflow hasn’t changed, even though output has increased. | | Stage 3: Agentic: Human in the loop | Stage 3 is where AI starts becoming a collaborator. Agents can propose changes like pipeline modifications, failure fixes and schema migrations, but nothing executes without explicit human approval. Engineers review code instead of writing it, which will help radically shorten incident response time. | | Stage 4: Agentic: Human on the loop | Automation benefits begin to truly shine at this point. Agents act independently on select, clearly defined tasks, such as detecting anomalies and adapting pipelines to upstream changes, without waiting for approval. Engineers still have observability into what agents are doing and can override agents’ actions when necessary. | | Stage 5: Autonomous Data Engineering | At Stage 5, data engineering has fundamentally changed what it means to work in data. Pipelines detect their own failures and root-cause issues across the stack, apply fixes, validate outcomes and document what changed without human intervention. Operational overhead approaches zero. Data engineers define the policies, standards and business semantics. They also decide what the platform is allowed to do autonomously and what requires their judgment, so high-impact changes are routed to human review. Their role is fully shifted from execution to governance, and they’ve elevated from pipeline builders to genuine business partners. |
“Doing everything exclusively in Snowflake was game-changing. Now we design something on a Thursday, and by Tuesday it’s in production.”
Thomas Bodenski
COO & Chief Data & Analytics Officer, TS Imagine
Redefining how data products are built in the agentic era #
One of the big shifts that enables this is a move from thinking about data pipelines to thinking about data products, which encompass not just data but semantics, quality and everything an agent needs to be able to accomplish its tasks with the data. For example, a customer loyalty data product may include customer churn-risk data, which pairs each individual account activity data with a semantic view defining what the business defines as “at-risk,” along with documented quality checks. Any agent or analyst accessing this data product would then get a consistent, trustworthy answer for their customer loyalty efforts.
The traditional model starts from infrastructure and builds up: build the data foundation, define schemas, create pipelines, then serve a business need. That worked when data products were few and stable.
Modern data teams are running this in reverse: Start with the business problem and identify the agent workflow that addresses it. Determine what data that agent needs, and build the data product to deliver it. This reorients the data estate around business outcomes rather than technical layers and introduces a vertical approach to data products, where pipelines are designed for results, not schemas.
In true autonomous data engineering, the platform matters as much as the choice of agent framework. High-performing teams must first solidify version control, automated testing and CI/CD pipelines — the same best practices that high-performing data teams have been adopting for years. Governance, interoperability and unified data are the preconditions that make it safe to trust autonomous workflows.
Build the foundation; autonomous data engineering will follow #
The shift to autonomous data engineering is an operational necessity. The teams that reach maturity fastest won't get there because they had the best models. They'll get there because they elevated the role of the data engineer to a business partner and built a foundation strong enough for autonomous agents to operate on with governance solid enough to trust them.
To be successful in this journey, customers need to bet on a platform that will not only serve their needs today but continue to evolve and absorb new innovations in both data engineering and AI, without having to stitch together or build everything themselves. Learn how Snowflake is setting the standard to measure quality and token efficiency for autonomous data engineering work with the Data-eng-benchmark.
For a practical guide to getting started with autonomous data engineering, including architecture patterns, governance frameworks and the engineering practices that make agentic workflows reliable, read Build Pipelines for AI: An Essential Guide to Smarter Data Engineering.