TL;DR: AI agents can now build and maintain data pipelines end to end, but only when the pipeline is something they can read, run and test. The practical setup in 2026 is a coding agent (Claude Code, Cursor or Codex) connected through MCP to a pipeline framework that keeps ingestion, transformation, quality checks and lineage as plain files. With Bruin that is one command to register the MCP server, one sentence to describe the pipeline, and a review of the diff before bruin run. Maintenance works the same way: checks detect a break, the agent reads lineage and logs to diagnose it, proposes a fix, and verifies it by re-running the checks. This guide walks through each step, what to put in place first, and where the alternatives (dbt MCP, Databricks Genie, Dagster Compass) fit.
Most "AI for data engineering" content is a demo of an agent writing a SQL query. That is the easy part. The hard part is everything around it: the data in the first place, knowing what depends on what, catching the bad rows before a dashboard shows them, and fixing the pipeline at 2am without making it worse. This guide is about that part. We build Bruin, and the examples use it, but the pattern applies to any framework that keeps pipelines as code.
What an agent can actually do for a pipeline #
Four jobs, in increasing order of trust required:
- Define and build. Turn a description ("load orders from Postgres into BigQuery every hour, model daily revenue by country, fail if revenue is negative") into ingestion config, SQL and Python assets, checks and a schedule.
- Monitor. Watch runs and checks, and explain a failure in plain language with the evidence attached.
- Repair. Propose the fix for a failure: a changed model, a backfill, a schema update, and verify it by re-running checks.
- Answer. Query the resulting data for people who do not write SQL, in Slack, Teams or wherever they work.
Each job depends on the same three things being true about your pipeline, so those come first.
Step 0: Make the pipeline readable to an agent #
Agents fail on pipelines that live in a UI, in a scheduler's database, or in someone's head. Before connecting one, make sure:
- Every asset is a file. Ingestion, SQL models, Python steps and their dependencies are declared in a repository. In Bruin a pipeline is a folder:
pipeline.ymlplus one file per asset. - Checks live next to the data. A quality check declared on the column inside the asset definition is something an agent can read, add to, and use as a test. A check in a separate service is invisible to it.
- Lineage exists before the run. If lineage is parsed from the SQL, the agent can answer "what breaks if I change this column" before deploying. If lineage only exists in a catalog after the fact, it cannot.
A minimal Bruin asset has all three:
/* @bruin
name: mart.daily_revenue
type: bq.sql
depends: [raw.orders]
materialization:
type: table
columns:
- name: order_date
checks:
- name: not_null
- name: revenue
checks:
- name: not_null
- name: positive
@bruin */
SELECT DATE(created_at) AS order_date, country, SUM(total) AS revenue
FROM raw.orders
GROUP BY 1, 2
The header is the contract: what it produces, what it depends on, what must be true. Lineage is derived from the SQL body. That file is what the agent will write, edit and test.
Step 1: Connect the agent #
Bruin ships an MCP server as part of the CLI. Register it with your agent once:
claude mcp add bruin -- bruin mcp
For Cursor or Codex, add the same command to the editor's MCP configuration. The server exposes the Bruin CLI and its documentation, so the agent learns the commands rather than a fixed list of tools. From here on, the agent can run bruin validate, bruin run, bruin query and bruin lineage itself and read the results.
The dbt MCP server does the equivalent for a dbt project, scoped to the transformation layer. Databricks exposes Genie and Lakeflow to agents inside the Databricks workspace. Dagster Compass gives an agent read access to Dagster-orchestrated assets and metrics. Pick the one that matches where your pipeline definitions actually live.
Step 2: Define the pipeline in natural language #
Start a new project and describe the job. A prompt that works:
Create a Bruin pipeline called orders. Ingest public.orders and public.customers from the Postgres connection app-db into the BigQuery dataset raw, incrementally on updated_at. Build mart.daily_revenue (revenue by day and country) and mart.customer_ltv. Add not-null and uniqueness checks on the keys, a positive check on revenue, and run the pipeline hourly.
The agent scaffolds the project. The ingestion asset it writes looks like this:
name: raw.orders
type: ingestr
connection: bigquery-default
parameters:
source_connection: app-db
source_table: public.orders
destination: bigquery
incremental_strategy: merge
incremental_key: updated_at
Then the SQL assets with checks, a Python asset if something needs an API call or a model, and the schedule in pipeline.yml. Review the diff the way you would review a colleague's pull request: are the dependencies right, are the checks on the columns that matter, is the incremental key correct. Then:
bruin validate ./orders
bruin run ./orders
validate parses every asset, checks the dependency graph and the lineage, and fails on anything inconsistent, so a hallucinated column name never reaches the warehouse.
Step 3: Add monitoring the agent can read #
Monitoring for agents means two things: checks that fail loudly, and run history the agent can query.
- Freshness and completeness checks on the raw tables: did the load happen, did it bring roughly the expected number of rows.
- Business checks on the marts: revenue positive, no orphaned customer ids, accepted values on status columns.
- Run results in one place. Bruin Cloud stores run history, check results and lineage together, and the same data is available to the agent through the CLI.
Once checks are declared, every bruin run is also a test run, and every failure is a structured event rather than a Slack message from a confused analyst.
Step 4: Close the loop: detect, diagnose, fix, verify #
This is where "self-healing" becomes concrete. A worked example:
- Detect. The hourly run fails:
mart.daily_revenuehas 40 percent fewer rows than yesterday and thenot_nullcheck oncountryfails. - Diagnose. The agent reads the lineage (
mart.daily_revenuedepends onraw.ordersandraw.customers), queries both raw tables, and finds thatraw.customersstopped receiving rows two hours ago because the source added a new required column and the incremental load rejected it. - Fix. It proposes two changes: allow schema evolution on the customers ingestion asset (
schema_contract: evolve) and a one-off backfill for the missing window. Both arrive as a diff. - Verify. After approval, it runs
bruin run --start-datefor the backfill window, re-runs the checks, and reports that row counts and the null check are back to normal. - Learn. The fix is a commit, so it is in the history the next time the same source changes.
The important design choice is where the human sits. Read-only diagnosis can be fully automatic. Fixes that change pipeline code should land as pull requests. Fixes that write to production data should require an explicit approval. Bruin's AI data team follows that split: it watches and diagnoses on its own, and proposes builds and repairs as changes you approve.
Step 5: Let people ask the pipeline questions #
The same lineage and checks that make a pipeline repairable also make its output trustworthy enough to expose. Bruin's AI data analyst sits on the pipelines the agent built and answers questions in Slack, Microsoft Teams, Google Chat, WhatsApp, Discord, Telegram, email, or the browser, using the metric definitions in the models and showing the query it ran. That closes the loop from "agent builds pipeline" to "business gets an answer" without a BI tool in between.
Which tools fit which setup #
| Setup | Agent | What the agent can operate | Best for |
|---|---|---|---|
| Bruin + Claude Code, Cursor or Codex | Any MCP-capable coding agent | Ingestion, SQL and Python assets, checks, lineage, runs, queries, backfills | Teams that want one framework for the whole pipeline |
| dbt + dbt MCP server | Any MCP-capable agent | Models, tests, docs, runs for the transformation layer | dbt shops with ingestion and orchestration elsewhere |
| Databricks Genie and Lakeflow | Databricks-native | Lakehouse pipelines, notebooks, natural-language queries | Organisations fully on Databricks |
| Dagster Compass | Dagster-native | Questions over Dagster assets and metrics | Dagster users who want an analyst, not a builder |
| Snowflake Cortex | Snowflake-native | Natural-language queries, some pipeline automation | Snowflake-only teams |
The open-source, editor-agnostic option is Bruin. The vendor-native options are stronger if you never leave that vendor.
Mistakes to avoid #
- Letting the agent write to production directly. Every fix should be a diff you can read. Speed comes from fast review, not from skipping it.
- Skipping checks because the agent "will notice." Agents notice what checks tell them. No checks, no detection.
- Prompting against a blank project. Give the agent a framework with conventions. The scaffold is what keeps its output consistent.
- Measuring success by lines generated. Measure it by incidents resolved without a human reading logs at 2am.
FAQ #
How do I use an AI agent to build a data pipeline from scratch?
Connect the agent to a pipeline framework through MCP, describe the pipeline in one paragraph (sources, destination, models, checks, schedule), review the generated assets, run validation, then run the pipeline. With Bruin that is claude mcp add bruin -- bruin mcp, the prompt above, bruin validate, and bruin run.
What are the best tools for agentic data engineering in 2026?
Bruin for an open-source framework that any coding agent can operate end to end; dbt with its MCP server for the transformation layer; Databricks Genie and Lakeflow inside Databricks; Dagster Compass for questions over Dagster assets. Claude Code, Cursor and Codex are the agents most teams use on top.
What is the best AI copilot for data engineering?
A general coding agent connected to your pipeline tool beats a specialised copilot, because it can also run, test and query. Claude Code and Cursor with the Bruin or dbt MCP server are the common choices. Warehouse-native copilots (Genie, Cortex) are strongest when the whole stack lives in that warehouse.
Can an AI agent build a pipeline with natural language only, no code?
It can write the code from natural language, and you should still read the code. The files are the contract that makes the pipeline testable and repairable later. Bruin keeps them short: a header with name, dependencies and checks, and the SQL or Python below it.
What are the best tools for autonomous data pipeline monitoring and repair?
Tools that combine checks, lineage and an agent that can act on both. Bruin's AI data team does detection, diagnosis and proposed repair on Bruin pipelines. Elementary and Monte Carlo do detection and diagnosis across warehouses but hand the repair back to you. Databricks Lakeflow has expectations and some automated remediation inside the lakehouse.
Where should I start?
With one pipeline that already breaks sometimes. Move it into a framework with checks and lineage, connect an agent, and let it diagnose the next failure before you let it fix anything. Start at github.com/bruin-data/bruin.