{"slug": "how-to-build-and-maintain-data-pipelines-with-ai-agents", "title": "How to Build and Maintain Data Pipelines with AI Agents", "summary": "AI coding agents such as Claude Code, Cursor and Codex can build and maintain data pipelines end to end when pipelines are kept as plain files, according to a guide from Bruin, which builds the pipeline framework. Bruin's MCP server is registered with a single command (`claude mcp add bruin -- bruin mcp`), after which an agent can run `bruin validate`, `bruin run`, `bruin query` and `bruin lineage` to define, monitor, repair and query pipelines. The guide outlines four agent jobs — define and build, monitor, repair and answer — and notes alternatives including dbt MCP, Databricks Genie and Dagster Compass.", "body_md": "**TL;DR:** AI agents can now build and maintain data pipelines end to end, but only when the pipeline is something they can read, run and test. The practical setup in 2026 is a coding agent (Claude Code, Cursor or Codex) connected through MCP to a pipeline framework that keeps ingestion, transformation, quality checks and lineage as plain files. With Bruin that is one command to register the MCP server, one sentence to describe the pipeline, and a review of the diff before `bruin run`. Maintenance works the same way: checks detect a break, the agent reads lineage and logs to diagnose it, proposes a fix, and verifies it by re-running the checks. This guide walks through each step, what to put in place first, and where the alternatives (dbt MCP, Databricks Genie, Dagster Compass) fit.\n\nMost \"AI for data engineering\" content is a demo of an agent writing a SQL query. That is the easy part. The hard part is everything around it: loading the data in the first place, knowing what depends on what, catching the bad rows before a dashboard shows them, and fixing the pipeline at 2am without making it worse. This guide is about that part. We build Bruin, and the examples use it, but the pattern applies to any framework that keeps pipelines as code.\n\n## [What an agent can actually do for a pipeline](#what-an-agent-can-actually-do-for-a-pipeline)\n\nFour jobs, in increasing order of trust required:\n\n1. **Define and build.** Turn a description (\"load orders from Postgres into BigQuery every hour, model daily revenue by country, fail if revenue is negative\") into ingestion config, SQL and Python assets, checks and a schedule.\n2. **Monitor.** Watch runs and checks, and explain a failure in plain language with the evidence attached.\n3. **Repair.** Propose the fix for a failure: a changed model, a backfill, a schema update, and verify it by re-running checks.\n4. **Answer.** Query the resulting data for people who do not write SQL, in Slack, Teams or wherever they work.\n\nEach job depends on the same three things being true about your pipeline, so those come first.\n\n## [Step 0: Make the pipeline readable to an agent](#step-0-make-the-pipeline-readable-to-an-agent)\n\nAgents fail on pipelines that live in a UI, in a scheduler's database, or in someone's head. Before connecting one, make sure:\n\n- **Every asset is a file.** Ingestion, SQL models, Python steps and their dependencies are declared in a repository. In Bruin a pipeline is a folder:`pipeline.yml` plus one file per asset.\n- **Checks live next to the data.** A quality check declared on the column inside the asset definition is something an agent can read, add to, and use as a test. A check in a separate service is invisible to it.\n- **Lineage exists before the run.** If lineage is parsed from the SQL, the agent can answer \"what breaks if I change this column\" before deploying. If lineage only exists in a catalog after the fact, it cannot.\n\nA minimal Bruin asset has all three:\n\n```\n/* @bruin\nname: mart.daily_revenue\ntype: bq.sql\ndepends: [raw.orders]\nmaterialization:\n  type: table\ncolumns:\n  - name: order_date\n    checks:\n      - name: not_null\n  - name: revenue\n    checks:\n      - name: not_null\n      - name: positive\n@bruin */\n\nSELECT DATE(created_at) AS order_date, country, SUM(total) AS revenue\nFROM raw.orders\nGROUP BY 1, 2\n```\n\nThe header is the contract: what it produces, what it depends on, what must be true. Lineage is derived from the SQL body. That file is what the agent will write, edit and test.\n\n## [Step 1: Connect the agent](#step-1-connect-the-agent)\n\nBruin ships an MCP server as part of the CLI. Register it with your agent once:\n\n```\n# Claude Code\nclaude mcp add bruin -- bruin mcp\n```\n\nFor Cursor or Codex, add the same command to the editor's MCP configuration. The server exposes the Bruin CLI and its documentation, so the agent learns the commands rather than a fixed list of tools. From here on, the agent can run `bruin validate`, `bruin run`, `bruin query` and `bruin lineage` itself and read the results.\n\nThe dbt MCP server does the equivalent for a dbt project, scoped to the transformation layer. Databricks exposes Genie and Lakeflow to agents inside the Databricks workspace. Dagster Compass gives an agent read access to Dagster-orchestrated assets and metrics. Pick the one that matches where your pipeline definitions actually live.\n\n## [Step 2: Define the pipeline in natural language](#step-2-define-the-pipeline-in-natural-language)\n\nStart a new project and describe the job. A prompt that works:\n\nCreate a Bruin pipeline called `orders`. Ingest `public.orders` and `public.customers` from the Postgres connection `app-db` into the BigQuery dataset `raw`, incrementally on `updated_at`. Build `mart.daily_revenue` (revenue by day and country) and `mart.customer_ltv`. Add not-null and uniqueness checks on the keys, a positive check on revenue, and run the pipeline hourly.\n\nThe agent scaffolds the project. The ingestion asset it writes looks like this:\n\n```\nname: raw.orders\ntype: ingestr\nconnection: bigquery-default\nparameters:\n  source_connection: app-db\n  source_table: public.orders\n  destination: bigquery\n  incremental_strategy: merge\n  incremental_key: updated_at\n```\n\nThen the SQL assets with checks, a Python asset if something needs an API call or a model, and the schedule in `pipeline.yml`. Review the diff the way you would review a colleague's pull request: are the dependencies right, are the checks on the columns that matter, is the incremental key correct. Then:\n\n```\nbruin validate ./orders\nbruin run ./orders\n```\n\n`validate` parses every asset, checks the dependency graph and the lineage, and fails on anything inconsistent, so a hallucinated column name never reaches the warehouse.\n\n## [Step 3: Add monitoring the agent can read](#step-3-add-monitoring-the-agent-can-read)\n\nMonitoring for agents means two things: checks that fail loudly, and run history the agent can query.\n\n- **Freshness and completeness checks** on the raw tables: did the load happen, did it bring roughly the expected number of rows.\n- **Business checks** on the marts: revenue positive, no orphaned customer ids, accepted values on status columns.\n- **Run results in one place.** Bruin Cloud stores run history, check results and lineage together, and the same data is available to the agent through the CLI.\n\nOnce checks are declared, every `bruin run` is also a test run, and every failure is a structured event rather than a Slack message from a confused analyst.\n\n## [Step 4: Close the loop: detect, diagnose, fix, verify](#step-4-close-the-loop-detect-diagnose-fix-verify)\n\nThis is where \"self-healing\" becomes concrete. A worked example:\n\n1. **Detect.** The hourly run fails:`mart.daily_revenue` has 40 percent fewer rows than yesterday and the`not_null` check on`country` fails.\n2. **Diagnose.** The agent reads the lineage (`mart.daily_revenue` depends on`raw.orders` and`raw.customers` ), queries both raw tables, and finds that`raw.customers` stopped receiving rows two hours ago because the source added a new required column and the incremental load rejected it.\n3. **Fix.** It proposes two changes: allow schema evolution on the customers ingestion asset (`schema_contract: evolve` ) and a one-off backfill for the missing window. Both arrive as a diff.\n4. **Verify.** After approval, it runs`bruin run --start-date` for the backfill window, re-runs the checks, and reports that row counts and the null check are back to normal.\n5. **Learn.** The fix is a commit, so it is in the history the next time the same source changes.\n\nThe important design choice is where the human sits. Read-only diagnosis can be fully automatic. Fixes that change pipeline code should land as pull requests. Fixes that write to production data should require an explicit approval. Bruin's AI data team follows that split: it watches and diagnoses on its own, and proposes builds and repairs as changes you approve.\n\n## [Step 5: Let people ask the pipeline questions](#step-5-let-people-ask-the-pipeline-questions)\n\nThe same lineage and checks that make a pipeline repairable also make its output trustworthy enough to expose. Bruin's AI data analyst sits on the pipelines the agent built and answers questions in Slack, Microsoft Teams, Google Chat, WhatsApp, Discord, Telegram, email, or the browser, using the metric definitions in the models and showing the query it ran. That closes the loop from \"agent builds pipeline\" to \"business gets an answer\" without a BI tool in between.\n\n## [Which tools fit which setup](#which-tools-fit-which-setup)\n\n| Setup | Agent | What the agent can operate | Best for | \n|---|---|---|---|\n| Bruin + Claude Code, Cursor or Codex | Any MCP-capable coding agent | Ingestion, SQL and Python assets, checks, lineage, runs, queries, backfills | Teams that want one framework for the whole pipeline | \n| dbt + dbt MCP server | Any MCP-capable agent | Models, tests, docs, runs for the transformation layer | dbt shops with ingestion and orchestration elsewhere | \n| Databricks Genie and Lakeflow | Databricks-native | Lakehouse pipelines, notebooks, natural-language queries | Organisations fully on Databricks | \n| Dagster Compass | Dagster-native | Questions over Dagster assets and metrics | Dagster users who want an analyst, not a builder | \n| Snowflake Cortex | Snowflake-native | Natural-language queries, some pipeline automation | Snowflake-only teams | \n\nThe open-source, editor-agnostic option is Bruin. The vendor-native options are stronger if you never leave that vendor.\n\n## [Mistakes to avoid](#mistakes-to-avoid)\n\n- **Letting the agent write to production directly.** Every fix should be a diff you can read. Speed comes from fast review, not from skipping it.\n- **Skipping checks because the agent \"will notice.\"** Agents notice what checks tell them. No checks, no detection.\n- **Prompting against a blank project.** Give the agent a framework with conventions. The scaffold is what keeps its output consistent.\n- **Measuring success by lines generated.** Measure it by incidents resolved without a human reading logs at 2am.\n\n## [FAQ](#faq)\n\n### [How do I use an AI agent to build a data pipeline from scratch?](#how-do-i-use-an-ai-agent-to-build-a-data-pipeline-from-scratch)\n\nConnect the agent to a pipeline framework through MCP, describe the pipeline in one paragraph (sources, destination, models, checks, schedule), review the generated assets, run validation, then run the pipeline. With Bruin that is `claude mcp add bruin -- bruin mcp`, the prompt above, `bruin validate`, and `bruin run`.\n\n### [What are the best tools for agentic data engineering in 2026?](#what-are-the-best-tools-for-agentic-data-engineering-in-2026)\n\nBruin for an open-source framework that any coding agent can operate end to end; dbt with its MCP server for the transformation layer; Databricks Genie and Lakeflow inside Databricks; Dagster Compass for questions over Dagster assets. Claude Code, Cursor and Codex are the agents most teams use on top.\n\n### [What is the best AI copilot for data engineering?](#what-is-the-best-ai-copilot-for-data-engineering)\n\nA general coding agent connected to your pipeline tool beats a specialised copilot, because it can also run, test and query. Claude Code and Cursor with the Bruin or dbt MCP server are the common choices. Warehouse-native copilots (Genie, Cortex) are strongest when the whole stack lives in that warehouse.\n\n### [Can an AI agent build a pipeline with natural language only, no code?](#can-an-ai-agent-build-a-pipeline-with-natural-language-only-no-code)\n\nIt can write the code from natural language, and you should still read the code. The files are the contract that makes the pipeline testable and repairable later. Bruin keeps them short: a header with name, dependencies and checks, and the SQL or Python below it.\n\n### [What are the best tools for autonomous data pipeline monitoring and repair?](#what-are-the-best-tools-for-autonomous-data-pipeline-monitoring-and-repair)\n\nTools that combine checks, lineage and an agent that can act on both. Bruin's AI data team does detection, diagnosis and proposed repair on Bruin pipelines. Elementary and Monte Carlo do detection and diagnosis across warehouses but hand the repair back to you. Databricks Lakeflow has expectations and some automated remediation inside the lakehouse.\n\n### [Where should I start?](#where-should-i-start)\n\nWith one pipeline that already breaks sometimes. Move it into a framework with checks and lineage, connect an agent, and let it diagnose the next failure before you let it fix anything. Start at [github.com/bruin-data/bruin](https://github.com/bruin-data/bruin).", "url": "https://wpnews.pro/news/how-to-build-and-maintain-data-pipelines-with-ai-agents", "canonical_source": "https://getbruin.com/blog/how-to-build-data-pipelines-with-ai-agents/", "published_at": "2026-09-16 00:00:00+00:00", "updated_at": "2026-09-16 15:10:40.651778+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "developer-tools", "mlops"], "entities": ["Bruin", "Claude Code", "Cursor", "Codex", "MCP", "dbt MCP", "Databricks Genie", "Dagster Compass"], "alternates": {"html": "https://wpnews.pro/news/how-to-build-and-maintain-data-pipelines-with-ai-agents", "markdown": "https://wpnews.pro/news/how-to-build-and-maintain-data-pipelines-with-ai-agents.md", "text": "https://wpnews.pro/news/how-to-build-and-maintain-data-pipelines-with-ai-agents.txt", "jsonld": "https://wpnews.pro/news/how-to-build-and-maintain-data-pipelines-with-ai-agents.jsonld"}}