An AI agent for generating data pipelines from Slack requests

GoDaddy's domain data platform team built Pipeline Build Agent, a conversational AI agent that turns a Slack message into a complete data pipeline — PySpark transformation, Airflow DAG, dev/stage/prod policy files, Lake Naming Standards checks and a pull request — cutting a task that took 2–4 hours to minutes, the team reported for its 2026 Hackathon. The agent runs on GoDaddy's internal go-agent-python SDK, is deployed on the company's Katana continuous delivery platform, and uses a manager agent that routes requests to a Pipeline Creation Agent or a Column Addition Agent, with both specialists querying GoKnowb, GoDaddy's internal AI knowledge base, for naming standards. The team chose an agent over a web form or CLI script because pipeline creation requires clarifying questions and automatic S3 schema inference, and wrapped it in a FastAPI service to expose Slack, REST API, Swagger UI and GitHub Actions entry points.

Background On our domain data platform team, bringing a new data source into the data lake means writing a PySpark transformation, wiring up an Airflow DAG, generating policy files for dev, stage, and prod, checking everything against our Lake Naming Standards, and opening a PR. It's the same sequence every time. For our 2026 Hackathon, we built Pipeline Build Agent — a conversational AI agent that does all of that from a single Slack message, taking a task that used to take hours and reducing it to minutes. The cost of repetition Every time a product team needed logs from a new S3 bucket in the data lake, an engineer had to sit down and manually walk through the same steps: open the templates, infer the schema, look up naming conventions, fill in the right bucket paths and table names, generate the files, and open a PR. The work wasn't hard — it was just time-consuming and easy to get wrong. The frustrating part was that we already had everything we needed to do it automatically. We had a Lake Naming Standards knowledge base, a well-defined file structure, and templates for every artifact. The knowledge existed; it just wasn't being applied without a human in the loop. That gap — between "we know exactly how to do this" and "it still takes 2–4 hours each time" — was the problem we wanted to solve. We also had a secondary problem: compliance. Our naming standards aren't optional. Pipelines that don't follow them fail review. Engineers who were unfamiliar with the standards or in a hurry would sometimes generate files that needed to be fixed before the PR could merge, adding more back-and-forth. Why we chose an AI agent Our first instinct was to build a script or CLI wizard that walked engineers through the required inputs. But that approach hits a wall quickly: pipeline creation isn't a form you fill out — it's a conversation. You might not know the output table name upfront, or whether to include all fields, or what to name the pipeline. You figure those things out by talking through them. We evaluated a few options: - A web form — forces all inputs upfront, doesn't handle optional or derived fields well - A CLI script — automates generation but still needs a human to drive every step - An AI agent — can ask clarifying questions, infer schema from S3 automatically, and handle the full workflow end-to-end The agent approach also let us meet engineers where they already are. Our team lives in Slack. Wrapping the agent in a FastAPI service gave us Slack, REST API, Swagger UI, and GitHub Actions as entry points — all with the same underlying conversation model. We built the system on our internal go-agent-python SDK and deployed it on Katana GoDaddy's internal continuous delivery platform . How it works The agent uses a multi-agent architecture where a "manager agent" classifies intent and routes to one of two "specialist agents". User Slack / REST / Swagger / GHA ↓ FastAPI runner.py ↓ Manager Agent ──→ Pipeline Creation Agent ──→ Column Addition Agent The Pipeline Creation Agent handles new pipelines. It checks S3 schemas, generates all required files, and opens a PR. The Column Addition Agent handles modifications. It finds the relevant pipeline, propagates the change to downstream tables, and opens a PR. Each specialist has a focused set of tools: | Agent | Key tools | |---|---| | Pipeline Creation | check s3 schema , create pyspark job , create dag , create policy files , create pull request | | Column Addition | find pipeline by table , get related tables , read pipeline script , write modified pipeline script , create pull request | Both specialists also have access to GoKnowb GoDaddy's internal AI knowledge base , which queries our Lake Naming Standards knowledge base. The knowledge base-first pattern Early on we ran into a naming compliance problem. The agent would generate code, then check it against our standards, find violations, and need to rewrite and re-push. This produced messy PRs with multiple corrective commits. The fix was to flip the order. Every specialist now queries the naming standards knowledge base before generating any code. The standards land in the LLM's context first, so pipeline names, table names, database names, and file paths are compliant from the first generation. The update pr files tool exists as a safety net but is rarely needed. A full session The following conversation illustrates what a pipeline creation conversation looks like end-to-end: 👤 User: I need logs from s3://example-data/raw-logs/app-logs/ processed 🤖 Agent: ✅ Found schema: • timestamp datetime • user id string • event type string • value integer Do you need all fields or want to exclude any? 👤 User: All fields 🤖 Agent: What should we name this pipeline? Suggestion: app logs pipeline 👤 User: app logs pipeline 🤖 Agent: ✅ Created: - PySpark job: myteam/app logs pipeline/src/pyspark/app logs pipeline job.py - DAG: myteam/app logs pipeline/app logs pipeline.py - Policy files dev/stage/prod + pipeline Create a pull request? 👤 User: yes 🤖 Agent: ✅ PR ready: https://github.com/my-repo/demo-piplines/pull/123 Six files committed, one PR opened, naming-standards compliant. The generated file structure looks like this: myteam/<pipeline name / ├── <pipeline name .py Airflow DAG └── src/ ├── pyspark/ │ └── <pipeline name job.py PySpark transformation └── policies/ ├── environment.dev private.yaml ├── environment.stage.yaml ├── environment.prod.yaml └── <pipeline name .yaml The PySpark job reads from S3, writes to the Glue catalog with Hive-style partitioning snap date=YYYY-MM-DD , and integrates with Switchboard. The policy files are pre-populated with the correct AWS account numbers per environment. Conversation state is managed via a conversation id so the agent can reference earlier answers like the pipeline name without asking again. What we got wrong first Keeping the PR clean was harder than expected. Without a strict rule, the agent would sometimes call create pull request more than once per session, opening duplicate PRs. We solved this by making "one PR per run" a hard invariant. create pull request is called exactly once, and update pr files is the only path for any corrections. That single constraint cleaned up most of the edge cases. Results and what's next Since deploying on Katana, our team has been using the agent to handle routine pipeline requests through Slack. What used to take a few hours of manual work including looking up standards and wiring up templates now takes a few minutes of conversation. We learned a few general lessons from this project: - Treat internal standards as context to inject early, not guardrails to check at the end. - Separate agents with narrow responsibilities are easier to maintain and debug than one large agent trying to handle everything. - For tasks with optional or derived inputs, a conversational interface is a better fit than a form. We're planning to add Redshift and S3 CSV export as pipeline destinations next, along with pre-PR syntax validation for generated DAGs and PySpark code. If you're interested in working on problems like this, we're hiring — check out our open engineering roles at GoDaddy Careers https://careers.godaddy.com/ .