{"slug": "building-an-agent-that-turns-datahub-metadata-into-airflow-dags", "title": "Building an Agent That Turns DataHub Metadata Into Airflow DAGs", "summary": "A developer and their teammate built a tool for DataHub's 'Build with DataHub: The Agent Hackathon' that generates Airflow DAGs from DataHub metadata. The tool walks lineage graphs, reads tags and glossary terms, and maps them to tasks, with an LLM producing a structured plan that a deterministic renderer converts into code. It supports agent and script modes, with the LLM never writing code directly to avoid arbitrary code in production.", "body_md": "Built this with my teammate for DataHub's \"Build with DataHub: The Agent Hackathon.\" The hackathon has wrapped up, but we're continuing to develop it.\n\n**The problem**\n\nIf you're using DataHub, you already have lineage, freshness SLAs, PII tags, and glossary terms sitting there. But turning that into an actual Airflow DAG still means doing it by hand.\n\n**What it does**\n\nGive it a table name, and it:\n\nWalks the lineage graph backwards to find every upstream dependency\n\nReads tags (pii, daily_refresh) and glossary terms (FreshnessSLA, EmptyLoad) on each table\n\nMaps signals to tasks, pii → audit task, FreshnessSLA → freshness check\n\nTopologically sorts and reduces the graph, then renders a clean, deterministic Airflow 3 DAG\n\nCan optionally open a GitHub PR with a lineage summary\n\n**How it's built**\n\nTwo modes: an agent mode, where an LLM (Claude, or any model via OpenRouter) explores the DataHub graph through the MCP protocol and produces a plan, and a script mode that skips the LLM entirely for CI use.\n\nThe design decision I'd love feedback on: the LLM never writes code directly. It only outputs a structured plan, and a separate deterministic renderer turns that into the actual Airflow task code. Felt like the right boundary between \"useful agent\" and \"don't let an LLM generate arbitrary code that ends up scheduled in prod.\"\n\n**Still rough around the edges**\n\nNo dbt or Spark operator support yet (just shell task stubs), no schema drift detection, and it's mostly only been tested against the DataHub NYC Taxi sample dataset so far.\n\n**Try it**\n\nGitHub: [github.com/tdm291104/datahub-dag-generator](//github.com/tdm291104/datahub-dag-generator)\n\nDevpost: [devpost.com/software/datahub-dag-generator](//devpost.com/software/datahub-dag-generator)\n\nDemo: [youtube.com/watch?v=2f0uNNmSBfc](//youtube.com/watch?v=2f0uNNmSBfc)\n\nFeedback and contributions welcome, and if you find it useful, a star on the repo goes a long way!!!", "url": "https://wpnews.pro/news/building-an-agent-that-turns-datahub-metadata-into-airflow-dags", "canonical_source": "https://dev.to/mark2k4/building-an-agent-that-turns-datahub-metadata-into-airflow-dags-gkk", "published_at": "2026-08-26 07:36:11+00:00", "updated_at": "2026-08-26 07:43:41.104897+00:00", "lang": "en", "topics": ["artificial-intelligence", "developer-tools", "mlops"], "entities": ["DataHub", "Airflow", "Claude", "OpenRouter", "GitHub", "Devpost", "YouTube"], "alternates": {"html": "https://wpnews.pro/news/building-an-agent-that-turns-datahub-metadata-into-airflow-dags", "markdown": "https://wpnews.pro/news/building-an-agent-that-turns-datahub-metadata-into-airflow-dags.md", "text": "https://wpnews.pro/news/building-an-agent-that-turns-datahub-metadata-into-airflow-dags.txt", "jsonld": "https://wpnews.pro/news/building-an-agent-that-turns-datahub-metadata-into-airflow-dags.jsonld"}}