{"slug": "airflow-ai-provider-1-0-agents-sandboxes-and-human-approval-in-production-dags", "title": "Airflow AI Provider 1.0: Agents, Sandboxes, and Human Approval in Production DAGs", "summary": "Apache Airflow released AI Provider 1.0, adding agent orchestration primitives to its workflow engine as DAG tasks rather than a standalone agent framework. The provider, which installs on Airflow 2.11 and later, introduces agent tasks with sandboxed code execution, human-in-the-loop approval gates on Airflow 3.3+, unified model connections for OpenAI, Anthropic, Google and Bedrock via a 'pydanticai' connection type, and retry modes that distinguish transient model failures from invalid tool calls. The sandbox is a restricted Python subprocess rather than a container or VM, so teams needing stronger isolation are advised to wrap agent tasks in a KubernetesPodOperator.", "body_md": "Apache Airflow just shipped AI Provider 1.0, bringing agent orchestration primitives to a workflow engine already running in thousands of production data pipelines. This is not a new agent framework. It is a set of DAG tasks that let you run LLM agents inside existing Airflow workflows, with sandboxing, human approval gates, and retry logic that understands when a model fails versus when a tool call fails.\n\nThe provider installs on Airflow 2.11 and later, so teams still on Airflow 2 can start using agents without upgrading the entire platform. The core primitives are agent tasks, sandboxed code execution, and human-in-the-loop approval gates that integrate with Airflow's XCom system for passing context between tasks.\n\nThe Common AI provider (`apache-airflow-providers-common-ai`) adds three main capabilities to Airflow DAGs:\n\nAn agent task is a model working in a loop. It calls tools (functions the DAG author allows), reads results, and picks the next step until it has an answer. The provider wraps this loop in Airflow's TaskFlow API, so it behaves like any other DAG task with retries, logging, and XCom data passing.\n\nThe provider supports OpenAI, Anthropic, Google, and Bedrock models through a unified connection type called `pydanticai`. You configure the model and API key in an Airflow connection, and the agent task reads it at runtime.\n\nAirflow DAGs are directed acyclic graphs where each node is a task. The AI Provider adds agent tasks that fit into this topology like any other task. The difference is what happens inside the task.\n\nA traditional Airflow task runs a function once. An agent task runs a loop:\n\nThe agent task returns its final answer through XCom, so downstream tasks can use it. If the agent fails (model timeout, invalid tool call, sandbox error), Airflow's retry logic kicks in. The provider lets you configure whether retries happen automatically or only after a model decides the failure is transient.\n\nWhen an agent writes code, the provider runs it in a sandbox instead of the Airflow worker process. This prevents agent-generated code from:\n\nThe sandbox is a separate process with limited filesystem access and no access to Airflow's internal state. The agent task passes code to the sandbox, waits for the result, and returns it to the model. If the sandbox crashes or times out, the agent task fails and Airflow retries it.\n\nThe sandbox does not use containers or VMs. It is a Python subprocess with restricted imports and filesystem access. This keeps overhead low but means you need to trust the sandbox implementation to enforce boundaries. For higher isolation, you would wrap the agent task in a KubernetesPodOperator or similar.\n\nOn Airflow 3.3 and later, agent tasks can pause and wait for human approval before executing a tool call. This is useful when the agent wants to post to Slack, send an email, or run a write query.\n\nThe approval gate works through Airflow's existing approval mechanism. When the agent calls a tool marked as requiring approval, the task pauses and creates an approval request. A person reviews the request in the Airflow UI, approves or rejects it, and the agent task resumes.\n\nThe approval request includes:\n\nIf the person rejects the tool call, the agent task fails and Airflow retries it (or not, depending on retry configuration). If approved, the tool executes and the result flows back to the model.\n\nThis integrates with Airflow's XCom system. The approval request is an XCom value that downstream tasks can read. The approval decision is another XCom value that the agent task reads to continue or fail.\n\nAirflow retries failed tasks automatically, but agent tasks need smarter retry logic. A model timeout is transient. An invalid SQL query is not.\n\nThe provider adds two retry modes:\n\n**Automatic retries** treat every failure as transient. If the model times out, the tool call fails, or the sandbox crashes, Airflow retries the entire agent task. This works for flaky APIs or rate limits but wastes tokens on persistent errors.\n\n**Model-decided retries** let the model decide whether to retry. When a tool call fails, the error message flows back to the model. The model can:\n\nThis mode requires the model to understand error messages and make retry decisions. It saves tokens on persistent errors but adds latency because the model has to process the error and decide.\n\nYou configure retry mode per agent task. The default is automatic retries with a max retry count.\n\nThe provider exposes metrics for each agent task:\n\nThese metrics flow into Airflow's existing observability stack. You can query them through Airflow's REST API, export them to Prometheus, or view them in the task logs.\n\nThe provider also logs every tool call with arguments and results. This is useful for debugging why an agent made a particular decision or why a tool call failed. The logs include:\n\nFor token usage tracking, the provider reads token counts from the model API response. If the model does not expose token counts, the provider estimates them using a tokenizer.\n\nHere is an agent task that queries a warehouse to explain a revenue drop:\n\n``` python\nfrom airflow.decorators import dag, task\nfrom datetime import datetime\nfrom airflow.providers.common.ai.decorators import agent_task\n\n@dag(start_date=datetime(2026, 10, 1), schedule=None)\ndef revenue_analysis():\n\n    @agent_task(\n        conn_id=\"pydanticai_default\",\n        tools=[\"sql_query\"],\n        sandbox=True,\n        require_approval=False\n    )\n    def explain_revenue_drop(context):\n        \"\"\"\n        Query the warehouse to explain why revenue dropped 15% last week.\n        You have read-only access to the sales and customer tables.\n        \"\"\"\n        return {\n            \"prompt\": \"Revenue dropped 15% last week. Query the warehouse to find the cause.\",\n            \"max_iterations\": 10\n        }\n\n    @task\n    def post_to_slack(explanation: str):\n        # Post the explanation to Slack\n        pass\n\n    result = explain_revenue_drop()\n    post_to_slack(result)\n\nrevenue_analysis()\n```\n\nThe `agent_task` decorator wraps the function in an agent loop. The `tools` parameter lists available tools (in this case, a SQL query tool). The `sandbox=True` parameter runs agent-generated code in a sandbox. The `require_approval=False` parameter allows the agent to execute tool calls without human approval.\n\nThe agent task returns a dictionary with the prompt and max iterations. The provider runs the agent loop, calls tools as needed, and returns the final answer through XCom. The downstream `post_to_slack` task receives the explanation and posts it.\n\n| Dimension | Airflow AI Provider | Standalone Agent Framework | \n|---|---|---|\n| **Integration** | Runs inside existing DAGs, shares connections and secrets | Requires separate deployment and credential management | \n| **Retry logic** | Uses Airflow's retry system, can retry entire DAG branches | Custom retry logic per framework | \n| **Observability** | Airflow UI, logs, and metrics | Framework-specific observability | \n| **Sandboxing** | Subprocess isolation, low overhead | Varies by framework (containers, VMs, none) | \n| **Human approval** | Built into Airflow UI, integrates with XCom | Custom approval UI or API | \n| **Latency overhead** | DAG scheduling adds 1-5 seconds per task | Direct execution, no scheduling overhead | \n| **State management** | XCom for passing data between tasks | Framework-specific state stores | \n\nAirflow adds scheduling overhead. If your agent workflow is latency-sensitive (sub-second responses), Airflow is not the right tool. If your agent workflow runs on a schedule or as part of a larger data pipeline, Airflow makes sense.\n\nThe sandbox isolation is weaker than containers or VMs. If you need strong isolation (untrusted code, compliance requirements), wrap the agent task in a KubernetesPodOperator or run it in a separate cluster.\n\nHuman approval gates are useful for high-risk actions (posting to Slack, running write queries) but add latency. If your agent workflow is fully automated, disable approval gates.\n\n**Model timeouts**: The provider sets a default timeout of 60 seconds per model call. If the model does not respond, the task fails and Airflow retries it. You can increase the timeout in the agent task configuration.\n\n**Invalid tool calls**: If the model calls a tool with invalid arguments, the tool returns an error. In automatic retry mode, Airflow retries the entire task. In model-decided retry mode, the error flows back to the model, which can retry with different arguments.\n\n**Sandbox crashes**: If the sandbox process crashes, the agent task fails. Airflow retries it. The sandbox does not persist state between retries, so the agent starts from scratch.\n\n**Approval timeouts**: If a human does not approve a tool call within the timeout (default 1 hour), the agent task fails. You can configure the timeout per task.\n\n**Token limit exhaustion**: If the agent loop uses too many tokens, the model API returns an error. The provider does not track cumulative token usage across retries, so you need to monitor this separately.\n\nThe provider installs on Airflow 2.11, but some features require Airflow 3.3:\n\nOn Airflow 2.11, change imports from `airflow.decorators` to `airflow.operators.python`. The `agent_task` decorator is the same, but the import path is different.\n\n**Use Airflow AI Provider when:**\n\n**Avoid it when:**\n\nThe provider is useful for teams that already run Airflow and want to add agent tasks without deploying a separate agent framework. It is not a replacement for standalone agent frameworks like LangGraph or CrewAI, but it fills a gap for teams that want to run agents inside existing workflow orchestration.", "url": "https://wpnews.pro/news/airflow-ai-provider-1-0-agents-sandboxes-and-human-approval-in-production-dags", "canonical_source": "https://dev.to/mech_app_ai/airflow-ai-provider-10-agents-sandboxes-and-human-approval-in-production-dags-gb", "published_at": "2026-10-21 00:09:03+00:00", "updated_at": "2026-10-21 00:19:40.215876+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "developer-tools", "mlops", "ai-infrastructure"], "entities": ["Apache Airflow", "OpenAI", "Anthropic", "Google", "Amazon Bedrock", "KubernetesPodOperator", "pydanticai"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/airflow-ai-provider-1-0-agents-sandboxes-and-human-approval-in-production-dags", "markdown": "https://wpnews.pro/news/airflow-ai-provider-1-0-agents-sandboxes-and-human-approval-in-production-dags.md", "text": "https://wpnews.pro/news/airflow-ai-provider-1-0-agents-sandboxes-and-human-approval-in-production-dags.txt", "jsonld": "https://wpnews.pro/news/airflow-ai-provider-1-0-agents-sandboxes-and-human-approval-in-production-dags.jsonld"}}