Drowning in clicks: Scaling Black Friday pipelines with Managed Airflow agents Google Cloud's Managed Service for Apache Airflow (formerly Cloud Composer) now offers agentic help across the pipeline lifecycle, with a Managed Airflow Agent in the Google Cloud console that investigates failed tasks, recommends environment improvements, and checks DAG readiness for newer Airflow versions, plus a Data Agent Kit that writes, migrates, and deploys DAGs in IDEs and coding agents. The features are illustrated through fictional retailer Cymbal Shops, whose data engineer uses Antigravity with the Data Agent Kit plugin and Gemini to generate a clickstream_hourly DAG for a Managed Airflow Gen 3 environment running Airflow 2.11.1, with workers at 2 vCPUs and 4 GB of memory autoscaling between 1 and 6. The agent checks the target environment's Airflow version before writing code, then deploys the DAG and confirms Airflow loads it without errors. When an Airflow pipeline fails in production, the hard part is rarely the fix. It's finding out what to fix. A task dies without a traceback. Is it the code, the data, or the environment? Hard to say, as clues are scattered across task logs, scheduler logs, and worker metrics. And once you know what the problem is, the question becomes whether to add memory, rewrite the DAG https://en.wikipedia.org/wiki/Directed acyclic graph , or finally take on the major version upgrade you've been putting off. Answering these questions takes time and deep Airflow expertise, usually while you're on call in the middle of an incident. AI agents can increasingly do much of this legwork. Managed Service for Apache Airflow formerly Cloud Composer now offers agentic help across a pipeline's entire lifecycle. In the Google Cloud console, Managed Airflow Agent https://docs.cloud.google.com/composer/docs/composer-3/agentic-capabilities-overview investigates failed tasks, recommends improvements to your environment, and checks whether your DAGs are ready for a newer Airflow version. In your IDE or coding agent, Data Agent Kit https://docs.cloud.google.com/data-agent-kit/overview writes, migrates, and deploys DAGs. In this post, we take you on a tour of these features through a story of a fictional company, Cymbal Shops. Amira is a data engineer at Cymbal Shops, an online retailer running cymbal-prod , a production Managed Airflow Gen 3 environment whose workers have 2 vCPUs, 4 GB of memory, and that autoscale between 1 and 6. Every click, search, and add-to-cart on Cymbal's website and mobile app goes to a Pub/Sub topic. A Cloud Storage subscription https://docs.cloud.google.com/pubsub/docs/cloudstorage writes the events as newline-delimited JSON files into hourly folders under gs://cymbal-clickstream-landing/events/ . It writes files in parallel and starts new ones every 5 minutes or every 128 MB, whichever comes first. Analysts, meanwhile, want these events in BigQuery, cleaned and with personal data hashed — and they want them fast. Ideally, each file would be processed the moment it lands. But Cymbal's Airflow environment runs Airflow 2.11.1, and Airflow 2 has no native way to start a DAG run when something happens outside Airflow. You can get close: Schedule a DAG every few minutes and let a sensor wait for new files, or build an extra service that triggers runs through the Airflow REST API. However, the team doesn't want all those moving parts, so they settle for the next best thing: an hourly batch. Amira writes her DAGs with an agent. She works in Antigravity https://antigravity.google/ with the Data Agent Kit plugin https://docs.cloud.google.com/data-agent-kit/overview , which adds Google Data Cloud skills and tools to the agent, and she uses Gemini as her model. Setup takes a few minutes: She installs the plugin, signs in with her Google account, and selects her project and region. In the agent panel, she describes what she needs with this prompt: "Write a DAG named clickstream hourly for the Managed Airflow environment cymbal-prod in us-central1. Every hour, load the previous hour of clickstream events into the BigQuery table clickstream.events. The events are newline-delimited JSON files in gs://cymbal-clickstream-landing/events/YYYY-MM-DD/HH/. Transform them with pandas: read each file into a DataFrame with the nested context object kept as a JSON string, combine the files with pd.concat, then drop malformed records, deduplicate on event id Pub/Sub delivers at least once , and hash user email and ip address. Re-running an hour must not create duplicates. If a run fails, email data-oncall@example.com. Validate the DAG and deploy it." The agent checks the target environment before it writes any code. Because cymbal-prod runs Airflow 2.11.1, the agent produces the code aligned to that version. It then deploys the DAG to cymbal-prod and confirms that Airflow loads it without errors. Amira reviews the code, and the DAG goes live in early August. It's good code: readable, idempotent, and correct for Airflow 2.11. A busy hour sees about 200 files and 350 MB, and the task peaks below 2 GB — well under the 4 GB worker limit. Look at how the task builds its input, though. It lists every file in the hour's folder, reads each one into a DataFrame, and then pd.concat combines them into a single DataFrame. The unit of work is a clock hour, so the task needs more memory as traffic grows. That's fine until traffic changes... Late October brings two changes in the same week. Cymbal launches its "Fall Into Savings" campaign, and a new mobile app release adds much more detailed event tracking. Events from the new app version are about three times larger than their predecessors, and the campaign brings more traffic on top, so an hour of clicks grows from about 350 MB to about 1.4 GB. Holding a whole hour in pandas takes several times its raw size, so load previous hour clickstream now needs more memory than a worker’s allotted 4 GB. Autoscaling doesn't help here: It adds workers when tasks queue up, but each task still runs on a single worker and can't use more memory than that worker has. On October 23, the 14:00 UTC run of clickstream hourly fails. All three attempts die the same way, with no error in the task log. Amira opens cymbal-prod in the console, where Managed Airflow Agent is built in. It diagnoses the problem and recommends a fix; Amira decides what to change. She goes to the DAGs tab, selects clickstream hourly , and opens Run history. Next to the failed 14:00 run there's an Investigate button with a sparkle icon. She clicks it, and a short ‘Investigate DAG run failure’ dialog asks her to confirm her choice. The Ask agent panel then opens next to the page, and the agent starts investigating. At the end of the report, the agent recommends three fixes: load the files straight into BigQuery, process them in batches, or give the workers more memory. Look at what the agent delivers Every conclusion includes the evidence that led to it, so Amira can check the reasoning herself. Two of the recommendations are code changes, and code changes take time to write and test. With every hourly run now failing, Amira needs the quick fix first: more memory for the workers. But how much? The right number depends on the whole environment, not on one task. So she asks the agent in the same panel: "clickstream hourly runs out of memory on 4 GB workers. Analyze the cymbal-prod environment and recommend how much worker memory it needs, along with any other improvements." The agent recommends increasing worker memory from 4 GB to 8 GB, and flags three production-readiness gaps: Database retention is disabled, the environment isn't highly resilient, and it has no scheduled snapshots. Amira acts on the first recommendation right away. The agent is read-only, so she makes the change herself and increases worker memory to 8 GB. She schedules the other two recommendations for the next change window. With 8 GB workers, the pipeline gets through the rest of the campaign. But that only buys time. Black Friday traffic is forecast at 8 to 10 times a normal peak hour, and at that volume the hourly batch will need well over 8 GB. The troubleshooting report had already named the real problem: a unit of work that grows with traffic. That brings Amira back to what the team wanted in the first place, which was to process each file as soon as it lands. Airflow 3 makes that much easier, because it supports event-driven scheduling natively. An asset watcher listens to an external source, such as a Pub/Sub subscription. When new messages arrive, it updates an asset, and any DAG scheduled on that asset starts a run. Under the hood, the watcher still checks the subscription every few seconds, but it does so inside Airflow's triggerer. No DAG needs to be scheduled just so a sensor can wait for files, and no extra service has to trigger runs. Each run gets the new messages in its payload, so it can work on just the files that landed, one task per file, and memory per task depends on file size, not on how busy the hour is. A major version upgrade of a production environment isn't something to do blind. Before anyone touches the code, Amira asks the agent in the same panel: "Check whether cymbal-prod is ready to upgrade to Airflow 3." Managed Airflow Agent can also check upgrade readiness. It checks compatibility of the environment's DAGs with a given version. Amira is checking what problems she might have while migrating to Airflow 3.3.1 composer-3-airflow-3.3.1-build.3 . The agent reports the following observations: Two findings are blockers. The first comes as a surprise: a maintenance DAG that cleans up the Airflow database by hand, which Airflow 3 no longer allows. The agent's fix is to retire it and turn on database retention, the same feature it recommended during the incident. The second, marketing attribution , still uses execution date , which Airflow 3 removed. Four DAGs also need schedule interval renamed to schedule . In a few minutes, the agent estimated the work to be done. To perform the upgrade, the report suggests using Data Agent Kit's migrations skill. Rather than upgrading in place, Amira creates a new Airflow 3 environment alongside cymbal-prod to move the DAGs there. Back in Antigravity, Data Agent Kit's migrations skill takes over. It adapts existing DAGs to the target Airflow version, deploys them, and verifies them. Finally, the agent deploys the DAGs to cymbal-prod-af3 and confirms that they load without errors. Two design choices remove the original problem: Scheduling moves into the watcher. PubsubPullTrigger is an event trigger, so an AssetWatcher can use it directly. The triggerer checks the subscription every 10 seconds in the background. Each batch of messages becomes an asset event that starts a run, with the messages in its payload. Nothing waits for the top of the hour, and no extra service has to call the Airflow REST API. Each task processes one file rather than a full hour. Dynamic task mapping gives each new file its own task, and the subscription caps files at 128 MB. A busier hour means more tasks, not bigger ones. Once the new DAG proves itself on live traffic, Amira pauses the hourly DAG in cymbal-prod . Success In a load test that replayed eight times normal peak traffic, clickstream ingest simply ran more often. Memory per task didn't change. | Dimension | Before: Airflow 2, hourly batch | After: Airflow 3, event-driven | | Unit of work | One clock hour: about 350 MB normally, 1.4 GB at campaign peak | One file, a few MB each at most 128 MB | | Peak memory per task | Grows with traffic: about 5 to 6 GB at campaign peak, well over 8 GB forecast for Black Friday | Flat, well under 1 GB | | Time from click to BigQuery | Over an hour | A few minutes | Each agent works from the right vantage point. The console agent inspects production telemetry with your permissions and doesn't change anything, while the coding agent works in the IDE where code changes get reviewed, tested, and deployed. Each step sets up the next one. The troubleshooting report offered a wider review, and each recommendation came with actionable advice. The upgrade check ended with next steps for the migrations skill, turning its report into the brief for the migration. Agents propose and people decide. Amira scaled the workers, ran the infrastructure commands, and reviewed every change. Static inspection estimates the work before live migration. Checking compatibility in the console scoped the effort in minutes, and the migration agent then made and verified the changes in a real environment. Amira used the agents built into the console and her IDE. Two other capabilities didn't fit her story but are worth knowing about: You can bring your own agent with the Managed Airflow MCP server https://docs.cloud.google.com/composer/docs/composer-3/use-composer-mcp , now generally available, so any MCP-compatible agent can manage your environments, allowing you to list and trigger DAGs, check run status, and inspect failed task instances. Access is controlled through IAM, and read-only tools are flagged so you can limit agents to them. Run agentic steps inside your pipelines with the Apache Airflow Common AI provider https://airflow.apache.org/docs/apache-airflow-providers-common-ai/stable/index.html , which adds model-driven tasks that run side by side with deterministic ones in the same Airflow 3 DAG. Examples include LLM-based branching https://airflow.apache.org/docs/apache-airflow-providers-common-ai/stable/operators/llm branch.html , agentic retries https://airflow.apache.org/docs/apache-airflow-providers-common-ai/stable/retry policies.html that tell a transient error from a bad input, and schema drift detection https://airflow.apache.org/docs/apache-airflow-providers-common-ai/stable/operators/llm schema compare.html . Explore Managed Airflow Agent troubleshooting https://docs.cloud.google.com/composer/docs/composer-3/troubleshoot-tasks-and-dag-runs-with-agent and the overview of agentic capabilities https://docs.cloud.google.com/composer/docs/composer-3/agentic-capabilities-overview . Install Google Cloud Data Agent Kit https://cloud.google.com/products/data-agent-kit using the documentation https://docs.cloud.google.com/data-agent-kit , the extension for VS Code https://marketplace.visualstudio.com/items?itemName=GoogleCloudTools.datacloud , the plugin for coding agents https://github.com/GoogleCloudPlatform/data-agent-kit-plugin Antigravity, Claude Code, Gemini CLI, and Codex CLI , or the open-source Airflow agent skills https://github.com/google/skills . Learn about Airflow 3 event-driven scheduling https://airflow.apache.org/docs/apache-airflow/stable/authoring-and-scheduling/event-scheduling.html . Connect your agents to the Managed Airflow MCP server https://docs.cloud.google.com/composer/docs/composer-3/use-composer-mcp . Build model-driven tasks and agents in DAGs with the Apache Airflow Common AI provider https://airflow.apache.org/blog/common-ai-provider-1.0/ . Special thanks to Jacek Iżykowski, Software Engineer on Managed Service for Apache Airflow, for authoring the original Cymbal Shops pipeline story and agent workflows featured in this post. Note: As a fictional company, any agent outputs from Cymbal Shops are shortened and illustrative, as are traffic and memory numbers.