{"slug": "re-engineering-apache-airflow-for-speed-and-scale", "title": "Re-engineering Apache Airflow for speed and scale", "summary": "Astronomer's Astro Runtime sustained 500,000 concurrent Apache Airflow tasks in a single deployment and achieved 228 milliseconds p95 task-start latency at 100,000 concurrent tasks, compared to 23.582 seconds for open-source Airflow at half that load. The company re-engineered Airflow's scheduling, execution, scaling, and recovery systems while maintaining compatibility with Airflow 3+ releases.", "body_md": "# The Astro Runtime: Airflow re-engineered for speed and scale\n\n36 min read |\n\nApache Airflow is the open-source standard for defining and running data workflows. Astronomer builds Astro, a managed Airflow platform for teams running those workflows in production.\n\nAirflow is no longer used only for scheduled batch pipelines. Teams now use it to coordinate thousands of data and AI workflows across an enterprise: loading warehouse tables, building dbt models, preparing training data, running model evaluations, and reacting to external events. These workflows can release tens of thousands of tasks at once, keep hundreds of thousands running, and still require a newly ready task to start in hundreds of milliseconds.\n\nOur customers hit ceilings in how fast Airflow starts a task and how many it keeps running, so we rebuilt the path behind both.\n\nOver the past several years, we have rebuilt Airflow's scheduling, execution, scaling, and recovery systems on Astro to perform beyond any current demands. And we did it in a way where Astro still works seamlessly with Airflow 3+ releases, meaning nothing needs to change with how pipelines are defined.\n\nIn our load tests, Astro sustained **500,000 concurrent Airflow tasks** in a\nsingle deployment and reached **228 milliseconds p95 task-start latency at 100,000 concurrent tasks**—less than one-hundredth of the 23.582 seconds we measured when we pushed\nopen-source Airflow to just half that load. At 300,000 concurrent tasks, p95 remained 294 milliseconds.\n\nWe made these changes without replacing Airflow's workflow model. User Dags, task dependencies, task execution status, and operator code remain compatible with open-source Airflow. Astro just changes components under the hood to make task execution faster and more scalable while improving reliability:\n\nCoordination\n\nExecution— one executor per deployment\n\nPlatform\n\n*Heavy borders mark the systems Astronomer built; the rest is Apache\nAirflow. This is the control path around a task, not every service in a\ndeployment.*\n\n## How Airflow got here\n\nAirflow began 10 years ago with a clear job: read workflows defined as Python code, decide which tasks were ready, and send them somewhere to run. A scheduler made those decisions, an executor handled the delivery, and a SQL database recorded Dag runs and task states.\n\nThat design gave Airflow two useful properties. Dag authors could choose how tasks ran without changing their workflow code, and a failed scheduler could rebuild its view from the database. The database, rather than any scheduler process, held the lasting account of the work.\n\nAs deployments added more Dags and tasks, the scheduler had to parse more Python files, create more Dag runs, check more dependencies, and queue more task instances. One scheduler also left the whole deployment dependent on one process. Airflow 2.0 and beyond addressed those limits with a few large changes, without replacing the database model.\n\n[AIP-15](https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/103092651/AIP-15%2BSupport%2BMultiple-Schedulers%2Bfor%2BHA%2BBetter%2BScheduling%2BPerformance)\nlet several scheduler replicas run at once. Each could examine Dag runs in\nparallel, then use database row locks when moving task instances from\n`scheduled`\n\ninto an executor. Airflow also changed to store\n[serialized Dags](https://airflow.apache.org/docs/apache-airflow/2.3.3/dag-serialization.html)\nin the database, so a scheduler could make decisions from a stored graph\ninstead of importing every Python Dag file itself.\n\nAirflow 2.2 introduced ways to offload tasks that were waiting for some\nexternal system via the triggerer. Consider a daily ingestion Dag that cannot\nstart until a partner uploads `complete.json`\n\nto an object-storage bucket. A\nsensor may spend hours checking for that file while doing almost no work, yet it\ncan still occupy a worker slot. A\n[deferrable operator](https://airflow.apache.org/docs/apache-airflow/2.2.2/authoring-and-scheduling/deferring.html)\nstores what it needs to resume, releases the worker, and hands the wait to a\ntriggerer. When the file arrives, the trigger fires and the scheduler checks\nthe task again before it can run.\n\nAirflow 2.3 then introduced the separate Dag parser component, which separated out user code from the core scheduler component and loop. This was much better from a security/isolation perspective and also meant that Dag processing could scale separately from scheduling.\n\nAirflow 3 gave task execution a clear API contract. Under\n[AIP-72](https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/311626182/AIP-72%2BTask%2BExecution%2BInterface%2Baka%2BTask%2BSDK),\ntask processes no longer need direct access to the metadata database like they\ndid in Airflow 2.x. The Task SDK and Execution API define what a task receives,\nincluding its identity, connections, and variables, and what it reports,\nincluding heartbeats, state changes, and XComs (used for cross-task\ncommunication). Workers and task code use that interface instead of reading and\nupdating Airflow's internal tables.\n\nModern Airflow divides the original system's work among Dag processing, active-active schedulers, triggerers, executors, API servers, and workers. This is the architecture we helped build in Airflow: durable database state, active-active scheduling, deferred execution, and a clear task API. At Astronomer, we led the HA scheduler, the triggerer and deferred tasks, and the Task SDK and Execution API, and we contributed to moving Dag processing out of the scheduler loop. As deployments grew and workflows carried more weight, we also saw where the task-start path needed a different design.\n\n## The lifecycle of an Airflow task\n\nHere is what happens after one task finishes, first in open-source Airflow and then in Astro. The Dag is a basic three-task ETL.\n\nIn our example Dag, when `extract`\n\nsucceeds, Airflow records that result in the\nmetadata database. The scheduler must notice the change, confirm that\n`transform`\n\nmay run, check shared limits, queue it through the configured\nexecutor, and wait for a worker to start its process. Only then does `transform`\n\nbegin running.\n\nThese steps run in order. `transform`\n\ncannot skip any of them, so each adds its\nown wait to the total.\n\n### 1. The upstream task finishes\n\nThe `dag_run`\n\nstate for `example_etl`\n\nis running, the `task_instance`\n\nstate for\n`extract`\n\nis `success`\n\n, and the rows for `transform`\n\nand `publish`\n\nhave a `NULL`\n\nstate. Airflow reads the dependency from the serialized Dag and evaluates it\nagainst the task states for this run on a loop.\n\nThe lasting record of `transform`\n\nis its `task_instance`\n\nrow, not an in-memory\nobject passed from one process to another. Different parts of Airflow update\nthat row as they make decisions about it.\n\n### 2. A scheduling pass must discover the change\n\nThe standard scheduler works in batches. It queries for active Dag runs, calls\nAirflow's dependency logic, and returns later for another pass, on a loop. Until\none of those passes examines our Dag run, `transform`\n\nremains `NULL`\n\neven though\n`extract`\n\nhas already recorded `success`\n\n.\n\nEvery tuning knob here trades one cost for another. Larger batches raise\nthroughput but let one scheduler hoard runs; running the loop more often cuts\nthe wait but issues more queries when nothing has changed, and at high load\nthose passes compete with heartbeats, state updates, and API calls for the same\ndatabase. Adding schedulers helps throughput and availability, but the replicas\nstill coordinate through database locks. When a scheduler finds `transform`\n\nand\nits dependencies pass, it changes the row to `scheduled`\n\n.\n\n*By this point, *`transform`\n\n* has waited for a scheduling pass and dependency\nchecks.*\n\n### 3. Shared limits must clear before the task queues\n\nWhen a task is `scheduled`\n\n, that does not necessarily mean it's ready to run.\nAirflow has a robust queueing and pooling system, which lets a Dag author\ndeclare cross-Dag behavior for how tasks run. For example, an external service\nmay impose API rate limits, so the author declares a pool that limits how many\ntasks interacting with that API can run.\n\nBefore changing a task to `queued`\n\n, the scheduler must account for limits shared\nby many Dags: pool slots, global system concurrency, Dag concurrency, task\nconcurrency, and executor capacity. Its critical section protects those shared\nlimits while it selects a batch of tasks. More scheduler replicas can therefore\nincrease both scheduling capacity and contention for the same database state.\n\n*By this point, *`transform`\n\n* has waited for a scheduling pass, dependency checks,\nshared-limit checks, and a database update.*\n\n### 4. The executor must deliver the workload\n\nAirflow separates the decision to run a task from the system that starts it.\nOnce the scheduler marks `transform`\n\nas `queued`\n\n, the configured executor takes\nover. LocalExecutor starts a process on the same machine, CeleryExecutor sends a workload to a pool of workers through a broker, usually\nRedis or RabbitMQ, or KubernetesExecutor asks\nKubernetes to create a pod.\n\nWe focus on\n[CeleryExecutor](https://airflow.apache.org/docs/apache-airflow-providers-celery/stable/celery_executor.html)\nbecause it has long been a common choice for distributed Airflow deployments\nand thus is the open-source execution path used in our benchmarks. Celery adds a\ndelivery chain on top of the scheduling work: the scheduler serializes a\nworkload, the broker stores it on the right queue, and a worker consumes it\nsubject to its free concurrency. During a burst, every link in that chain\nhandles more connections, deeper queues, and workers already holding prefetched\nwork—and adding workers does not remove the broker round trip.\n\n*By this point, *`transform`\n\n* has waited for a scheduling pass, dependency checks,\nshared-limit checks, a database update, and a broker round trip.*\n\n### 5. A worker must have a free slot and start the process\n\nReceiving the workload does not start the task. `transform`\n\nremains `queued`\n\nuntil the Celery worker has a concurrency slot and starts a child process. Under\nAirflow 3, that process uses the Task SDK and Execution API to report that it is\n`running`\n\n, send heartbeats, fetch the values it needs, and report its final\nstate.\n\nIf every matching worker is full, the workload waits in the broker. If no worker exists, the deployment must start a worker VM or pod before the process can run. Broker wait, worker capacity, and infrastructure startup therefore all count toward the delay after the scheduler queues the task.\n\n*By this point, *`transform`\n\n* has waited for a scheduling pass, dependency checks,\nshared-limit checks, a database update, a broker round trip, and a worker slot.\nOnly now does it become *`running`\n\n*.*\n\n### Capacity must already exist\n\nFor there to be no infrastructure holdups, every step above assumes the schedulers, database, broker, and workers it needs are already running. That assumption has its own timing problem.\n\n**Infrastructure metrics describe what a deployment is doing now, while\nAirflow's tables often show what it will need next.** A trigger can wait for an\nexternal event while using little CPU, then make thousands of task instances\nrunnable when the event arrives. A timetable can show that a large Dag run is\ndue before its first task consumes any worker resource. A growing queue can\nappear in the database before a queue depth metric crosses a scaling threshold.\n\nIf queue depth is the only scaling input, new workers begin to start only after the existing workers are busy and tasks are waiting. Schedulers, API servers, triggerers, and database proxies have the same timing problem: each sees one part of demand, but a task needs capacity across the whole path.\n\n### How the waits add up\n\nAt modest load, each stage can finish quickly. As task counts grow, the wait at every gate below grows with it.\n\nTwo of those waits happen after the row says `queued`\n\n. The row does not\nchange again, so nothing you can query tells you which wait the task is in.\n\nThe usual fixes address one step at a time:\n\n- Polling faster just makes the database do more work.\n- Add schedulers, and shared locking and database work remain.\n- Tune Celery, and the broker remains a second delivery system.\n- Scale on CPU, and the metric moves after Airflow has already created the demand.\n\nThe same choices that give Airflow durable state, recovery, and broad executor support also define this task-start path. We kept those properties and changed what happens between the database write and the running process.\n\n## Making it go bigger and faster\n\nOur job at Astronomer is to make Airflow work at the largest scales our\ncustomers run, and the waits in the lifecycle above all have the same shape:\nloops. The scheduler loops over Dag runs looking for ready tasks. Workers sit\nwaiting for tasks. The autoscaler waits for task demand deciding when to add\ncapacity. Each loop wakes on its own timer, checks for changes, and goes back to\nsleep. A task starts only after every loop on its path has come around. Yet the\nprocess that wrote `success`\n\nalready had the information every one of those\nloops is polling for. Airflow gives it no way to pass that on.\n\nSo we rebuilt the systems that start a task: whoever changes state now tells the next service to look, right away, by passing along a \"hint\" instead of leaving each loop to find the change on its own timer. The Astro Scheduler replaces polling discovery and reacts the moment work changes. The Astro Executor replaces the Celery delivery path and assigns tasks to workers directly. The Astro Hypervisor scales deployments according to schedules operators define and prepares capacity from Airflow's own state before tasks need it. Around them, the Astro Runtime image packages the software the path runs, and cross-region recovery protects the database everything else depends on.\n\nWe also studied each component to understand its scaling properties in an effort to increase scalability. As customers demand more of the system, we want to make sure that it's not only faster, but also can be run with more (and more frequently running) Dags and tasks. For example, we undertook a huge effort with the Airflow scheduler to reduce the database critical section so that we can run across more replicas in parallel. For reference, with the standard scheduler we can generally run up to 10 replicas but with the Astro Scheduler we've run hundreds of replicas concurrently.\n\nAn effort like this only counts if the result is still Airflow: the same Dags, the same task states, the same operators, and the ecosystem built on them. Rebuilding the core of a large open-source project risks producing a fork that shares only the name. So we fixed two design principles before we wrote code:\n\n**The Airflow metadata database remains the only source of truth for Dag-run and task execution status****Existing Airflow Dags and tasks do not need to be rewritten to use the new path**\n\nThe first principle keeps the hints we pass around deliberately small. On the\nnew path, services do not generally poll to find work. When a service changes\nstate, it publishes a hint—a few bytes over a message broker—that wakes the next\nservice. The hint can say which `dag_id`\n\nto look at, and nothing more. It does not carry the workload, and it does not replace the\n`dag_run`\n\nand `task_instance`\n\nrows.\n\nWhy not put the full details in the hint? Because the hint would then become a second record of the work and recovery would have to keep that record and the database in agreement after every crash. Small, disposable hints avoid the problem. The database stays the only record of the work, and the fallback checks still run if a hint is lost.\n\nThe second principle holds because the new path is built on Airflow's own\ninterfaces. `extract`\n\n, `transform`\n\n, and `publish`\n\nstill use Airflow's task\nstates and dependency rules. The new scheduler uses Airflow's internal\nscheduling and task state functions as much as possible to avoid creating a fork\nsituation, where we're racing to keep up with the open-source project. (Note:\nWe've actually seen customers and prospects fork Airflow with their own\ncustomizations, and it inevitably leads to lots of maintenance work and a very\npainful upgrade.) The Astro Executor implements Airflow's existing executor\ncontract and Airflow 3's Task SDK carries the workload back through the Execution API.\n\nTogether, the two rules split the path so each part has one job:\n\n- The metadata database stores Dag runs, task states, ownership, and other facts that must survive a process restart.\n- The message broker carries small hints that something changed.\n- The Astro Scheduler applies Airflow's Dag-run and dependency logic when a hint arrives.\n- The Astro Executor assigns queued tasks to workers without a Celery broker, as soon as a hint reaches a worker with room.\n- Workers ask for work and run assigned task processes.\n- The Astro Hypervisor reads Airflow and Kubernetes state to scale services and repair known faults.\n\nDatabase discovery remains as the recovery path for a hint lost after a successful API request, a hint removed by a process that then stops, or a scheduler that restarts with no local record of its Dag runs. The fallback scans still cost database work, but they no longer set the usual task-start delay.\n\n## The Astro Scheduler follows hints\n\nNot every run needs a\n[sub-second start](https://www.astronomer.io/docs/astro/sub-second-pipelines). A\ndaily run that starts at midnight every night can afford the standard\nscheduler's next pass. The runs that cannot wait—an external system reacting to\nan event, a service fanning out work—typically arrive through the API. So that\nis where the new path begins: eligible Dags take it, and other Dags continue through the standard scheduler (for now, although we're continuing work on\nour scheduler to let it support a wider variety of Dags).\n\nWe've designed every hop on this path around the same three things: 1) commit the change to the database 2) drop a hint, and 3) the next service wakes, reads the database, and acts. To see how this works in action, let's return to our three-task Dag. Step through it once before reading the hops below.\n\n### Hop 1: the API creates a run\n\nAn external system creates a run with `POST /api/v2/dags/example_etl/dagRuns`\n\n.\nThe API transaction inserts one `dag_run`\n\nrow and three `task_instance`\n\nrows,\ntelling Airflow that it has work to do. Only after that transaction commits does\nan API-server plugin push a `schedule_dagruns`\n\nhint to a message broker. Its\npayload identifies `example_etl`\n\n; it does not repeat the Dag run, the task rows, or the workload. The message broker tells a scheduler where to look, and the\nscheduler can then look at the database for the state it must act on, so the hint is a pointer, not the record.\n\nThe Astro Scheduler consumes the hint from our message broker and then runs one database statement that does two jobs:\n\n- It finds eligible queued runs for that Dag that do not already have an owner.\n- It inserts ownership rows for as many runs as the replica has room to manage.\n\nWe make use of PostgreSQL's `ON CONFLICT DO NOTHING`\n\nso that two replicas can\nrace on the same run, but only one insert returns that run to its caller.\n\nOn its first pass it loads the Dag run record, changes it from `queued`\n\nto `running`\n\n, and commits. It then reuses as much of Airflow's internal scheduling\nlogic to maintain behavior with the open-source project. (As Ash Berlin-Taylor,\na prominent Airflow PMC member, likes to say, for a project as large and\nwell-adopted as Airflow, we need to keep even the \"bugs\" consistent.) For a new\nrun, that may make `extract`\n\nrunnable.\n\n### Hop 2: a task finishes\n\nThe same pattern moves the run forward. When `extract`\n\nfinishes, its worker\nreports `success`\n\nthrough the Execution API, which turns into a database write\nto the `task_instance`\n\ntable. A task-state hint wakes the run's scheduler, which\ncalls the same dependency functions and finds `transform`\n\nready. And again, we\nre-use Airflow's actual dependency engine here so that as we add more\nfunctionality to the open-source Airflow project, we're not maintaining our own\nfork.\n\nFor each runnable task, the scheduler performs a guarded database update. The production statement selects rows by their Airflow task-instance identity; in plain SQL, the important part is:\n\n```\nUPDATE task_instance\nSET state = 'queued', queued_dttm = now(), external_executor_id = NULL\nWHERE id IN (...)\n  AND state = 'scheduled';\n```\n\nThe `state = 'scheduled'`\n\nguard matters. If another scheduler moved the row\nafter this replica read it, PostgreSQL updates zero rows. If the update\nsucceeds, the row now carries `state=queued`\n\n, the time it entered the queue, and\nno worker assignment. The hint only told the scheduler when to look.\n\n### Hop 3: a hint wakes the executor\n\nAfter the commit, the Astro Scheduler publishes a wake-up hint via the API server. We'll talk about this more in the next section, but the Astro Executor workers will long poll the API server so that they consume work immediately, giving the API server a mechanism to forward the hint to the workers directly. And at this point, queued tasks already exist in the database, so the Astro Executor can find them through its fallback check if the hint disappears.\n\n*On this path, *`transform`\n\n* reached *`queued`\n\n* after hints and a few guarded\ndatabase statements—no polling pass and no batch tradeoff.*\n\n### The system recovers gracefully if hints break\n\nWe've designed this to be purely additive, so nothing actually *depends* on\nhints arriving. Each failure below ends the same way: a database check finds the\nwork, so we are, in effect, gracefully degrading to Airflow's normal behavior.\n\n**The API succeeds, but hint publication fails.** Because the database\ntransaction committed first, the Dag run and its task instances still exist. The\nplugin logs the broker error and returns the successful API result. On its next pass, the fallback check finds a `queued`\n\nor `running`\n\nrun with no ownership row, takes ownership, and continues down our scheduler's path, usually in less than a\nsecond.\n\n**Two hints arrive for the same Dag.** Both can make replicas query the\ndatabase, but the guarded ownership insert returns the run to only one of them.\nThe database objects supply the identity, so the hint needs no ID of its own.\n\n**A task-state hint disappears.** The state reported through Airflow's Execution\nAPI remains in `task_instance`\n\neven if our message broker loses the hint that\nwould wake this run at once. The per-run scheduler later wakes on its fallback\ntimer and calls `update_state()`\n\nagain; this usually happens within a few\nseconds.\n\n**An Astro Scheduler replica stops.** Each replica updates the heartbeat on its\nownership rows and if there hasn't been an active heartbeat, another replica can\nremove the stale row, find the active Dag run, and take ownership. The replacement\nreads the latest database state rather than replaying a workload held by the\nfailed process. A stale replica that resumes cannot double-queue a task: the\nguarded `scheduled`\n\n-to-`queued`\n\nupdate is a no-op once another scheduler has\nmoved it, and worker assignment adds one more check in the next section.\n\n**The message broker remains unavailable.** Every hint stops, so the fallback checks must find the runs and the task changes, and the Astro\nExecutor must poll for queued work—slower and heavier on the database, but\nacting on the same stored rows. This degrades to\nAirflow's natural behavior, running slower but still correctly.\n\nAt the end of this section, the `transform`\n\ntask has status `queued`\n\n: the\ndatabase records that scheduling chose it, but no user process is running. A\nworker must now ask the API server for work so the Astro Executor can assign the\ntask without putting a workload in a broker.\n\n## The Astro Executor assigns `transform`\n\nThe Astro Executor removes the Celery broker between workers and the executor in favor of workers long polling the Airflow API server. As described above, this lets our hint system propagate events from the Airflow system components (i.e. scheduler and API server) to the workers directly.\n\nEach worker tells the API server how much room it has. The API server assigns a queued task by writing the worker's ID onto the same database row the scheduler queued, and the worker learns about the task in the response to a request it already had open. On the worker, task processes fork from a pre-warmed parent, so the process start is cheap too.\n\nEach worker heartbeat includes its identity, the queues it serves, its total and free task slots, and whether it is shutting down. A worker with no free slots receives no work, while a cordoned worker—one marked to stop taking new work—can finish its current tasks without taking another one.\n\nWhen a worker has room, the API server holds its heartbeat request open until a matching task appears. This long poll removes the need for the worker to open a new request every few milliseconds.\n\nSuppose `worker-17`\n\nreports four free slots on the `default`\n\nqueue. The API\nserver's allocator—the code that assigns queued tasks to waiting workers—\nregisters that worker as a waiter, then selects a batch of queued, unassigned\ntasks for the queues its waiting workers serve. The selection rests on two plain\nPostgreSQL primitives. `FOR UPDATE`\n\nlocks the selected rows for this\ntransaction. `SKIP LOCKED`\n\nmakes this statement pass over rows another replica already holds\ninstead of waiting for them. Every replica runs the same statement, so they\nfill worker capacity at once without two of them taking the same task. And if the\nexecutor has a hint that a task is ready, it matches that workload against free worker slots and sends it at once.\n\nFor each selected row, the Astro Executor finds a waiting worker that serves the\nqueue and still has a free slot. When it chooses `worker-17`\n\n, it writes that\nworker's ID to `external_executor_id`\n\n, decrements the capacity held for the\nheartbeat, and builds the workload. The workload carries\n`dag_id`\n\n, `run_id`\n\n, `task_id`\n\n, `map_index`\n\n, the try number, details of the Dag\nbundle (the versioned set of Dag files the worker runs from), queue and trace\ndata, plus a signed token for the Airflow Execution API.\n\nThe task assignment and worker identity live in the same metadata row that the scheduler already queued. There is no broker message to reconcile with that row. The worker receives the task in its heartbeat response once the assignment is written.\n\n### The worker starts the process\n\nThe worker receives its assignments in the heartbeat response. A local coordinator reserves a slot for each one, then writes the workload to the worker process.\n\nThe worker validates the task data and forks an Airflow supervisor process. The parent records the child process ID and reports that the process started back to the local coordinator. The child calls Airflow's Task SDK supervisor with the Dag bundle, task instance, log path, signed token, and Execution API address. The Task SDK then starts the task runtime that imports the Dag and enters user code.\n\nForking from a warm parent, while not unique to Astro, is part of what makes that start cheap. Before any task arrives, the worker parent imports the expensive parts of Airflow once (the Task SDK, the task runner, the standard operators). Each forked task process inherits those imports instead of paying for them again. To make this as fast as possible, we have measured and optimized every single step and action between the workload arriving at the worker and the user's task code actually running.\n\nBefore the task transitions to `running`\n\n, there's a final check with the API\nserver to mark it as such, which also serves as a final check against the same\ntask running on two workers at the same time.\n\n*By this point, *`transform`\n\n* is *`running`\n\n*. It waited for a hint, a guarded\nassignment, one heartbeat response, and a fork from a warm parent—no broker.*\n\nAirflow 3 makes this split possible. The task process does not need a database\npassword or a route to the metadata database. Its short-lived token lets the supervisor use the Execution API for heartbeats, state changes, connections,\nvariables, and XComs. The same contract means a worker can run *anywhere* that\ncan reach the API server. Astro's remote execution is built on this: workers run\ninside the customer's own network—a cloud account, a data center, an on-prem GPU\ncluster—and pull work from an orchestration plane that never sees their code,\nlogs, or data.\n\nWhen something fails, the database acts as the source of truth for how far the\ntask has progressed and what needs to be done to recover. A worker that\ndisappears while `transform`\n\nis still `queued`\n\nhas lost only an assignment:\nrecovery clears `external_executor_id`\n\nfrom its rows, and another healthy worker\nclaims them. A partitioned worker that returns and starts its cleared assignment\nanyway cannot cause a double start: every task process must report `running`\n\nthrough the Execution API before it reaches user code, that transition is\nguarded like every other state change, and only one process wins it. A worker\nthat disappears after the process reports `running`\n\nis harder, because user code\nmay already have changed an external system; Airflow's heartbeat timeout marks\nthe task failed and its retry rules decide whether another try runs. Airflow\ndoes not guarantee only-once execution so Dag authors make side effects safe to\nretry (i.e., tasks are idempotent). If every worker is full, the task stays\n`queued`\n\nin the database until a worker has room. There is no in-memory list for\nan API server restart to lose.\n\nThe Astro Scheduler normally publishes a wake-up hint after it queues work. That wakes the assignment loop without making the message broker the source of the task. The loop also checks for work on a short fallback interval, so a missed hint delays the assignment rather than losing it.\n\nThe difference is how many hops sit between a queued row and a running process.\n\nOn the same hardware, an Astro Executor worker can run 70% more concurrent tasks than a Celery worker—more task processes per worker, not faster user code. Combined with the Astro Scheduler, this execution path has sustained 500,000 concurrent tasks. The load tests at the end of this post give the test and the method behind that number.\n\n## The Astro Hypervisor scales the system around `transform`\n\nThe task path reaches user code only when each service has capacity at the same time: a worker needs a free slot, an API server must hold the heartbeat, and the Astro Scheduler must have room to own the Dag run.\n\nWe originally built the Astro Hypervisor to help us manage the operations of many Airflow deployments in a single cluster. We run one Hypervisor service in every cluster to interact with potentially hundreds of Airflow deployments, without needing to go to every deployment individually. In doing so, it also acts as a translation layer—every Airflow version might have a slightly different metadata database structure and API format, so our Hypervisor acts as a central place to put logic to help decide what to do according to the Airflow version. It also helps with things like database connection pooling for efficiency.\n\nKubernetes can scale a service from CPU or memory, but those measures describe load after it reaches a process. Airflow has earlier signals. It knows how many tasks are queued, how many Dag runs remain active, which triggers are waiting, which runs are due soon, and whether schedulers and workers still send heartbeats.\n\nThe Astro Hypervisor turns that Airflow state into deployment-level decisions. Currently this works very well for the Airflow triggerer, which needs to scale rapidly because a single mapped task can result in thousands of deferred tasks to be placed on the triggerer. We have prototypes of scaling some of the other Airflow components, like the scheduler and API server, according to Airflow's state. Having an Airflow-specific hypervisor allows us to more intelligently make scaling decisions than if we were using a more general-purpose scaling system.\n\nFor each deployment, the Hypervisor can read three groups of input:\n\n- The Airflow metadata database:\n`task_instance`\n\nrows in`queued`\n\nor`running`\n\ngrouped by queue, active and upcoming Dag runs, late runs,`trigger`\n\nrows, worker slots, scheduler heartbeats, and table sizes. - Kubernetes: pods, Deployments, custom resources, and current replica counts.\n- Service health and metrics: worker health, Astro Scheduler latency, and component-level metrics.\n\nThose facts can, in theory, lead to different actions:\n\n| Stored or observed fact | Potential Hypervisor action |\nAn unfinished `dag_run` exists | Keep scheduling services available |\n| A Dag's next run enters the look-ahead window | Start needed services before the run is due |\nActive rows remain in `trigger` | Keep a triggerer available |\n| A Kubernetes task pod has no matching active task instance, or belongs to an older try | Check whether the pod is safe to remove |\n| A required scheduler metric is missing | Block scheduler scale-down rather than treating it as 0 |\n\nThe controller set depends on the deployment's Airflow version, executor, and features, so the Hypervisor does not apply one replica formula to every pod. Some of Airflow's features make autoscaling things like the scheduler very tricky; for example, a user can declare a \"dynamic Dag\" that changes every time it's parsed. Scheduler autoscaling is the piece still in prototype. We are working through these edge cases internally before we release it to customers.\n\n### Scaling from planned work\n\nBefore `extract`\n\nfinishes, worker CPU metrics have no way to show that\n`transform`\n\nwill soon become runnable, but the unfinished Dag run and dependency\nstate in the metadata database already place it in planned work. This is where\nhaving a domain-specific hypervisor becomes interesting.\n\nThe Hypervisor can keep schedulers active while a Dag run remains unfinished or\nthe Dag model says another run is due soon, and can keep the triggerer active\nwhile `trigger`\n\nrows exist. We'll still use traditional autoscalers like KEDA,\nthe Kubernetes autoscaler, but the Hypervisor adds Airflow context, can pause\nworker scaling during hibernation, and sequences scale-to-zero so one service\ndoes not vanish while another still needs it.\n\nAirflow has already written down the work that is coming, so the Hypervisor reads that instead of guessing from CPU metrics.\n\n### What the Hypervisor repairs\n\nThe Hypervisor can also detect late Dag runs, stale heartbeats, stuck tasks, slow parsing, large database tables, unhealthy workers, and Astro Scheduler latency. And because we operate thousands of production Airflow deployments, we know which failures repeat and exactly what state marks them, so the Hypervisor repairs those itself:\n\n- A Celery worker has free slots but has stopped accepting queued work.\n- A worker pod remains stuck in\n`Terminating`\n\nafter its tasks are gone. - A Kubernetes task pod no longer matches an active task instance or try.\n\nEach repair first checks Airflow and Kubernetes state. If that check fails, the repair stops instead of acting on partial evidence. Other conditions become incidents for an operator or another system.\n\n### Hibernation is an ordered stop\n\nThe Hypervisor also turns hibernation schedules and manual overrides into one decision: it scales services to zero in dependency order and later restores their normal replica sources. Hibernation is not a lossless pause. Shutdown marks remaining Dag runs and tasks failed or skipped, removes task pods, and writes heartbeats for services stopped on purpose so health checks do not report the shutdown as a fault. That trade fits idle non-production deployments; a deployment with running production tasks keeps its dependencies active.\n\nAcross the measured deployment group, triggerer scaling and the repair controls above cut Dag failures by 85%. The figure counts the failure classes the Hypervisor can repair; faults in user code, external services, or data remain the Dag author's.\n\n## The software beneath the path\n\nThe Astro Runtime image does its work before the Dag run arrives. It pins the Airflow and Task SDK versions and sets bounds for the providers, Python packages, database clients, and system code used by each service. Customer projects can add packages, but they start from a known set.\n\nWe test that set across supported Python versions, processor architectures, base operating systems, databases, and executors. Release candidates pass package, Docker, CLI, Kubernetes, migration, and security tests. Supported releases also receive daily security scans and selected backports for key bugs and CVEs for two years.\n\n## When a region fails\n\nSo far, every failure has assumed the metadata database remains available. A message broker hint can disappear, a scheduler can stop, and a worker can lose its connection while another process reads the same stored state.\n\nA regional outage can remove the database, Kubernetes cluster, object storage, workers, and network paths at once.\n\nAstro cross-region recovery prepares a second dedicated cluster in another region of the same cloud. The primary cluster runs deployments during normal operation. The secondary remains ready for promotion. Three kinds of state move between them:\n\n- The Multi-Region DB copies deployment metadata, including Dag-run history, task-instance state, XComs, connections, variables, and configuration.\n- Multi-region object storage copies task logs.\n- Image replication makes user-deployed images available in the second region.\n\n### Promoting the secondary cluster\n\nAn Organization Owner begins failover through the Astro UI or API. Astro promotes the secondary cluster and starts the deployments there. Deployment IDs, settings, and service hostnames remain the same. Airflow UI and API traffic begins routing to the active cluster in the second region.\n\nPromotion must therefore restore more than a database replica.\n\nTasks can fail during the switch. After promotion, an operator checks deployment health and retries work whose state requires it.\n\nFor our example, the outcome depends on the last state committed before the outage:\n\n- If\n`transform`\n\ncompleted and reported success, the secondary region can use that state to consider`publish`\n\n. - If\n`transform`\n\nwas queued but had not started, the active recovery deployment can assign it to a worker. - If\n`transform`\n\nwas running when the primary failed, its process is gone. Its final database state and Airflow retry policy decide what happens next. - If user code changed an external system before failing, its retry still needs safe external writes.\n\n### What recovery promises\n\nFor deployments using the task-log replication SLA, Astro's recovery point objective is under 15 minutes and measures how far copied state may lag. The metadata database is usually seconds old; this matters most, because it holds the real state of the Dags, including what has and hasn't been run. Task logs stored in object storage are generally minutes as opposed to seconds.\n\nThe recovery time objective is under one hour and measures how long the secondary cluster and its deployments take to become available. We constantly run our own internal benchmarks and verified both objectives under load in a cross-region test with more than 80 deployments and more than 1,250 concurrent task runs.\n\nFailback is the same kind of switch. Astro reverses replication so the former primary can catch up; once it has, an Organization Owner moves services and hostnames back, and tasks can again fail during the change.\n\nAstro manages the paired cluster, replication, promotion, stable service hostnames, and failback. Customers prepare regional network routes, private endpoints, identities, image credentials, and Dag settings, then check deployment health and retry failed work after a switch.\n\nWe have now followed one task across the main failure boundaries: a lost hint, a lost worker, a failed pod, and a lost region. Each layer handles a larger failure. None changes the basic rule: durable task state belongs in the metadata database, while user code must remain safe to retry.\n\n## What the full system changes\n\nIn our benchmarks, we've pushed Astro to 500,000 sustained concurrent tasks, 228 milliseconds p95 at 100,000, and an 85% cut in Dag failures. No single test measures the whole architecture, so we tested its layers apart and together.\n\n### Isolating the execution path\n\nWe wanted to measure the Astro Scheduler and Astro Executor, not how fast Kubernetes could add a worker or how soon an autoscaler noticed load. Before each run, we started more worker slots than the target required. A 50,000-task run, for example, had more than 60,000 slots ready before the test began. We also provisioned enough API server capacity and raised the replica and resource limits that Astro exposes in the product, past the ceilings a user can set today. These runs measure the Astro Scheduler and Astro Executor, not the current product caps.\n\nThat setup does not make the test free of infrastructure. Reaching these counts found limits elsewhere. Some database queries needed missing indexes before the control path could progress to the next load level. At high pod counts, the Kubernetes scheduler used for bin packing ran out of memory. We hit a vCPU ceiling on our Kubernetes cluster twice. We had to separate those failures from Astro Scheduler and Astro Executor behavior rather than counting every failed run as the same limit.\n\nThe test holds a large group of long-sleeping tasks at a steady level, then triggers a small probe Dag. The first measure asks how long the first task in that new run takes to start. The second asks how long each following task waits after its upstream task finishes. This shows whether a deployment can begin and advance new work while hundreds of thousands of other tasks remain active.\n\n### Task-start latency under load\n\nAt 50,000 concurrent tasks, the standard scheduler with Celery recorded 23.582 seconds of p95 task-start latency. We stopped the Celery baseline there: task lag at that level already makes a deployment hard to operate, and pushing further mainly measures the queue growing. With the Astro Scheduler and Astro Executor, p95 was 228 milliseconds at 100,000 concurrent tasks—at twice the load, less than one-hundredth of the latency. At 300,000 concurrent tasks, p95 remained 294 milliseconds.\n\n### Maximum concurrent tasks\n\nWe also tested how much resident task state the execution path could manage. These tests use long-running tasks so concurrency rises and stays high instead of draining before the next group starts. The result measures how much running state the path can hold, not how many short tasks it can start each second.\n\nWith the standard scheduler, the Astro Executor sustained 200,000 concurrent tasks. Combining it with the Astro Scheduler reached 500,000 sustained concurrent tasks.\n\n| Scheduler | Executor | Sustained concurrent tasks |\n| Standard Airflow | Astro Executor | 200,000 |\n| Astro Scheduler | Astro Executor | 500,000 |\n\nThe first row separates Astro Executor capacity from Astro Scheduler capacity. At 200,000 tasks, the standard scheduler becomes the next limit. Replacing both halves of the path raises the result to 500,000, so this is not an Astro Executor result alone.\n\nEach result depends on the system we tested it on. Worker size, database capacity, service replicas, Dag shape, task duration, queue count, and settings all affect where the next limit appears.\n\n## Back to `transform`\n\nAirflow still records its dependencies and state. The Astro Scheduler reacts\nwhen `extract`\n\nsucceeds, takes ownership of the Dag run, and moves\n`transform`\n\nto `queued`\n\nthrough a guarded database update. The Astro Executor assigns it to a worker that has room. That worker starts an Airflow Task SDK\nprocess and reports state through the Execution API. The Astro Hypervisor reads\nthe same deployment state to keep the needed services active. The Astro Runtime\nimage supplies the tested software beneath them. If the whole region fails, the\nMulti-Region DB and secondary cluster preserve the stored record from which\nAirflow can recover.\n\nThroughout that path, the Dag and Airflow task states stay the same. Astro changes how quickly the system sees those states, how it assigns the work, and how the services around the path scale.\n\nWe're really proud of the work we've done and don't plan to slow down with either the open-source Apache Airflow project nor our commercial products. The market is incredibly dynamic and we welcome this new class and scale of workloads that have emerged as LLMs have unlocked new workflow use cases and made it easier to produce code for the use cases we know and love.\n\n*P.S. We're hiring. If you want to work on the systems in this post,\n**we'd love to speak with you**.*", "url": "https://wpnews.pro/news/re-engineering-apache-airflow-for-speed-and-scale", "canonical_source": "https://www.astronomer.io/blog/astro-airflow-re-engineered-for-speed-and-scale/", "published_at": "2026-08-31 18:28:08+00:00", "updated_at": "2026-08-31 18:52:17.058996+00:00", "lang": "en", "topics": ["ai-infrastructure", "ai-tools"], "entities": ["Astronomer", "Apache Airflow", "Astro"], "alternates": {"html": "https://wpnews.pro/news/re-engineering-apache-airflow-for-speed-and-scale", "markdown": "https://wpnews.pro/news/re-engineering-apache-airflow-for-speed-and-scale.md", "text": "https://wpnews.pro/news/re-engineering-apache-airflow-for-speed-and-scale.txt", "jsonld": "https://wpnews.pro/news/re-engineering-apache-airflow-for-speed-and-scale.jsonld"}}