Re-engineering Apache Airflow for speed and scale

Astronomer's Astro Runtime sustained 500,000 concurrent Apache Airflow tasks in a single deployment and achieved 228 milliseconds p95 task-start latency at 100,000 concurrent tasks, compared to 23.582 seconds for open-source Airflow at half that load. The company re-engineered Airflow's scheduling, execution, scaling, and recovery systems while maintaining compatibility with Airflow 3+ releases.

The Astro Runtime: Airflow re-engineered for speed and scale 36 min read | Apache Airflow is the open-source standard for defining and running data workflows. Astronomer builds Astro, a managed Airflow platform for teams running those workflows in production. Airflow is no longer used only for scheduled batch pipelines. Teams now use it to coordinate thousands of data and AI workflows across an enterprise: loading warehouse tables, building dbt models, preparing training data, running model evaluations, and reacting to external events. These workflows can release tens of thousands of tasks at once, keep hundreds of thousands running, and still require a newly ready task to start in hundreds of milliseconds. Our customers hit ceilings in how fast Airflow starts a task and how many it keeps running, so we rebuilt the path behind both. Over the past several years, we have rebuilt Airflow's scheduling, execution, scaling, and recovery systems on Astro to perform beyond any current demands. And we did it in a way where Astro still works seamlessly with Airflow 3+ releases, meaning nothing needs to change with how pipelines are defined. In our load tests, Astro sustained 500,000 concurrent Airflow tasks in a single deployment and reached 228 milliseconds p95 task-start latency at 100,000 concurrent tasks —less than one-hundredth of the 23.582 seconds we measured when we pushed open-source Airflow to just half that load. At 300,000 concurrent tasks, p95 remained 294 milliseconds. We made these changes without replacing Airflow's workflow model. User Dags, task dependencies, task execution status, and operator code remain compatible with open-source Airflow. Astro just changes components under the hood to make task execution faster and more scalable while improving reliability: Coordination Execution— one executor per deployment Platform Heavy borders mark the systems Astronomer built; the rest is Apache Airflow. This is the control path around a task, not every service in a deployment. How Airflow got here Airflow began 10 years ago with a clear job: read workflows defined as Python code, decide which tasks were ready, and send them somewhere to run. A scheduler made those decisions, an executor handled the delivery, and a SQL database recorded Dag runs and task states. That design gave Airflow two useful properties. Dag authors could choose how tasks ran without changing their workflow code, and a failed scheduler could rebuild its view from the database. The database, rather than any scheduler process, held the lasting account of the work. As deployments added more Dags and tasks, the scheduler had to parse more Python files, create more Dag runs, check more dependencies, and queue more task instances. One scheduler also left the whole deployment dependent on one process. Airflow 2.0 and beyond addressed those limits with a few large changes, without replacing the database model. AIP-15 https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/103092651/AIP-15%2BSupport%2BMultiple-Schedulers%2Bfor%2BHA%2BBetter%2BScheduling%2BPerformance let several scheduler replicas run at once. Each could examine Dag runs in parallel, then use database row locks when moving task instances from scheduled into an executor. Airflow also changed to store serialized Dags https://airflow.apache.org/docs/apache-airflow/2.3.3/dag-serialization.html in the database, so a scheduler could make decisions from a stored graph instead of importing every Python Dag file itself. Airflow 2.2 introduced ways to offload tasks that were waiting for some external system via the triggerer. Consider a daily ingestion Dag that cannot start until a partner uploads complete.json to an object-storage bucket. A sensor may spend hours checking for that file while doing almost no work, yet it can still occupy a worker slot. A deferrable operator https://airflow.apache.org/docs/apache-airflow/2.2.2/authoring-and-scheduling/deferring.html stores what it needs to resume, releases the worker, and hands the wait to a triggerer. When the file arrives, the trigger fires and the scheduler checks the task again before it can run. Airflow 2.3 then introduced the separate Dag parser component, which separated out user code from the core scheduler component and loop. This was much better from a security/isolation perspective and also meant that Dag processing could scale separately from scheduling. Airflow 3 gave task execution a clear API contract. Under AIP-72 https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/311626182/AIP-72%2BTask%2BExecution%2BInterface%2Baka%2BTask%2BSDK , task processes no longer need direct access to the metadata database like they did in Airflow 2.x. The Task SDK and Execution API define what a task receives, including its identity, connections, and variables, and what it reports, including heartbeats, state changes, and XComs used for cross-task communication . Workers and task code use that interface instead of reading and updating Airflow's internal tables. Modern Airflow divides the original system's work among Dag processing, active-active schedulers, triggerers, executors, API servers, and workers. This is the architecture we helped build in Airflow: durable database state, active-active scheduling, deferred execution, and a clear task API. At Astronomer, we led the HA scheduler, the triggerer and deferred tasks, and the Task SDK and Execution API, and we contributed to moving Dag processing out of the scheduler loop. As deployments grew and workflows carried more weight, we also saw where the task-start path needed a different design. The lifecycle of an Airflow task Here is what happens after one task finishes, first in open-source Airflow and then in Astro. The Dag is a basic three-task ETL. In our example Dag, when extract succeeds, Airflow records that result in the metadata database. The scheduler must notice the change, confirm that transform may run, check shared limits, queue it through the configured executor, and wait for a worker to start its process. Only then does transform begin running. These steps run in order. transform cannot skip any of them, so each adds its own wait to the total. 1. The upstream task finishes The dag run state for example etl is running, the task instance state for extract is success , and the rows for transform and publish have a NULL state. Airflow reads the dependency from the serialized Dag and evaluates it against the task states for this run on a loop. The lasting record of transform is its task instance row, not an in-memory object passed from one process to another. Different parts of Airflow update that row as they make decisions about it. 2. A scheduling pass must discover the change The standard scheduler works in batches. It queries for active Dag runs, calls Airflow's dependency logic, and returns later for another pass, on a loop. Until one of those passes examines our Dag run, transform remains NULL even though extract has already recorded success . Every tuning knob here trades one cost for another. Larger batches raise throughput but let one scheduler hoard runs; running the loop more often cuts the wait but issues more queries when nothing has changed, and at high load those passes compete with heartbeats, state updates, and API calls for the same database. Adding schedulers helps throughput and availability, but the replicas still coordinate through database locks. When a scheduler finds transform and its dependencies pass, it changes the row to scheduled . By this point, transform has waited for a scheduling pass and dependency checks. 3. Shared limits must clear before the task queues When a task is scheduled , that does not necessarily mean it's ready to run. Airflow has a robust queueing and pooling system, which lets a Dag author declare cross-Dag behavior for how tasks run. For example, an external service may impose API rate limits, so the author declares a pool that limits how many tasks interacting with that API can run. Before changing a task to queued , the scheduler must account for limits shared by many Dags: pool slots, global system concurrency, Dag concurrency, task concurrency, and executor capacity. Its critical section protects those shared limits while it selects a batch of tasks. More scheduler replicas can therefore increase both scheduling capacity and contention for the same database state. By this point, transform has waited for a scheduling pass, dependency checks, shared-limit checks, and a database update. 4. The executor must deliver the workload Airflow separates the decision to run a task from the system that starts it. Once the scheduler marks transform as queued , the configured executor takes over. LocalExecutor starts a process on the same machine, CeleryExecutor sends a workload to a pool of workers through a broker, usually Redis or RabbitMQ, or KubernetesExecutor asks Kubernetes to create a pod. We focus on CeleryExecutor https://airflow.apache.org/docs/apache-airflow-providers-celery/stable/celery executor.html because it has long been a common choice for distributed Airflow deployments and thus is the open-source execution path used in our benchmarks. Celery adds a delivery chain on top of the scheduling work: the scheduler serializes a workload, the broker stores it on the right queue, and a worker consumes it subject to its free concurrency. During a burst, every link in that chain handles more connections, deeper queues, and workers already holding prefetched work—and adding workers does not remove the broker round trip. By this point, transform has waited for a scheduling pass, dependency checks, shared-limit checks, a database update, and a broker round trip. 5. A worker must have a free slot and start the process Receiving the workload does not start the task. transform remains queued until the Celery worker has a concurrency slot and starts a child process. Under Airflow 3, that process uses the Task SDK and Execution API to report that it is running , send heartbeats, fetch the values it needs, and report its final state. If every matching worker is full, the workload waits in the broker. If no worker exists, the deployment must start a worker VM or pod before the process can run. Broker wait, worker capacity, and infrastructure startup therefore all count toward the delay after the scheduler queues the task. By this point, transform has waited for a scheduling pass, dependency checks, shared-limit checks, a database update, a broker round trip, and a worker slot. Only now does it become running . Capacity must already exist For there to be no infrastructure holdups, every step above assumes the schedulers, database, broker, and workers it needs are already running. That assumption has its own timing problem. Infrastructure metrics describe what a deployment is doing now, while Airflow's tables often show what it will need next. A trigger can wait for an external event while using little CPU, then make thousands of task instances runnable when the event arrives. A timetable can show that a large Dag run is due before its first task consumes any worker resource. A growing queue can appear in the database before a queue depth metric crosses a scaling threshold. If queue depth is the only scaling input, new workers begin to start only after the existing workers are busy and tasks are waiting. Schedulers, API servers, triggerers, and database proxies have the same timing problem: each sees one part of demand, but a task needs capacity across the whole path. How the waits add up At modest load, each stage can finish quickly. As task counts grow, the wait at every gate below grows with it. Two of those waits happen after the row says queued . The row does not change again, so nothing you can query tells you which wait the task is in. The usual fixes address one step at a time: - Polling faster just makes the database do more work. - Add schedulers, and shared locking and database work remain. - Tune Celery, and the broker remains a second delivery system. - Scale on CPU, and the metric moves after Airflow has already created the demand. The same choices that give Airflow durable state, recovery, and broad executor support also define this task-start path. We kept those properties and changed what happens between the database write and the running process. Making it go bigger and faster Our job at Astronomer is to make Airflow work at the largest scales our customers run, and the waits in the lifecycle above all have the same shape: loops. The scheduler loops over Dag runs looking for ready tasks. Workers sit waiting for tasks. The autoscaler waits for task demand deciding when to add capacity. Each loop wakes on its own timer, checks for changes, and goes back to sleep. A task starts only after every loop on its path has come around. Yet the process that wrote success already had the information every one of those loops is polling for. Airflow gives it no way to pass that on. So we rebuilt the systems that start a task: whoever changes state now tells the next service to look, right away, by passing along a "hint" instead of leaving each loop to find the change on its own timer. The Astro Scheduler replaces polling discovery and reacts the moment work changes. The Astro Executor replaces the Celery delivery path and assigns tasks to workers directly. The Astro Hypervisor scales deployments according to schedules operators define and prepares capacity from Airflow's own state before tasks need it. Around them, the Astro Runtime image packages the software the path runs, and cross-region recovery protects the database everything else depends on. We also studied each component to understand its scaling properties in an effort to increase scalability. As customers demand more of the system, we want to make sure that it's not only faster, but also can be run with more and more frequently running Dags and tasks. For example, we undertook a huge effort with the Airflow scheduler to reduce the database critical section so that we can run across more replicas in parallel. For reference, with the standard scheduler we can generally run up to 10 replicas but with the Astro Scheduler we've run hundreds of replicas concurrently. An effort like this only counts if the result is still Airflow: the same Dags, the same task states, the same operators, and the ecosystem built on them. Rebuilding the core of a large open-source project risks producing a fork that shares only the name. So we fixed two design principles before we wrote code: The Airflow metadata database remains the only source of truth for Dag-run and task execution status Existing Airflow Dags and tasks do not need to be rewritten to use the new path The first principle keeps the hints we pass around deliberately small. On the new path, services do not generally poll to find work. When a service changes state, it publishes a hint—a few bytes over a message broker—that wakes the next service. The hint can say which dag id to look at, and nothing more. It does not carry the workload, and it does not replace the dag run and task instance rows. Why not put the full details in the hint? Because the hint would then become a second record of the work and recovery would have to keep that record and the database in agreement after every crash. Small, disposable hints avoid the problem. The database stays the only record of the work, and the fallback checks still run if a hint is lost. The second principle holds because the new path is built on Airflow's own interfaces. extract , transform , and publish still use Airflow's task states and dependency rules. The new scheduler uses Airflow's internal scheduling and task state functions as much as possible to avoid creating a fork situation, where we're racing to keep up with the open-source project. Note: We've actually seen customers and prospects fork Airflow with their own customizations, and it inevitably leads to lots of maintenance work and a very painful upgrade. The Astro Executor implements Airflow's existing executor contract and Airflow 3's Task SDK carries the workload back through the Execution API. Together, the two rules split the path so each part has one job: - The metadata database stores Dag runs, task states, ownership, and other facts that must survive a process restart. - The message broker carries small hints that something changed. - The Astro Scheduler applies Airflow's Dag-run and dependency logic when a hint arrives. - The Astro Executor assigns queued tasks to workers without a Celery broker, as soon as a hint reaches a worker with room. - Workers ask for work and run assigned task processes. - The Astro Hypervisor reads Airflow and Kubernetes state to scale services and repair known faults. Database discovery remains as the recovery path for a hint lost after a successful API request, a hint removed by a process that then stops, or a scheduler that restarts with no local record of its Dag runs. The fallback scans still cost database work, but they no longer set the usual task-start delay. The Astro Scheduler follows hints Not every run needs a sub-second start https://www.astronomer.io/docs/astro/sub-second-pipelines . A daily run that starts at midnight every night can afford the standard scheduler's next pass. The runs that cannot wait—an external system reacting to an event, a service fanning out work—typically arrive through the API. So that is where the new path begins: eligible Dags take it, and other Dags continue through the standard scheduler for now, although we're continuing work on our scheduler to let it support a wider variety of Dags . We've designed every hop on this path around the same three things: 1 commit the change to the database 2 drop a hint, and 3 the next service wakes, reads the database, and acts. To see how this works in action, let's return to our three-task Dag. Step through it once before reading the hops below. Hop 1: the API creates a run An external system creates a run with POST /api/v2/dags/example etl/dagRuns . The API transaction inserts one dag run row and three task instance rows, telling Airflow that it has work to do. Only after that transaction commits does an API-server plugin push a schedule dagruns hint to a message broker. Its payload identifies example etl ; it does not repeat the Dag run, the task rows, or the workload. The message broker tells a scheduler where to look, and the scheduler can then look at the database for the state it must act on, so the hint is a pointer, not the record. The Astro Scheduler consumes the hint from our message broker and then runs one database statement that does two jobs: - It finds eligible queued runs for that Dag that do not already have an owner. - It inserts ownership rows for as many runs as the replica has room to manage. We make use of PostgreSQL's ON CONFLICT DO NOTHING so that two replicas can race on the same run, but only one insert returns that run to its caller. On its first pass it loads the Dag run record, changes it from queued to running , and commits. It then reuses as much of Airflow's internal scheduling logic to maintain behavior with the open-source project. As Ash Berlin-Taylor, a prominent Airflow PMC member, likes to say, for a project as large and well-adopted as Airflow, we need to keep even the "bugs" consistent. For a new run, that may make extract runnable. Hop 2: a task finishes The same pattern moves the run forward. When extract finishes, its worker reports success through the Execution API, which turns into a database write to the task instance table. A task-state hint wakes the run's scheduler, which calls the same dependency functions and finds transform ready. And again, we re-use Airflow's actual dependency engine here so that as we add more functionality to the open-source Airflow project, we're not maintaining our own fork. For each runnable task, the scheduler performs a guarded database update. The production statement selects rows by their Airflow task-instance identity; in plain SQL, the important part is: UPDATE task instance SET state = 'queued', queued dttm = now , external executor id = NULL WHERE id IN ... AND state = 'scheduled'; The state = 'scheduled' guard matters. If another scheduler moved the row after this replica read it, PostgreSQL updates zero rows. If the update succeeds, the row now carries state=queued , the time it entered the queue, and no worker assignment. The hint only told the scheduler when to look. Hop 3: a hint wakes the executor After the commit, the Astro Scheduler publishes a wake-up hint via the API server. We'll talk about this more in the next section, but the Astro Executor workers will long poll the API server so that they consume work immediately, giving the API server a mechanism to forward the hint to the workers directly. And at this point, queued tasks already exist in the database, so the Astro Executor can find them through its fallback check if the hint disappears. On this path, transform reached queued after hints and a few guarded database statements—no polling pass and no batch tradeoff. The system recovers gracefully if hints break We've designed this to be purely additive, so nothing actually depends on hints arriving. Each failure below ends the same way: a database check finds the work, so we are, in effect, gracefully degrading to Airflow's normal behavior. The API succeeds, but hint publication fails. Because the database transaction committed first, the Dag run and its task instances still exist. The plugin logs the broker error and returns the successful API result. On its next pass, the fallback check finds a queued or running run with no ownership row, takes ownership, and continues down our scheduler's path, usually in less than a second. Two hints arrive for the same Dag. Both can make replicas query the database, but the guarded ownership insert returns the run to only one of them. The database objects supply the identity, so the hint needs no ID of its own. A task-state hint disappears. The state reported through Airflow's Execution API remains in task instance even if our message broker loses the hint that would wake this run at once. The per-run scheduler later wakes on its fallback timer and calls update state again; this usually happens within a few seconds. An Astro Scheduler replica stops. Each replica updates the heartbeat on its ownership rows and if there hasn't been an active heartbeat, another replica can remove the stale row, find the active Dag run, and take ownership. The replacement reads the latest database state rather than replaying a workload held by the failed process. A stale replica that resumes cannot double-queue a task: the guarded scheduled -to- queued update is a no-op once another scheduler has moved it, and worker assignment adds one more check in the next section. The message broker remains unavailable. Every hint stops, so the fallback checks must find the runs and the task changes, and the Astro Executor must poll for queued work—slower and heavier on the database, but acting on the same stored rows. This degrades to Airflow's natural behavior, running slower but still correctly. At the end of this section, the transform task has status queued : the database records that scheduling chose it, but no user process is running. A worker must now ask the API server for work so the Astro Executor can assign the task without putting a workload in a broker. The Astro Executor assigns transform The Astro Executor removes the Celery broker between workers and the executor in favor of workers long polling the Airflow API server. As described above, this lets our hint system propagate events from the Airflow system components i.e. scheduler and API server to the workers directly. Each worker tells the API server how much room it has. The API server assigns a queued task by writing the worker's ID onto the same database row the scheduler queued, and the worker learns about the task in the response to a request it already had open. On the worker, task processes fork from a pre-warmed parent, so the process start is cheap too. Each worker heartbeat includes its identity, the queues it serves, its total and free task slots, and whether it is shutting down. A worker with no free slots receives no work, while a cordoned worker—one marked to stop taking new work—can finish its current tasks without taking another one. When a worker has room, the API server holds its heartbeat request open until a matching task appears. This long poll removes the need for the worker to open a new request every few milliseconds. Suppose worker-17 reports four free slots on the default queue. The API server's allocator—the code that assigns queued tasks to waiting workers— registers that worker as a waiter, then selects a batch of queued, unassigned tasks for the queues its waiting workers serve. The selection rests on two plain PostgreSQL primitives. FOR UPDATE locks the selected rows for this transaction. SKIP LOCKED makes this statement pass over rows another replica already holds instead of waiting for them. Every replica runs the same statement, so they fill worker capacity at once without two of them taking the same task. And if the executor has a hint that a task is ready, it matches that workload against free worker slots and sends it at once. For each selected row, the Astro Executor finds a waiting worker that serves the queue and still has a free slot. When it chooses worker-17 , it writes that worker's ID to external executor id , decrements the capacity held for the heartbeat, and builds the workload. The workload carries dag id , run id , task id , map index , the try number, details of the Dag bundle the versioned set of Dag files the worker runs from , queue and trace data, plus a signed token for the Airflow Execution API. The task assignment and worker identity live in the same metadata row that the scheduler already queued. There is no broker message to reconcile with that row. The worker receives the task in its heartbeat response once the assignment is written. The worker starts the process The worker receives its assignments in the heartbeat response. A local coordinator reserves a slot for each one, then writes the workload to the worker process. The worker validates the task data and forks an Airflow supervisor process. The parent records the child process ID and reports that the process started back to the local coordinator. The child calls Airflow's Task SDK supervisor with the Dag bundle, task instance, log path, signed token, and Execution API address. The Task SDK then starts the task runtime that imports the Dag and enters user code. Forking from a warm parent, while not unique to Astro, is part of what makes that start cheap. Before any task arrives, the worker parent imports the expensive parts of Airflow once the Task SDK, the task runner, the standard operators . Each forked task process inherits those imports instead of paying for them again. To make this as fast as possible, we have measured and optimized every single step and action between the workload arriving at the worker and the user's task code actually running. Before the task transitions to running , there's a final check with the API server to mark it as such, which also serves as a final check against the same task running on two workers at the same time. By this point, transform is running . It waited for a hint, a guarded assignment, one heartbeat response, and a fork from a warm parent—no broker. Airflow 3 makes this split possible. The task process does not need a database password or a route to the metadata database. Its short-lived token lets the supervisor use the Execution API for heartbeats, state changes, connections, variables, and XComs. The same contract means a worker can run anywhere that can reach the API server. Astro's remote execution is built on this: workers run inside the customer's own network—a cloud account, a data center, an on-prem GPU cluster—and pull work from an orchestration plane that never sees their code, logs, or data. When something fails, the database acts as the source of truth for how far the task has progressed and what needs to be done to recover. A worker that disappears while transform is still queued has lost only an assignment: recovery clears external executor id from its rows, and another healthy worker claims them. A partitioned worker that returns and starts its cleared assignment anyway cannot cause a double start: every task process must report running through the Execution API before it reaches user code, that transition is guarded like every other state change, and only one process wins it. A worker that disappears after the process reports running is harder, because user code may already have changed an external system; Airflow's heartbeat timeout marks the task failed and its retry rules decide whether another try runs. Airflow does not guarantee only-once execution so Dag authors make side effects safe to retry i.e., tasks are idempotent . If every worker is full, the task stays queued in the database until a worker has room. There is no in-memory list for an API server restart to lose. The Astro Scheduler normally publishes a wake-up hint after it queues work. That wakes the assignment loop without making the message broker the source of the task. The loop also checks for work on a short fallback interval, so a missed hint delays the assignment rather than losing it. The difference is how many hops sit between a queued row and a running process. On the same hardware, an Astro Executor worker can run 70% more concurrent tasks than a Celery worker—more task processes per worker, not faster user code. Combined with the Astro Scheduler, this execution path has sustained 500,000 concurrent tasks. The load tests at the end of this post give the test and the method behind that number. The Astro Hypervisor scales the system around transform The task path reaches user code only when each service has capacity at the same time: a worker needs a free slot, an API server must hold the heartbeat, and the Astro Scheduler must have room to own the Dag run. We originally built the Astro Hypervisor to help us manage the operations of many Airflow deployments in a single cluster. We run one Hypervisor service in every cluster to interact with potentially hundreds of Airflow deployments, without needing to go to every deployment individually. In doing so, it also acts as a translation layer—every Airflow version might have a slightly different metadata database structure and API format, so our Hypervisor acts as a central place to put logic to help decide what to do according to the Airflow version. It also helps with things like database connection pooling for efficiency. Kubernetes can scale a service from CPU or memory, but those measures describe load after it reaches a process. Airflow has earlier signals. It knows how many tasks are queued, how many Dag runs remain active, which triggers are waiting, which runs are due soon, and whether schedulers and workers still send heartbeats. The Astro Hypervisor turns that Airflow state into deployment-level decisions. Currently this works very well for the Airflow triggerer, which needs to scale rapidly because a single mapped task can result in thousands of deferred tasks to be placed on the triggerer. We have prototypes of scaling some of the other Airflow components, like the scheduler and API server, according to Airflow's state. Having an Airflow-specific hypervisor allows us to more intelligently make scaling decisions than if we were using a more general-purpose scaling system. For each deployment, the Hypervisor can read three groups of input: - The Airflow metadata database: task instance rows in queued or running grouped by queue, active and upcoming Dag runs, late runs, trigger rows, worker slots, scheduler heartbeats, and table sizes. - Kubernetes: pods, Deployments, custom resources, and current replica counts. - Service health and metrics: worker health, Astro Scheduler latency, and component-level metrics. Those facts can, in theory, lead to different actions: | Stored or observed fact | Potential Hypervisor action | An unfinished dag run exists | Keep scheduling services available | | A Dag's next run enters the look-ahead window | Start needed services before the run is due | Active rows remain in trigger | Keep a triggerer available | | A Kubernetes task pod has no matching active task instance, or belongs to an older try | Check whether the pod is safe to remove | | A required scheduler metric is missing | Block scheduler scale-down rather than treating it as 0 | The controller set depends on the deployment's Airflow version, executor, and features, so the Hypervisor does not apply one replica formula to every pod. Some of Airflow's features make autoscaling things like the scheduler very tricky; for example, a user can declare a "dynamic Dag" that changes every time it's parsed. Scheduler autoscaling is the piece still in prototype. We are working through these edge cases internally before we release it to customers. Scaling from planned work Before extract finishes, worker CPU metrics have no way to show that transform will soon become runnable, but the unfinished Dag run and dependency state in the metadata database already place it in planned work. This is where having a domain-specific hypervisor becomes interesting. The Hypervisor can keep schedulers active while a Dag run remains unfinished or the Dag model says another run is due soon, and can keep the triggerer active while trigger rows exist. We'll still use traditional autoscalers like KEDA, the Kubernetes autoscaler, but the Hypervisor adds Airflow context, can pause worker scaling during hibernation, and sequences scale-to-zero so one service does not vanish while another still needs it. Airflow has already written down the work that is coming, so the Hypervisor reads that instead of guessing from CPU metrics. What the Hypervisor repairs The Hypervisor can also detect late Dag runs, stale heartbeats, stuck tasks, slow parsing, large database tables, unhealthy workers, and Astro Scheduler latency. And because we operate thousands of production Airflow deployments, we know which failures repeat and exactly what state marks them, so the Hypervisor repairs those itself: - A Celery worker has free slots but has stopped accepting queued work. - A worker pod remains stuck in Terminating after its tasks are gone. - A Kubernetes task pod no longer matches an active task instance or try. Each repair first checks Airflow and Kubernetes state. If that check fails, the repair stops instead of acting on partial evidence. Other conditions become incidents for an operator or another system. Hibernation is an ordered stop The Hypervisor also turns hibernation schedules and manual overrides into one decision: it scales services to zero in dependency order and later restores their normal replica sources. Hibernation is not a lossless pause. Shutdown marks remaining Dag runs and tasks failed or skipped, removes task pods, and writes heartbeats for services stopped on purpose so health checks do not report the shutdown as a fault. That trade fits idle non-production deployments; a deployment with running production tasks keeps its dependencies active. Across the measured deployment group, triggerer scaling and the repair controls above cut Dag failures by 85%. The figure counts the failure classes the Hypervisor can repair; faults in user code, external services, or data remain the Dag author's. The software beneath the path The Astro Runtime image does its work before the Dag run arrives. It pins the Airflow and Task SDK versions and sets bounds for the providers, Python packages, database clients, and system code used by each service. Customer projects can add packages, but they start from a known set. We test that set across supported Python versions, processor architectures, base operating systems, databases, and executors. Release candidates pass package, Docker, CLI, Kubernetes, migration, and security tests. Supported releases also receive daily security scans and selected backports for key bugs and CVEs for two years. When a region fails So far, every failure has assumed the metadata database remains available. A message broker hint can disappear, a scheduler can stop, and a worker can lose its connection while another process reads the same stored state. A regional outage can remove the database, Kubernetes cluster, object storage, workers, and network paths at once. Astro cross-region recovery prepares a second dedicated cluster in another region of the same cloud. The primary cluster runs deployments during normal operation. The secondary remains ready for promotion. Three kinds of state move between them: - The Multi-Region DB copies deployment metadata, including Dag-run history, task-instance state, XComs, connections, variables, and configuration. - Multi-region object storage copies task logs. - Image replication makes user-deployed images available in the second region. Promoting the secondary cluster An Organization Owner begins failover through the Astro UI or API. Astro promotes the secondary cluster and starts the deployments there. Deployment IDs, settings, and service hostnames remain the same. Airflow UI and API traffic begins routing to the active cluster in the second region. Promotion must therefore restore more than a database replica. Tasks can fail during the switch. After promotion, an operator checks deployment health and retries work whose state requires it. For our example, the outcome depends on the last state committed before the outage: - If transform completed and reported success, the secondary region can use that state to consider publish . - If transform was queued but had not started, the active recovery deployment can assign it to a worker. - If transform was running when the primary failed, its process is gone. Its final database state and Airflow retry policy decide what happens next. - If user code changed an external system before failing, its retry still needs safe external writes. What recovery promises For deployments using the task-log replication SLA, Astro's recovery point objective is under 15 minutes and measures how far copied state may lag. The metadata database is usually seconds old; this matters most, because it holds the real state of the Dags, including what has and hasn't been run. Task logs stored in object storage are generally minutes as opposed to seconds. The recovery time objective is under one hour and measures how long the secondary cluster and its deployments take to become available. We constantly run our own internal benchmarks and verified both objectives under load in a cross-region test with more than 80 deployments and more than 1,250 concurrent task runs. Failback is the same kind of switch. Astro reverses replication so the former primary can catch up; once it has, an Organization Owner moves services and hostnames back, and tasks can again fail during the change. Astro manages the paired cluster, replication, promotion, stable service hostnames, and failback. Customers prepare regional network routes, private endpoints, identities, image credentials, and Dag settings, then check deployment health and retry failed work after a switch. We have now followed one task across the main failure boundaries: a lost hint, a lost worker, a failed pod, and a lost region. Each layer handles a larger failure. None changes the basic rule: durable task state belongs in the metadata database, while user code must remain safe to retry. What the full system changes In our benchmarks, we've pushed Astro to 500,000 sustained concurrent tasks, 228 milliseconds p95 at 100,000, and an 85% cut in Dag failures. No single test measures the whole architecture, so we tested its layers apart and together. Isolating the execution path We wanted to measure the Astro Scheduler and Astro Executor, not how fast Kubernetes could add a worker or how soon an autoscaler noticed load. Before each run, we started more worker slots than the target required. A 50,000-task run, for example, had more than 60,000 slots ready before the test began. We also provisioned enough API server capacity and raised the replica and resource limits that Astro exposes in the product, past the ceilings a user can set today. These runs measure the Astro Scheduler and Astro Executor, not the current product caps. That setup does not make the test free of infrastructure. Reaching these counts found limits elsewhere. Some database queries needed missing indexes before the control path could progress to the next load level. At high pod counts, the Kubernetes scheduler used for bin packing ran out of memory. We hit a vCPU ceiling on our Kubernetes cluster twice. We had to separate those failures from Astro Scheduler and Astro Executor behavior rather than counting every failed run as the same limit. The test holds a large group of long-sleeping tasks at a steady level, then triggers a small probe Dag. The first measure asks how long the first task in that new run takes to start. The second asks how long each following task waits after its upstream task finishes. This shows whether a deployment can begin and advance new work while hundreds of thousands of other tasks remain active. Task-start latency under load At 50,000 concurrent tasks, the standard scheduler with Celery recorded 23.582 seconds of p95 task-start latency. We stopped the Celery baseline there: task lag at that level already makes a deployment hard to operate, and pushing further mainly measures the queue growing. With the Astro Scheduler and Astro Executor, p95 was 228 milliseconds at 100,000 concurrent tasks—at twice the load, less than one-hundredth of the latency. At 300,000 concurrent tasks, p95 remained 294 milliseconds. Maximum concurrent tasks We also tested how much resident task state the execution path could manage. These tests use long-running tasks so concurrency rises and stays high instead of draining before the next group starts. The result measures how much running state the path can hold, not how many short tasks it can start each second. With the standard scheduler, the Astro Executor sustained 200,000 concurrent tasks. Combining it with the Astro Scheduler reached 500,000 sustained concurrent tasks. | Scheduler | Executor | Sustained concurrent tasks | | Standard Airflow | Astro Executor | 200,000 | | Astro Scheduler | Astro Executor | 500,000 | The first row separates Astro Executor capacity from Astro Scheduler capacity. At 200,000 tasks, the standard scheduler becomes the next limit. Replacing both halves of the path raises the result to 500,000, so this is not an Astro Executor result alone. Each result depends on the system we tested it on. Worker size, database capacity, service replicas, Dag shape, task duration, queue count, and settings all affect where the next limit appears. Back to transform Airflow still records its dependencies and state. The Astro Scheduler reacts when extract succeeds, takes ownership of the Dag run, and moves transform to queued through a guarded database update. The Astro Executor assigns it to a worker that has room. That worker starts an Airflow Task SDK process and reports state through the Execution API. The Astro Hypervisor reads the same deployment state to keep the needed services active. The Astro Runtime image supplies the tested software beneath them. If the whole region fails, the Multi-Region DB and secondary cluster preserve the stored record from which Airflow can recover. Throughout that path, the Dag and Airflow task states stay the same. Astro changes how quickly the system sees those states, how it assigns the work, and how the services around the path scale. We're really proud of the work we've done and don't plan to slow down with either the open-source Apache Airflow project nor our commercial products. The market is incredibly dynamic and we welcome this new class and scale of workloads that have emerged as LLMs have unlocked new workflow use cases and made it easier to produce code for the use cases we know and love. P.S. We're hiring. If you want to work on the systems in this post, we'd love to speak with you .