The future of orchestration: Pine59’s journey to Airflow 3 on Google Cloud Pine59 migrated its data pipelines to Google Cloud's Managed Service for Apache Airflow (Gen 3) running Apache Airflow 3, after stress-testing production workloads showed immediate processing speed, task scheduling, and stability improvements. The location intelligence company, whose Daily Foot Traffic metric computes data for as many as 14 million distinct locations in a single job, moved ML inference onto a dedicated Google Kubernetes Engine cluster and built custom Airflow UI plugins including BigQuery Auto-linkify and DAG Run Configuration Search. Pine59 said the transition improved its MLOps capabilities, developer workflow, and pipeline speed across a monorepo containing hundreds of DAGs. Operating large data pipelines requires an orchestration layer that scales smoothly as workloads expand. When your pipelines process millions of complex data points every day to feed predictive models, staying up-to-date with your technology stack is a strategic necessity. Pine59 https://www.pine59.com/ provides location intelligence data through data pipelines that produce analytical metrics on cadences ranging from hourly to quarterly. One of the company’s most data-intensive metrics, Daily Foot Traffic, computes data for as many as 14 million distinct locations in a single job. To handle this massive volume, Pine59’s system runs entirely on Google Cloud, with the heavy lifting in BigQuery https://cloud.google.com/bigquery and all of it orchestrated by Managed Service for Apache Airflow https://cloud.google.com/products/managed-service-for-apache-airflow formerly Cloud Composer running Apache Airflow 3. As the company’s volume of data and number of machine learning workloads scaled up, Pine59 decided to modernize its monorepo, which contains hundreds of directed acyclic graphs DAGs . Here is a look at how that transition improved Pine59’s MLOps capabilities, developer workflow, and pipeline speed. Pine59 has long relied on a shared monorepo with code and tooling spanning multiple projects to run its metric production pipelines. As it considered its infrastructure’s future, the company wanted to help its data pipelines run faster and more reliably. That’s why it decided to stress-test production workloads against the newly available Managed Airflow Gen 3 architecture running Airflow 3. The initial results were unambiguous: the Gen 3 environment delivered immediate and significant processing speed, task scheduling, and overall stability improvements. Recognizing the clear potential for performance gains, Pine59 initiated a full transition to the new environment. Pine59’s pipelines don’t just move data; they drive complex ML models, so a core aspect of its migration was optimizing the orchestration of its ML inference workloads. Previously, Pine59 had used standard Kubernetes operators for these tasks. By moving to Managed Airflow Gen 3 , which features a highly optimized and abstracted infrastructure layer, the company’s engineering team refined its MLOps architecture. They did so by setting up a dedicated Google Kubernetes Engine https://cloud.google.com/kubernetes-engine GKE cluster that was specifically optimized for model inference and integrated it into the Pine59 pipelines. This clear separation of orchestration and heavy ML execution compute allows data processing and model inference to run efficiently, showcasing Managed Airflow as a resilient, scalable backbone for enterprise MLOps. Beyond infrastructure improvements, Pine59 was also able to immediately capitalize on Airflow 3’s delivery of a vastly improved developer workflow and user interface. Indeed, managing hundreds of interconnected DAGs requires excellent observability, and Pine59 found Airflow 3’s plugin authoring system remarkably easy to use. To improve internal developer velocity, the company quickly built a number of custom plugins that it integrated directly into its new Airflow UI: BigQuery Auto-linkify: A tool that automatically detects internal BigQuery table references within the Airflow Logs and XCom tabs, dynamically generating direct links to BigQuery Studio for faster debugging available as a public GitHub gist https://gist.github.com/jan-hajny-unacast/74e1e504e3e3c8765323bd019a87fb30 DAG Run Configuration Search: A custom search form added directly to the DAG overview page. It allows Pine59 engineers to query specific key-value pairs within DAG run payloads configs and instantly surface matching runs. This in turn drastically reduces troubleshooting time. In addition, the team also deployed a compatibility shim layer within its monorepo. This “compat” module dynamically abstracts logic between Airflow versions, streamlining operator migration across versions. For Pine59, migrating to Managed Airflow Gen 3 with Airflow 3 has yielded clear, quantifiable results. The most important improvement was the speed of its DAG runs. In the company’s previous setup, tasks often got stuck in a queued state during peak processing surges. With Gen 3, queue latency has dropped dramatically, allowing tasks to start running almost immediately. Consider the comparison below of total aggregated “queued” & “running” time of more than 300 runs of the same DAG between Managed Airflow Gen2 with Airflow 2.11 vs. Managed Airflow Gen3 with Airflow 3.1 below. As we can readily see, the difference in queued time is significant. Coupled with internal DAG optimizations made during the transition, the performance gains are also highly tangible. For example, the Daily Foot Traffic pipeline previously took nearly 38 minutes to complete. With the new instance, the same workload now takes less than 26 minutes —nearly 32% less processing time. Today, Pine59 processes all its production workloads on its new Managed Airflow Gen 3 instance. By moving to this next generation orchestration, the company improved its MLOps capabilities, equipped its developers with better tools, and built a faster, more resilient foundation for future workloads. If your engineering team spends more time managing infrastructure than delivering value, consider a similar transition and discover how it can help you move from maintaining servers to building the future of your data and AI pipelines today.