{"slug": "the-future-of-orchestration-pine59s-journey-to-airflow-3-on-google-cloud", "title": "The future of orchestration: Pine59’s journey to Airflow 3 on Google Cloud", "summary": "Pine59 migrated its data pipelines to Google Cloud's Managed Service for Apache Airflow (Gen 3) running Apache Airflow 3, after stress-testing production workloads showed immediate processing speed, task scheduling, and stability improvements. The location intelligence company, whose Daily Foot Traffic metric computes data for as many as 14 million distinct locations in a single job, moved ML inference onto a dedicated Google Kubernetes Engine cluster and built custom Airflow UI plugins including BigQuery Auto-linkify and DAG Run Configuration Search. Pine59 said the transition improved its MLOps capabilities, developer workflow, and pipeline speed across a monorepo containing hundreds of DAGs.", "body_md": "Operating large data pipelines requires an orchestration layer that scales smoothly as workloads expand. When your pipelines process millions of complex data points every day to feed predictive models, staying up-to-date with your technology stack is a strategic necessity.\n\n[Pine59](https://www.pine59.com/) provides location intelligence data through data pipelines that produce analytical metrics on cadences ranging from hourly to quarterly. One of the company’s most data-intensive metrics, Daily Foot Traffic, computes data for as many as 14 million distinct locations in a single job. To handle this massive volume, Pine59’s system runs entirely on Google Cloud, with the heavy lifting in [BigQuery](https://cloud.google.com/bigquery) and all of it orchestrated by [Managed Service for Apache Airflow](https://cloud.google.com/products/managed-service-for-apache-airflow) (formerly Cloud Composer) running Apache Airflow 3.\n\nAs the company’s volume of data and number of machine learning workloads scaled up, Pine59 decided to modernize its monorepo, which contains hundreds of directed acyclic graphs (DAGs). Here is a look at how that transition improved Pine59’s MLOps capabilities, developer workflow, and pipeline speed.\n\nPine59 has long relied on a shared monorepo with code and tooling spanning multiple projects to run its metric production pipelines. As it considered its infrastructure’s future, the company wanted to help its data pipelines run faster and more reliably.\n\nThat’s why it decided to stress-test production workloads against the newly available Managed Airflow (Gen 3) architecture running Airflow 3. The initial results were unambiguous: the Gen 3 environment delivered immediate and significant processing speed, task scheduling, and overall stability improvements. Recognizing the clear potential for performance gains, Pine59 initiated a full transition to the new environment.\n\nPine59’s pipelines don’t just move data; they drive complex ML models, so a core aspect of its migration was optimizing the orchestration of its ML inference workloads.\n\nPreviously, Pine59 had used standard Kubernetes operators for these tasks. By moving to Managed Airflow (Gen 3), which features a highly optimized and abstracted infrastructure layer, the company’s engineering team refined its MLOps architecture. They did so by setting up a dedicated [Google Kubernetes Engine](https://cloud.google.com/kubernetes-engine) (GKE) cluster that was specifically optimized for model inference and integrated it into the Pine59 pipelines.\n\nThis clear separation of orchestration and heavy ML execution compute allows data processing and model inference to run efficiently, showcasing Managed Airflow as a resilient, scalable backbone for enterprise MLOps.\n\nBeyond infrastructure improvements, Pine59 was also able to immediately capitalize on Airflow 3’s delivery of a vastly improved developer workflow and user interface. Indeed, managing hundreds of interconnected DAGs requires excellent observability, and Pine59 found Airflow 3’s plugin authoring system remarkably easy to use.\n\nTo improve internal developer velocity, the company quickly built a number of custom plugins that it integrated directly into its new Airflow UI:\n\n**BigQuery Auto-linkify:** A tool that automatically detects internal BigQuery table references within the Airflow Logs and XCom tabs, dynamically generating direct links to BigQuery Studio for faster debugging (available as a [public GitHub gist](https://gist.github.com/jan-hajny-unacast/74e1e504e3e3c8765323bd019a87fb30))\n\n**DAG Run Configuration Search:** A custom search form added directly to the DAG overview page. It allows Pine59 engineers to query specific key-value pairs within DAG run payloads (configs) and instantly surface matching runs. This in turn drastically reduces troubleshooting time.\n\nIn addition, the team also deployed a compatibility shim layer within its monorepo. This “compat” module dynamically abstracts logic between Airflow versions, streamlining operator migration across versions.\n\nFor Pine59, migrating to Managed Airflow (Gen 3) with Airflow 3 has yielded clear, quantifiable results.\n\nThe most important improvement was the speed of its DAG runs. In the company’s previous setup, tasks often got stuck in a queued state during peak processing surges. With Gen 3, queue latency has dropped dramatically, allowing tasks to start running almost immediately.\n\nConsider the comparison below of total aggregated “queued” & “running” time of more than 300 runs of the same DAG between Managed Airflow (Gen2) with Airflow 2.11 vs. Managed Airflow (Gen3) with Airflow 3.1 below. As we can readily see, the difference in queued time is significant.\n\nCoupled with internal DAG optimizations made during the transition, the performance gains are also highly tangible. For example, the Daily Foot Traffic pipeline previously took nearly 38 minutes to complete. With the new instance, the same workload now takes less than 26 minutes —nearly 32% less processing time.\n\nToday, Pine59 processes all its production workloads on its new Managed Airflow (Gen 3) instance. By moving to this next generation orchestration, the company improved its MLOps capabilities, equipped its developers with better tools, and built a faster, more resilient foundation for future workloads.\n\nIf your engineering team spends more time managing infrastructure than delivering value, consider a similar transition and discover how it can help you move from maintaining servers to building the future of your data and AI pipelines today.\n\n<sup>Special thanks to the following contributor to this post: Alexandre Crespo-Perez</sup>", "url": "https://wpnews.pro/news/the-future-of-orchestration-pine59s-journey-to-airflow-3-on-google-cloud", "canonical_source": "https://cloud.google.com/blog/topics/supply-chain-logistics/the-future-of-orchestration-pine59s-journey-to-airflow-3-on-google-cloud/", "published_at": "2026-09-17 17:00:00+00:00", "updated_at": "2026-09-17 18:54:57.899931+00:00", "lang": "en", "topics": ["mlops", "ai-infrastructure", "ai-tools", "developer-tools"], "entities": ["Pine59", "Google Cloud", "Managed Service for Apache Airflow", "Apache Airflow 3", "BigQuery", "Google Kubernetes Engine", "Daily Foot Traffic"], "alternates": {"html": "https://wpnews.pro/news/the-future-of-orchestration-pine59s-journey-to-airflow-3-on-google-cloud", "markdown": "https://wpnews.pro/news/the-future-of-orchestration-pine59s-journey-to-airflow-3-on-google-cloud.md", "text": "https://wpnews.pro/news/the-future-of-orchestration-pine59s-journey-to-airflow-3-on-google-cloud.txt", "jsonld": "https://wpnews.pro/news/the-future-of-orchestration-pine59s-journey-to-airflow-3-on-google-cloud.jsonld"}}