A technical team migrated its self-managed Apache Airflow orchestration from on-prem Linux boxes to Google Cloud Platform, building a self-managed Airflow VM plus a config service for under $150 a mon…
A developer outlines MLOps best practices for 2026, arguing that while model training has become trivial, deploying reliable, scalable, and compliant models is harder than ever amid agentic AI, the EU…
Astronomer's Astro Runtime sustained 500,000 concurrent Apache Airflow tasks in a single deployment and achieved 228 milliseconds p95 task-start latency at 100,000 concurrent tasks, compared to 23.582…
Google Cloud has released the Data Agent Kit, a free, open-source collection of data engineering and data science tools that integrates with IDEs and CLIs such as VS Code, Claude Code, and Codex, enab…
Bruin, an open-source data stack provider, launched a free program for early-stage startups that includes a planning call, a hands-on workshop with a Bruin engineer, and deployment on the startup's ow…
A context-assembly pipeline that turns raw issue tickets into verified briefs outperforms autonomous coding agents on real-world support queues, according to a design pattern published on dev.to this …
A developer explains Apache Airflow as a tool that lets users describe workflows as code, then schedules, runs, retries, and monitors them. The post covers core concepts like DAGs, tasks, operators, a…
Serial entrepreneur Qasar Younis announced on July 21, 2026, that his team has completed a major integration effort spanning more than nine years, connecting simulators, data engines, data pipelines, …
Prefect, a provider of a Python-based workflow automation platform, has agreed to acquire Dagster Labs, a declarative orchestration platform for data pipelines, to drive AI workflow automation. Prefec…
A developer tested four AI-powered data pipeline tools—dbt Cloud with Vanto AI, Apache Airflow 2.9 with Claude, Prefect 3.0, and Fivetran—over six weeks on a real microservices ETL system. dbt Cloud w…
A developer built a production-grade Retrieval-Augmented Generation (RAG) system called a Personal Health Digital Twin that transforms fragmented health data from sources like Apple Health CSVs, Oura …
Data engineering involves designing systems for collecting, storing, transforming, and managing data to support analytics, machine learning, and business decisions. The field relies on two primary ing…
Arize AI released the Arize AX Airflow Provider, an open-source Apache Airflow integration designed to automate the feedback loop for production LLM applications and AI agents. The provider includes o…
Chris Riccomini, a veteran software engineer and investor, argues that AI will soon handle the majority of data engineering work, potentially eliminating the distinct role of data engineer. In an inte…
AI-driven approach to improve failure detection and diagnosis in Apache Airflow, a tool for orchestrating ETL pipelines. The method combines large language models (LLMs) for classifying log messages, …