{"slug": "are-we-reinventing-the-tools-of-data-engineering-for-ai", "title": "Are We Reinventing the Tools of Data Engineering for AI?", "summary": "Data engineering practitioner and author ssp.sh argues that the emerging wave of AI orchestrators is repeating the same scheduling, error-logging, and restartability patterns already solved by 15 years of data orchestration tools such as Airflow, Dagster, Prefect, and Kestra. The author contends that vector operations and AI workloads should be integrated into existing data engineering lifecycles rather than built as a parallel stack, citing prior write-ups on vector databases and AI data pipelines. The piece frames this as convergent evolution, noting that new tools offer narrow advantages but remain orchestrators at the broader level.", "body_md": "# Are We Reinventing the Tools of Data Engineering for AI?\n\nWith data engineering and 15 years of building [Data Orchestration](/brain/data-orchestrators/) tools like Airflow, Dagster, Prefect, and Kestra, we are now doing the same for AI agents. There were times when new [AI Orchestrators](/brain/ai-orchestrators/) came out almost daily. Sure, you might deal with embeddings instead of datasets, but the nature of scheduling, logging the errors, restartability, etc., is all the same. ^8a63b1\n\nAs in tech, we like to reinvent the wheel- I wrote a \n[whole book](https://www.dedp.online/part-1/1-introduction/_intro-data-engineering.html) about [Repeated Terms in Tech], and it’s [Convergent Evolution]- I feel we are reinventing once more.\n\n## # Before We Did it for Vector Databases and It’s Orchestration\n\nI already felt that with [Vector Databases], see my write-up at \n[Don’t Build a Parallel Stack: Integrate Vectors into Data Engineering Workflow](https://www.ssp.sh/blog/vector-technologies-ai-data-stack/#dont-build-a-parallel-stack-integrate-vectors-into-data-engineering-workflow):\n\nThe data engineering lifecycle defines the end-to-end data engineering process, addressing all different components. When integrating vector operations into this lifecycle, we should aim to enhance rather than duplicate existing infrastructure. Vector operations should complement, not replace, your well-established data engineering practices.\n\nJust as we don’t replace existing data connectors (like ODBC/JDBC) with each new technology wave, we shouldn’t create an entirely separate infrastructure for AI workloads. Instead, we should leverage existing tools like orchestration, scheduling, and processing frameworks while adding vector capabilities where they provide clear benefits.\n\nThis approach prevents duplication, maintains consistency, and leverages your team’s existing expertise. The goal should be to add vector storage and processing capabilities within your existing data engineering cycle, not to build a parallel system.\n\nI continue with \n[Don’t Repeat Yourself with AI Data Pipelines](https://www.ssp.sh/blog/vector-technologies-ai-data-stack/#dont-repeat-yourself-with-ai-data-pipelines), and also my book on \n[Patterns of Data Engineering](https://www.dedp.online/) started for that reason. Still, I guess it’s easier to build something new than to use existing tools. Obviously, the new tools also have their advantages in the narrow use case, but on the broader level, it’s an orchestrator IMO 🙂.\n\n## # Further Reads\n\n- I write more at [Working with AI in Data Engineering]\n- [The Hype Cycles of AI](/brain/the-hype-cycles-of-ai/) and[Hype Cycle of Data Engineering]\n\nOrigin: [AI Orchestrators](/brain/ai-orchestrators/#are-we-reinventing-the-tools)", "url": "https://wpnews.pro/news/are-we-reinventing-the-tools-of-data-engineering-for-ai", "canonical_source": "https://www.ssp.sh/brain/are-we-reinventing-the-tools-of-data-engineering-for-ai/", "published_at": "2026-09-14 09:31:09+00:00", "updated_at": "2026-09-14 10:06:42.267463+00:00", "lang": "en", "topics": ["ai-infrastructure", "ai-tools", "mlops", "ai-agents"], "entities": ["Airflow", "Dagster", "Prefect", "Kestra"], "alternates": {"html": "https://wpnews.pro/news/are-we-reinventing-the-tools-of-data-engineering-for-ai", "markdown": "https://wpnews.pro/news/are-we-reinventing-the-tools-of-data-engineering-for-ai.md", "text": "https://wpnews.pro/news/are-we-reinventing-the-tools-of-data-engineering-for-ai.txt", "jsonld": "https://wpnews.pro/news/are-we-reinventing-the-tools-of-data-engineering-for-ai.jsonld"}}