With data engineering and 15 years of building Data Orchestration tools like Airflow, Dagster, Prefect, and Kestra, we are now doing the same for AI agents. There were times when new AI Orchestrators came out almost daily. Sure, you might deal with embeddings instead of datasets, but the nature of scheduling, logging the errors, restartability, etc., is all the same. ^8a63b1
As in tech, we like to reinvent the wheel- I wrote a
whole book about [Repeated Terms in Tech], and it’s [Convergent Evolution]- I feel we are reinventing once more.
# Before We Did it for Vector Databases and It’s Orchestration #
I already felt that with [Vector Databases], see my write-up at Don’t Build a Parallel Stack: Integrate Vectors into Data Engineering Workflow:
The data engineering lifecycle defines the end-to-end data engineering process, addressing all different components. When integrating vector operations into this lifecycle, we should aim to enhance rather than duplicate existing infrastructure. Vector operations should complement, not replace, your well-established data engineering practices.
Just as we don’t replace existing data connectors (like ODBC/JDBC) with each new technology wave, we shouldn’t create an entirely separate infrastructure for AI workloads. Instead, we should leverage existing tools like orchestration, scheduling, and processing frameworks while adding vector capabilities where they provide clear benefits.
This approach prevents duplication, maintains consistency, and leverages your team’s existing expertise. The goal should be to add vector storage and processing capabilities within your existing data engineering cycle, not to build a parallel system.
I continue with Don’t Repeat Yourself with AI Data Pipelines, and also my book on Patterns of Data Engineering started for that reason. Still, I guess it’s easier to build something new than to use existing tools. Obviously, the new tools also have their advantages in the narrow use case, but on the broader level, it’s an orchestrator IMO 🙂.
# Further Reads #
- I write more at [Working with AI in Data Engineering]
- [The Hype Cycles of AI](/brain/the-hype-cycles-of-ai/) and[Hype Cycle of Data Engineering]
Origin: [AI Orchestrators](/brain/ai-orchestrators/#are-we-reinventing-the-tools)