A few months back, a colleague of mine, a data engineer with maybe eight years of pipeline scars to show for it, asked me something that stuck around in my head longer than it probably should have. He said, “I know SQL, I know dbt, I’ve built Airflow DAGs that would make you weep with joy. So why does every AI conversation at work make me feel like I wandered into someone else’s meeting?”
That question is more common than people admit out loud. Data engineers built the infrastructure that modern AI runs on, and yet a strange number of us feel like outsiders the moment “embeddings” or “RAG” gets mentioned in a planning call. Which is a bit absurd, honestly, because if you’ve ever built an incremental load pipeline, you already understand half the hard problems in AI infrastructure. You just haven’t been told that yet.
So I’m starting a 20-part series here. Not a rushed one, no single 3,000-word mega-post trying to cram vector databases and fine-tuning and prompt engineering into one sitting. That approach never works, people skim it, save it for later, and later never comes. Instead, this is a slow build, one solid piece at a time, aimed specifically at engineers who already know data but feel a step behind on the AI layer sitting on top of it.
Most “AI for engineers” content out there assumes you’re starting from nothing, or worse, it assumes you’re a machine learning researcher who just needs a refresher. Neither is true for most of us. You already know how to move data reliably. You already understand schema evolution, partitioning, and why a naive full-table scan at 3am can ruin someone’s morning. What’s missing usually isn’t technical aptitude. It’s vocabulary, mental models, and a map of how these AI-native concepts actually connect back to the pipelines you’re already responsible for.
Here’s roughly how I see the territory breaking down, and honestly, drawing this out helped me almost as much as it’ll help you:
Five clusters, twenty parts. Not because twenty is a magic number, but because trying to compress this into ten would mean skipping things that genuinely trip people up, like the difference between fine-tuning and retrieval, which I still see conflated in serious engineering discussions.
This is where we build the bridge. Part 1 covers how LLMs actually consume data, tokens, context windows, why a document that looks small to you might blow past a model’s limit once it’s tokenized. Part 2 gets into embeddings, and I’ll be honest, I think most explanations of embeddings online are needlessly abstract. I’ll try to explain them the way I’d explain a hash function to someone who already understands hashing, similar inputs, similar output space, except the “similarity” here is semantic rather than exact.
Part 3 is about vector databases (Pinecone, Weaviate, pgvector, that whole family) and why they’re really just a specialized indexing problem, one you’ve arguably solved before in different clothing if you’ve ever tuned a search index. Part 4 closes the foundations with a look at how traditional ETL concepts map onto “AI data pipelines,” because spoiler, they map more cleanly than most tutorials suggest.
Here’s where things get properly engineering-flavored. Chunking strategies, why splitting a document into fixed-size chunks is often lazier than it should be, and what smarter chunking (semantic, recursive, whatever you want to call it) actually buys you. Data freshness for retrieval systems, which is a genuinely underrated problem, a RAG system serving three-week-old pricing data because nobody rebuilt the embedding index is not a hypothetical, I’ve seen it happen twice this year alone.
We’ll also get into metadata filtering, hybrid search (combining keyword and vector search, because pure semantic search alone is often worse than people expect), and monitoring data quality specifically for AI consumption, which is a different beast from monitoring a BI dashboard’s underlying tables.
This cluster is basically “Airflow, but for AI.” How do you orchestrate embedding refresh jobs. What does a retraining or re-indexing pipeline look like when your source data changes daily but re-embedding everything is expensive. We’ll talk about batching strategies for API calls to model providers (rate limits are real, and they will humble you if you don’t plan for them), and cost management, because a poorly designed pipeline calling an LLM API on every row of a million-row table will produce a bill that gets escalated to someone’s VP awfully fast.
This part probably won’t feel entirely comfortable at first, and that’s fine. We’re moving into evaluation, how do you actually know if your RAG system’s answers are good, beyond someone eyeballing a demo and saying “looks fine to me.” Structured output parsing, handling the model occasionally returning malformed JSON (it happens more than vendors like to admit), and building feedback loops that treat model output quality the way you’d treat data quality, with tests, thresholds, and alerts, not vibes.
The last stretch is deliberately less glamorous. Security and data governance for AI systems, because feeding sensitive customer data into a third-party model API without proper controls is the kind of mistake that ends careers, not just projects. Observability specifically for AI pipelines, since traditional pipeline monitoring often misses failure modes unique to this space, like silent embedding drift. And finally, a closing piece on where this is all likely headed, written with appropriate humility, since anyone claiming certainty about AI infrastructure eighteen months out is probably overselling something.
I want to be upfront about something. This series leans toward retrieval-based AI systems (RAG, semantic search, that family of problems) rather than deep dives into model training or fine-tuning internals. That’s a deliberate choice, not an oversight. Most data engineers, in my experience, will spend far more of their career building and maintaining the data layer that feeds AI applications than actually training models from scratch. If you’re hoping for a deep dive into transformer architecture internals, there are better resources out there already, and I’ll try to point toward some of them along the way rather than pretending this series covers everything.
There’s also a fair criticism that a 20-part series risks losing momentum, mine or yours. I’ve seen enough abandoned “part 1 of 10” series on Medium to take that risk seriously. So here’s my commitment: each part will stand alone well enough that you can read it independently if you stumble onto part 12 first without having read parts 1 through 11. Cross-references, sure, but no artificial cliffhangers forcing you to read in strict order.
If you’re a data engineer who’s felt a little sidelined by the AI conversation happening around you, this series is built with you specifically in mind, not the machine learning researcher, not the prompt engineering hobbyist, but the person who already knows what a good pipeline looks like and just needs the AI-specific vocabulary and architecture patterns layered on top. Part 1 goes up soon, on how LLMs actually consume data and why your intuitions about “just add more context” are usually wrong in ways that matter for real system design.
AI for Data Engineers was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.