I get asked some version of “how do I break into data engineering” at least once a week, usually in my DMs, sometimes at meetups where someone corners me near the coffee machine. And honestly, my answer has changed a lot over the past two years. Not because the fundamentals shifted, SQL is still SQL, but because the shape of the job has moved. Five years ago a data engineer mostly wrote batch ETL jobs and prayed the cron didn’t fail overnight. Now you’re expected to understand streaming, semi-structured data, cost optimization on cloud warehouses, and increasingly, how your pipelines feed into LLM applications. That’s a lot to throw at someone starting from zero.
So here’s my attempt at an honest, unglamorous roadmap. No “10x your career in 30 days” nonsense. Just what actually matters, roughly in the order you should learn it, with the reasoning behind each step.
Most roadmaps floating around LinkedIn are recycled versions of a 2019 blog post with “2026” pasted into the title. They’ll tell you to learn Hadoop (please don’t, unless you’re maintaining legacy systems at a bank), memorize every Airflow operator, and call it a day. That advice isn’t wrong exactly, it’s just incomplete. The market now wants engineers who can reason about data as a product, not just plumbing. Someone who understands that a broken pipeline three hops upstream can silently poison a machine learning model’s training data is worth more than someone who can recite Spark configuration flags from memory.
I’ll admit there’s some tension here. You do need the plumbing skills, deeply. But you also need enough architectural thinking to know why the plumbing is arranged the way it is. Let’s start with the boring stuff, because skipping it is the single biggest mistake I see.
Everyone says “learn SQL.” Fine, but what does that actually mean? It means being comfortable with window functions without googling the syntax every time. It means understanding why a LEFT JOIN followed by a WHERE clause on the right table quietly turns into an inner join (this trips up more people than it should). It means knowing when a CTE is clearer than a subquery, and when it isn't.
Practice on something real. I usually tell people to grab a messy dataset, maybe NYC taxi trip data or a Kaggle e-commerce dump, and try to answer questions that require actual thought: “what’s the month-over-month retention cohort for customers who made their first purchase in a discount period?” That kind of question forces you into window functions, date arithmetic, and self-joins all at once.
You don’t need to know scikit-learn cold. You need to be fluent in writing scripts that read from an API, handle pagination, retry on failure, and write clean records somewhere. Learn requests, learn how generators work (they matter a lot when you're streaming large files instead of them into memory), and get comfortable with pandas even though, controversially, I think most production pipelines shouldn't lean on pandas for anything beyond quick exploration.
I say this every time and people roll their eyes, but you will be SSH-ing into something eventually, whether it’s an EC2 box or a Kubernetes pod, and fumbling around with grep, awk, and basic shell scripting under pressure is a bad look. Spend a weekend on this. It pays off for years.
This is where a lot of self-taught engineers have a gap, in my experience. They can move data but they haven’t internalized why you’d structure a warehouse a certain way.
Learn the difference between Kimball-style dimensional modeling (facts and dimensions, star schemas) and the newer wide, denormalized approach that modern columnar warehouses like Snowflake and BigQuery tend to favor because storage is cheap and joins across huge tables are expensive. Neither approach is universally “correct.” Kimball still makes a lot of sense for BI-heavy orgs with well-understood reporting needs. Wide tables shine when you’re serving fast dashboards or feeding ML feature stores. Knowing which trade-off applies to your situation is the actual skill, not just knowing the vocabulary.
Here’s a simplified view of how data typically flows from raw sources into a modeled warehouse layer:
Notice the layering: raw data lands untouched (so you can always reprocess if something upstream breaks), staging cleans and standardizes it, and the mart layer is what analysts and BI tools actually touch. This “medallion” style pattern, sometimes called bronze/silver/gold, shows up constantly whether you’re on Databricks, Snowflake, or a homegrown stack. It’s not a trend, it’s just a sensible way to isolate failure.
This is the part everyone’s actually curious about. Extract, load, transform (notice the order, it flipped from the old ETL world once cloud warehouses got cheap and fast enough to do transformation after raw data).
Extraction and : tools like Fivetran, Airbyte, or Meltano handle the unglamorous work of pulling data from Salesforce, Postgres, Stripe, whatever, and dumping it into your warehouse. You don’t need to master all of them. Pick one open-source option (Airbyte is a reasonable choice since it’s free to run locally) and understand the pattern: connectors, incremental syncs, schema drift handling.
Transformation: this is dbt’s territory, and I don’t think that’s changing anytime soon despite some newer entrants nibbling at the edges. dbt made “transformation as software engineering” mainstream, meaning your SQL models get version control, testing, documentation, and lineage tracking, the stuff that used to only exist in application codebases. If you’ve read my dbt cheat sheet, you already know I think this is one of the higher-leverage tools to actually get fluent in, not just aware of.
Orchestration: Airflow is still the incumbent, though Dagster and Prefect have earned real market share by fixing some of Airflow’s rougher edges (asset-based thinking instead of pure task graphs, for instance). I wouldn’t stress too much over which one you learn first. The concepts transfer. DAGs, retries, sensors, backfills, these ideas are the same regardless of which tool’s syntax you’re typing.
Here’s roughly how these pieces connect in a typical modern stack:
I’ll be honest, this diagram is a simplification. Real pipelines have error handling, dead-letter queues, alerting hooks, and probably three Slack integrations nobody remembers setting up. But as a mental model for how the pieces fit, it holds up.
Batch processing will not disappear, whatever the conference talks imply. But an increasing share of jobs now expect at least conversational fluency with streaming concepts, and for some roles, hands-on experience.
Start with Kafka, not because it’s the only option (Kinesis, Pulsar, and managed alternatives exist) but because its concepts, topics, partitions, consumer groups, offsets, map cleanly onto almost every other streaming system you’ll encounter later. Understand the difference between at-least-once and exactly-once delivery semantics, and why exactly-once is genuinely hard to guarantee end to end, not just a checkbox some vendor ticks on a feature comparison page.
If you want a project that actually teaches you something, try building a small pipeline that ingests a simulated event stream (user clickstream data works well), processes it with something like Kafka Streams or Flink, and lands aggregated results into a warehouse table that updates every few minutes. That single project touches more real skills than most tutorials manage in twenty hours of video. Pick one cloud provider and go deep rather than skimming all three. I’d lean toward AWS or GCP depending on what’s more common in your target job market (AWS still dominates enterprise, GCP has strong footing in data-native companies given BigQuery’s reputation). Learn IAM basics, because access control mistakes cause more real incidents than people admit publicly. Get comfortable with infrastructure as code, Terraform specifically, since manually clicking through a console doesn’t scale and won’t survive an interview question about reproducibility.
Docker and basic Kubernetes literacy matter too, though you don’t need to become a platform engineer. You just need to understand why your Airflow worker pods keep getting OOM-killed at 2am (RAM limits set too conservatively, usually, in my experience).
Nobody puts this on roadmaps because it’s not flashy, but it’s increasingly where senior data engineers spend real time. Learn dbt tests, or a dedicated tool like Great Expectations or Soda. Understand what data contracts are and why some teams are moving toward enforcing schema agreements between producers and consumers before data even lands in the warehouse, rather than discovering a breaking schema change three dashboards downstream.
This matters more than ever now that so much pipeline output eventually feeds retrieval systems and LLM applications. Garbage in, hallucination-adjacent nonsense out, more or less. If you’re the engineer who catches a silently corrupted embedding source before it poisons a RAG pipeline, that’s the kind of quiet competence that gets you promoted, even if nobody claps for it in the moment.
By the time you’ve worked through the stages above, you should be able to look at something like this and understand every box in it, not just recognize the logos:
Sources feed extraction tools, extraction lands raw data in cloud storage or a warehouse, orchestration coordinates transformation jobs, quality checks gate what moves forward, and the output serves BI tools, ML feature stores, or increasingly, retrieval layers for LLM applications. It’s not a small system. Nobody builds all of it alone, and that’s worth saying out loud, because junior engineers sometimes feel like they need to master the entire diagram before applying to jobs. You don’t. You need to understand where each piece fits and be genuinely strong in two or three of them.
I want to push back gently on my own roadmap here, because no single path fits everyone. If you’re coming from a software engineering background, you’ll probably move faster through the Python and infrastructure sections and slower through data modeling, since dimensional thinking isn’t something backend engineers usually practice. If you’re coming from analytics, the reverse tends to be true, SQL and modeling feel familiar, but production-grade Python and orchestration concepts take longer to click.
Also, worth saying plainly: certifications help less than portfolio projects. A recruiter skimming your resume for six seconds is more likely to at “built a real-time pipeline processing 50k events per minute” than at a badge from a cloud certification exam, however much time that exam cost you.
There isn’t a finish line here, which is either the most annoying or most exciting part of this field, depending on your mood that day. The tools will keep shifting. Dbt might get real competition, some new orchestrator might eat Airflow’s lunch, who knows. But the underlying reasoning, how data should flow, where it can break, how to make failure visible instead of silent, that stays remarkably stable. Learn that reasoning deeply, and the tool-hopping becomes a lot less stressful.
If you’re just starting this journey, don’t try to do all six stages simultaneously. Pick stage one, get uncomfortable with SQL for a month, and build from there. If this was useful, I write regularly about the modern data stack, dbt, and AI infrastructure. Follow along for more deep dives.
Roadmap to Become a Data Engineer in 2026 was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.