I could have written about another tool I’ve picked up on my data engineering journey, but I found something a bit more fundamental.
Recently, while exploring PySpark and building out a modular ETL pipeline, I caught myself looking at the data and asking:
"Why am I cleaning this? How is this different from the normal cleaning I do in data science?🤔"
I'd already spent plenty of time cleaning datasets for analysis and machine learning. But as I started building production-oriented pipelines, I realized something:
The transformations themselves might look similar, but the problems we're solving are often very different.
That was the mindset shift.
In data science, cleaning is usually driven by the needs of the analysis or model.
You might investigate outliers, handle missing values, remove duplicates, transform distributions, or engineer new features. The right approach depends heavily on the question you're trying to answer and the assumptions your model makes.
The goal isn't simply to make the data “clean.” It's to make the data useful and appropriate for the analysis.
Data engineering has a different set of constraints.
When you're building a pipeline that runs automatically and feeds downstream systems, you also have to think about things like schema consistency, data contracts, failure handling, scalability and reproducibility.
A pipeline can't rely on someone opening a notebook tomorrow morning and noticing that yesterday's data suddenly looks strange.
So the goal becomes reliable data processing.
It's less about choosing the universally “correct” way to clean a value and more about making sure the rules are explicit, repeatable, observable, and appropriate for the systems consuming the data.
Missing data is a problem in both disciplines. What you do about it depends on why the data is missing and what happens downstream.
In data science, you might impute a missing age using the median. If you're training a model, preserving the observation may be more useful than dropping the row.
In a data pipeline, the answer depends on the role of that field.
Suppose customer_id
is missing. If it's required to identify a customer, that record might need to be rejected or quarantined.
But what about a missing email
address?
If email is optional, there's probably no reason to reject the record. Let it through as NULL. Not every imperfect value is bad data.
The important thing is understanding which fields are required, which are optional, and what the downstream system expects.
This is probably where the difference becomes most obvious.
When exploring a dataset in a notebook, you might notice something strange:
“Why are there negative values in this column?”
You investigate, figure out what happened, update your transformation, and run the notebook again.That's perfectly reasonable during exploration.But imagine the same problem occurring in a pipeline that runs every night.You won't necessarily be there to notice it.
This is where data quality checks become part of the pipeline itself.
For example, you might define rules such as:
customer_id
must not be nullA missing primary key might be a critical error that causes a record to be quarantined while a missing email might simply be allowed and monitored.
An unexpected but recoverable schema change might trigger an alert rather than immediately bringing the entire pipeline down.
The important shift is that the expectations are encoded into the system rather than living only in the engineer's head or notebook.
This is probably the biggest distinction I took away.
In data science, cleaning is often part of preparing a dataset for a particular analysis.
In data engineering, transformations are also part of creating a reliable data product.
That means asking questions beyond:
“Is this data clean?”
You start asking:
“What assumptions am I making?”
“What happens if those assumptions are violated?”
“Should this record be transformed, rejected, quarantined, or allowed through?”
“Will this still work when the dataset is 100 times larger?”
“What happens when the upstream schema changes?” “Can someone else understand and reproduce what this pipeline is doing?”
That's the mindset shift.
The code might still contain familiar operations like filtering nulls, casting types, removing duplicates and transforming columns.
But the context has changed. You're no longer just cleaning a dataset. You're building a system that has to keep processing data reliably, even when the data isn't perfect.
This shift in mindset naturally led me to another question:
“If I already know how to clean and transform data with Pandas, why can't I just use Pandas for my data engineering pipelines?”🤷♀️