When I started my first data analytics and machine learning project, I wanted to get straight to the exciting part: training a model.
I downloaded a dataset from Kaggle, chose a basic classification model, trained it, and started making predictions.
But the results were strange.
My predictions didn't look right, and metrics such as accuracy and ROC-AUC were much lower than I expected.
My first thought was:
Maybe I chose the wrong model.
But the problem started before the model.
I hadn't properly understood or prepared my data.
Instead of immediately switching algorithms, I went back to the dataset.
I started with:
df.info()
This gave me an overview of my columns, their data types, and how many non-null values were present.
Next:
df.isna().sum()
This showed me exactly where values were missing.
Then came an important lesson:
Finding missing data and deciding what to do with it are two different things.
Depending on the dataset and what the missing value represents, one option for a numerical column is to fill missing values using the median:
df["column"].fillna(df["column"].median())
The median can be useful because it is less affected by extreme values than the mean.
For categorical data, one possible approach is the **mode**:
`df["column"].fillna(df["column"].mode()[0])`
`[0]`
?Because `mode()`
returns a Series. There can be more than one mode, and [0]
selects the first result.
Sometimes, however, filling isn't appropriate at all.
I also learned about:
`df.drop()`
`df.dropna()`
And data cleaning wasn't only about missing values.
I needed to make sure my columns had the appropriate data types.
That's where tools such as these became useful:
`df.convert_dtypes()`
`df["column"].astype("string")`
`pd.to_datetime(df["date"])`
These commands weren't the exciting machine-learning model I originally wanted to build.
But I started realizing something:
This was part of machine learning too.
The phrase ** “garbage in, garbage out”** finally started making sense.
A machine-learning algorithm doesn't automatically understand that a column has the wrong data type, that missing values have been handled poorly, or that the data doesn't represent what I think it represents.
The model learns from what I give it.
I had been worrying about finding a better algorithm when I hadn't properly prepared the foundation.
While studying Genesis 1, I had written something in my notes: “Building up bit by bit.”
Genesis 1 doesn't move immediately from a formless and empty earth to the finished creation.
There is progression.
Light.
Separation.
Land.
Vegetation.
The lights in the sky.
Living creatures.
And finally, humanity.
There is order to the chapter.
Of course, training a machine-learning model isn't comparable to God's act of creation.
But Genesis 1 made me reflect on something about how I build.
I had wanted this:
Dataset → Model → Prediction
But my project actually required:
Dataset → Understand → Clean → Prepare → Train → Evaluate
I was rushing toward the result while overlooking the work that needed to happen first.
At first, data cleaning felt like the boring part standing between me and “real” machine learning.
I don't see it that way anymore.
Preparation is part of building.
Sometimes progress means checking one column.
Finding one missing value.
Correcting one data type.
Understanding one problem.
Then moving to the next.
Build carefully.
Build in order.
Build bit by bit.