How I applied Exploratory Data Analysis, Feature Engineering, Pipelines, and Ensemble Models to solve a real-world machine learning problem—and the lessons I learned along the way.
There comes a point in every machine learning learner's journey when watching tutorials and completing small practice exercises are no longer enough.
After spending weeks understanding statistics, exploratory data analysis (EDA), feature engineering, preprocessing techniques, and classical machine learning algorithms, I wanted to answer one question:
Can I apply everything I've learned to a real machine learning competition?
That's when I decided to participate in a Kaggle Playground competition.
Unlike classroom datasets, Kaggle competitions force you to think like a machine learning engineer. You're responsible for understanding messy data, building preprocessing pipelines, selecting models, evaluating performance, debugging errors, and finally creating a submission that competes with thousands of participants.
This article documents my complete journey—from the dataset to building production-style preprocessing pipelines and training multiple ensemble models. Along the way, I'll also share the challenges I faced, what worked well, and the lessons I'll carry into future competitions.
Learning machine learning isn't just about knowing algorithms.
Real-world ML requires answering questions like:
Kaggle provides an environment where all of these questions matter.
Instead of building a model that works only inside a notebook, you're solving a problem under realistic constraints and evaluating your solution on unseen data.
The objective of this Playground competition was to predict the target class based on a combination of numerical and categorical features related to health and lifestyle.
The workflow followed the same structure used in many real-world machine learning projects.
Raw Dataset
│
▼
Exploratory Data Analysis
│
▼
Missing Value Analysis
│
▼
Feature Engineering
│
▼
Preprocessing Pipeline
│
▼
Model Training
│
▼
Evaluation
│
▼
Prediction
│
▼
Kaggle Submission
Although the workflow appears simple, every stage requires careful decision-making.
The first step was both the training and testing datasets using Pandas.
train = pd.read_csv("train.csv")
test = pd.read_csv("test.csv")
Before writing a single machine learning model, I spent time understanding the structure of the data.
Some of the questions I explored were:
Skipping this stage often leads to poor model performance because preprocessing decisions depend entirely on the characteristics of the data.
EDA turned out to be one of the most valuable parts of the project.
Instead of immediately training a model, I wanted to understand how the dataset behaved.
I separated the features into two broad categories:
Examples included:
Examples included:
This separation made it much easier to apply different visualization techniques.
For numerical columns, I explored the data using:
These visualizations helped answer important questions.
Some variables showed distributions close to normal, while others exhibited noticeable skewness.
Boxplots revealed the presence of outliers in several numerical features.
Using skewness calculations together with KDE plots made it easier to understand whether transformations might be useful.
Instead of blindly preprocessing the data, these visualizations allowed me to make decisions based on evidence.
One of the first analyses I performed was checking the percentage of missing values across the dataset.
train.isnull().mean() * 100
Missing values are one of the most common problems in real-world datasets.
Ignoring them isn't an option because most machine learning algorithms cannot train with incomplete data.
Rather than replacing every missing value with a single constant, I decided to use appropriate preprocessing techniques later in the pipeline.
For categorical variables, I created count plots to understand the frequency distribution of each category.
These plots answered questions like:
I also explored relationships between different categorical variables using grouped visualizations.
This helped me develop an intuition about the dataset before training any model.
One lesson became very clear during EDA:
Better understanding often leads to better preprocessing.
Machine learning isn't just about choosing a powerful algorithm.
The quality of the input data has an enormous impact on the quality of the final predictions.
That's why I spent significant time exploring the data before moving on to feature engineering.
After understanding the dataset, the next step was preparing it for machine learning.
The first task was separating features and the target variable.
X = train.drop(columns=["health_condition"])
y = train["health_condition"]
This created a clear distinction between:
X
)y
)I then split the training data into training and validation sets.
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(
X,
y,
test_size=0.2,
random_state=42
)
Creating a validation set allowed me to evaluate model performance before making a Kaggle submission.
Different feature types require different preprocessing strategies.
For example:
| Feature Type | Preprocessing |
|---|---|
| Numerical | Imputation + Scaling |
| Ordinal Categorical | Ordinal Encoding |
| Nominal Categorical | One-Hot Encoding |
| Target Variable | Label Encoding |
Applying the wrong preprocessing technique can significantly reduce model performance.
Instead of manually transforming every column, I wanted a cleaner and more reusable solution.
That's where Scikit-learn Pipelines became extremely useful.
I divided the features into logical groups.
Continuous numerical variables requiring scaling and missing value handling.
Features with a natural ordering, such as activity or stress levels.
These can be safely converted into ordered numerical values.
Features such as gender or diet type have no natural ordering.
Assigning arbitrary numbers could introduce unintended relationships, so One-Hot Encoding is a better choice.
This separation made the preprocessing pipeline much easier to maintain.
Rather than writing preprocessing code repeatedly for every model, I used Scikit-learn's Pipeline
and ColumnTransformer
.
This approach provides several advantages:
Instead of manually applying transformations one by one, the pipeline automatically performs every preprocessing step before model training.
This is one of the biggest improvements I made compared to my earlier machine learning projects.
One of the biggest improvements I made in this project compared to my earlier machine learning exercises was moving away from manual preprocessing.
Instead of writing separate preprocessing code for every model, I built a reusable pipeline using Scikit-learn.
This approach follows the same philosophy used in production machine learning systems.
Rather than remembering dozens of preprocessing steps, everything is automated inside a single workflow.
Different feature types require different transformations.
My preprocessing workflow looked like this:
Raw Dataset
│
▼
Separate Numerical & Categorical Features
│
├───────────────┐
│ │
▼ ▼
Numerical Categorical
│ │
KNN Imputer Simple Imputer
│ │
StandardScaler Encoding
│ │
└──────┬────────┘
▼
ColumnTransformer
▼
Machine Learning Model
Instead of transforming every column manually, the pipeline automatically applied the correct preprocessing to each feature type.
This made the code cleaner, reusable, and less error-prone.
Missing values are unavoidable in real-world datasets.
Instead of dropping rows and losing valuable information, I applied different imputation strategies depending on the feature type.
For numerical features, I used KNN Imputer.
The idea behind KNN Imputation is simple.
Instead of filling missing values with a mean or median, it looks for the most similar observations and estimates the missing value based on those neighbors.
For categorical features, I used Simple Imputer with the most frequent category.
This preserved the integrity of categorical data while avoiding unnecessary data loss.
Machine learning models cannot understand text directly.
Therefore, categorical variables needed to be converted into numerical representations.
I divided categorical columns into two groups.
These have a natural order.
Examples include activity levels or stress levels.
For these variables, I used Ordinal Encoding.
Low → 0
Medium → 1
High → 2
Since these categories have meaningful ordering, ordinal encoding preserves that relationship.
Some features have no natural ordering.
Examples include gender or diet type.
Assigning numbers such as
Male = 0
Female = 1
could unintentionally imply a mathematical relationship.
Instead, I applied One-Hot Encoding, where each category receives its own binary column.
This prevents the model from assuming nonexistent order.
Without a ColumnTransformer
, preprocessing quickly becomes difficult to manage.
Different feature groups require different transformations.
Instead of manually applying each transformation, ColumnTransformer
allows everything to happen in one unified preprocessing stage.
This makes experimentation much easier because changing the model no longer requires rewriting preprocessing code.
After creating the preprocessing pipeline, I combined it with different machine learning algorithms.
Instead of writing separate preprocessing code for every model, I simply replaced the final estimator.
Conceptually, every pipeline followed this structure:
Preprocessing
│
▼
Machine Learning Model
│
▼
Predictions
This allowed me to compare multiple algorithms while keeping preprocessing completely consistent.
Rather than relying on a single algorithm, I trained several ensemble models to compare their performance.
The models included:
Each model has its own strengths.
Random Forest was my baseline ensemble model.
Its advantages include:
It gave me a strong benchmark before experimenting with boosting algorithms.
XGBoost is one of the most popular gradient boosting libraries in competitive machine learning.
In my implementation, I configured parameters such as:
The goal was to balance predictive performance with computational efficiency.
CatBoost is particularly effective when working with datasets containing categorical variables.
Even though my preprocessing pipeline already handled encoding, CatBoost still provided another strong ensemble model for comparison.
I trained it using GPU acceleration to reduce computation time.
LightGBM is designed for efficiency.
It uses histogram-based learning and leaf-wise tree growth, making it extremely fast while maintaining competitive performance.
Its ability to scale efficiently makes it a popular choice in Kaggle competitions.
Training a model is only half of the workflow.
The next step is evaluating how well it performs.
I compared the models using several evaluation metrics instead of relying on accuracy alone.
These included:
Each metric provides different insights into model behavior.
Looking at multiple metrics helps avoid misleading conclusions that can arise from using only accuracy.
To gain a deeper understanding of performance, I generated classification reports for each model.
These reports show:
Instead of treating the model as a black box, these reports reveal where it performs well and where it struggles.
Another valuable visualization was the confusion matrix.
Rather than simply knowing that a prediction was incorrect, the confusion matrix shows which classes the model confuses with one another.
This often reveals patterns that raw accuracy cannot.
For multiclass classification problems, confusion matrices provide valuable insights into model weaknesses.
Instead of evaluating models separately, I created a comparison table containing metrics such as:
| Model | Accuracy | Precision | Recall | F1 Score |
|---|---|---|---|---|
| XGBoost | Compared | Compared | Compared | Compared |
| CatBoost | Compared | Compared | Compared | Compared |
| LightGBM | Compared | Compared | Compared | Compared |
A comparison table makes it much easier to identify trade-offs between different algorithms.
Looking back, this competition taught me far more than simply training machine learning models.
Some of the most important lessons were:
A strong preprocessing pipeline often improves performance more than switching algorithms.
Instead of rewriting preprocessing code repeatedly, everything becomes modular and reusable.
Accuracy alone rarely tells the complete story.
Comparing multiple algorithms helped me understand the strengths and limitations of each approach.
Like every real project, this competition involved several debugging sessions.
I encountered issues related to:
Although these challenges were sometimes frustrating, solving them significantly improved my understanding of the complete machine learning lifecycle.
In hindsight, those debugging sessions taught me as much as the successful model training itself.
One of the biggest misconceptions about Kaggle is that success is measured only by leaderboard position.
For me, the real achievement wasn't just generating predictions.
It was learning how to build an end-to-end machine learning workflow.
This project required me to combine concepts that I had previously learned separately:
For the first time, these concepts came together in a single project.
This competition marked an important milestone in my machine learning journey.
Before this project, preprocessing techniques, pipelines, ensemble models, and evaluation metrics were topics I had studied individually.
Working through a real Kaggle competition showed me how these pieces fit together to solve an end-to-end machine learning problem.
While there is still plenty to learn—from advanced feature engineering and hyperparameter optimization to model ensembling and deployment—this experience gave me confidence that I can move beyond tutorials and apply machine learning concepts in practical settings.
If you're just beginning your Kaggle journey, my biggest advice is simple:
Don't chase the leaderboard first. Chase understanding.
A solid foundation in data analysis, preprocessing, and experimentation will take you much further than memorizing model parameters. Every competition is an opportunity to learn, refine your workflow, and become a better machine learning practitioner.
For me, this wasn't just another notebook—it was the project that transformed weeks of studying into real, hands-on experience. And that's exactly what makes Kaggle such a valuable platform for anyone serious about learning machine learning.