Hey everyone! As I’m working through my ML fundamentals, I wanted to share a deep-dive Exploratory Data Analysis (EDA) I just completed on the classic Ames, Iowa Housing Dataset.
If you are unfamiliar, the Ames dataset is basically the "final boss" version of the Boston Housing dataset. It has 2,930 residential properties and 80+ explanatory variables.
Instead of just running a standard df.describe()
, I wanted to focus on Feature Engineering and understanding the actual business logic behind the data. Here are my biggest takeaways and a few counter-intuitive findings!
When I first looked at the data, the square footage was split across multiple columns (basement, 1st floor, 2nd floor). Instead of feeding the model raw variables, I created a few composite features:
Total_Usable_SF
Price_Per_SqFt
Total_Usable_SF
, I created a normalized metric. This made comparing different neighborhoods Sale_Price
was heavily right-skewed (a few multi-million dollar mansions were dragging the tail). Applying a logarithmic transformation (np.log1p
) instantly normalized the distribution, which is crucial for linear modeling assumptions.
import numpy as np
import seaborn as sns
df['Log_Sale_Price'] = np.log1p(df['Sale_Price'])
sns.histplot(df['Log_Sale_Price'], kde=True)
Usually, missing data is annoying. But in housing, "NaN" rarely means the data is missing—it usually means the house structurally lacks that feature.
I converted missing values for things like Pools, Fences, and Garages into binary presence/absence flags.
This was my favorite part of the analysis. Sometimes, features that sound like upgrades actually drag the price down.
For example, having a Fence or Alley access actually correlated with a lower median sale price. Why? It turns out this is a proxy for age and location. Fences and alleys are incredibly common in the older, denser urban tracts of Ames. The expensive, newly built golf-course developments don't have alleys at all!
I ran into one dilemma: Pools.
Pools definitely exhibited a price premium, but they were present in only 13 out of 2,930 homes (0.44%). They are an ultra-luxury edge case.
How do you typically handle features that exist in less than 1% of the dataset?
Do you keep them as binary flags because of their predictive power on high-end outliers, or do you drop them completely due to sparsity before feeding them into a model like XGBoost?
Let me know your thoughts in the comments!
If you want to see the full code, you can check out my project repo on GitHub!
https://github.com/deepakdoriya/ames-housing-eda.git