House Prices Prediction: My Production-Level Approach to Advanced Regression Techniques

A developer built a production-level solution for the Kaggle House Prices: Advanced Regression Techniques competition, organizing the project into reusable modules with centralized configuration. The solution uses XGBoost with 79 features to predict house sale prices in Ames, Iowa, and is designed for real-world deployment rather than just leaderboard performance.

Imagine you're a real estate agent in Ames, Iowa. You have a list of 2,000+ houses, each with 79 features — from obvious ones like square footage and bedroom count to subtle details like basement height, garage quality, and proximity to amenities. Your task: predict the exact sale price of each house. This is the essence of the Kaggle House Prices: Advanced Regression Techniques competition, one of the most popular machine learning challenges. It has helped thousands of data scientists develop practical regression skills. In this blog post, I'll walk you through my production-level solution — not just a quick notebook to climb the leaderboard, but a robust, maintainable, and scalable approach suitable for real-world deployment. House price prediction extends far beyond academic exercises. Real-world applications include: This dataset's richness — 79 features covering virtually every residential property aspect — provides an ideal environment for practicing advanced feature engineering, model selection, and hyperparameter tuning in realistic scenarios. The dataset contains information about residential homes sold in Ames, Iowa, between 2006 and 2010. Feature Categories Target Variable Dataset Statistics My solution stands out from typical Kaggle notebooks through its structure and rigor. Instead of a monolithic Jupyter notebook, the project is organized into reusable modules: House-Prices-Advanced-Regression-Techniques-Production-Level/ ├── data/ │ ├── raw/ │ ├── processed/ │ └── external/ ├── notebooks/ │ ├── 01 exploratory data analysis.ipynb │ ├── 02 feature engineering.ipynb │ └── 03 model experimentation.ipynb ├── src/ │ ├── init .py │ ├── data loader.py │ ├── feature engineer.py │ ├── model trainer.py │ ├── evaluation.py │ └── config.py ├── models/ │ └── saved models/ ├── requirements.txt ├── README.md └── main.py All paths, parameters, and settings are centralized: python config.py from pathlib import Path class Config: Paths BASE DIR = Path file .parent.parent DATA DIR = BASE DIR / 'data' RAW DATA = DATA DIR / 'raw' PROCESSED DATA = DATA DIR / 'processed' MODELS DIR = BASE DIR / 'models' Data files TRAIN FILE = RAW DATA / 'train.csv' TEST FILE = RAW DATA / 'test.csv' Model settings RANDOM SEED = 42 TEST SIZE = 0.2 TARGET = 'SalePrice' Feature engineering NUMERICAL FEATURES = CATEGORICAL FEATURES = DATE FEATURES = 'YearBuilt', 'YearRemodAdd', 'GarageYrBlt' Model parameters XGBOOST PARAMS = { 'objective': 'reg:squarederror', 'n estimators': 1000, 'learning rate': 0.05, 'max depth': 6, 'subsample': 0.8, 'colsample bytree': 0.8 } python data loader.py import pandas as pd from pathlib import Path from typing import Tuple from .config import Config class DataLoader: """Handles all data loading and initial preprocessing.""" def init self, config: Config : self.config = config def load raw data self - Tuple pd.DataFrame, pd.DataFrame : """Load training and test datasets.""" try: train df = pd.read csv self.config.TRAIN FILE test df = pd.read csv self.config.TEST FILE print f"✅ Data loaded: {train df.shape 0 } train, {test df.shape 0 } test samples" return train df, test df except FileNotFoundError as e: print f"❌ Error loading data: {e}" raise def save processed data self, df: pd.DataFrame, filename: str - None: """Save processed dataframe to disk.""" output path = self.config.PROCESSED DATA / filename output path.parent.mkdir parents=True, exist ok=True df.to csv output path, index=False print f"✅ Saved processed data to {output path}" Understanding your data is crucial before modeling. Here's what I discovered: python import matplotlib.pyplot as plt import seaborn as sns plt.figure figsize= 12, 6 sns.histplot train df 'SalePrice' , bins=50, kde=True plt.title 'Distribution of Sale Prices', fontsize=16 plt.xlabel 'Sale Price $ ', fontsize=12 plt.ylabel 'Frequency', fontsize=12 plt.grid True, alpha=0.3 plt.show Key Insight: Sale prices are right-skewed, with most houses priced between $100K-$200K and a few luxury properties pulling the average up. This suggests applying a log transformation to normalize the distribution. Select numerical features and compute correlation with SalePrice numerical cols = train df.select dtypes include= 'int64', 'float64' .columns correlation matrix = train df numerical cols .corr plt.figure figsize= 14, 10 sns.heatmap correlation matrix 'SalePrice' .sort values by='SalePrice', ascending=False , annot=True, cmap='coolwarm', center=0, fmt='.2f' plt.title 'Feature Correlation with SalePrice', fontsize=16 plt.show Calculate percentage of missing values for each feature missing percent = train df.isnull .mean 100 missing df = pd.DataFrame { 'Feature': train df.columns, 'Missing Percent': missing percent } .sort values 'Missing Percent', ascending=False Features with missing values 10% print missing df missing df 'Missing Percent' 10 Calculate percentage of missing values for each feature missing percent = train df.isnull .mean 100 missing df = pd.DataFrame { 'Feature': train df.columns, 'Missing Percent': missing percent } .sort values 'Missing Percent', ascending=False Features with missing values 10% print missing df missing df 'Missing Percent' 10 Features with Significant Missing Values: Handling Strategy: Drop features with 80% missing values. For others, use appropriate imputation based on feature type. Raw data rarely performs well in models. We must transform, create, and select features to extract maximum predictive power. python feature engineer.py import numpy as np from sklearn.impute import SimpleImputer class FeatureEngineer: """Handles all feature engineering tasks.""" def init self, config: Config : self.config = config def handle missing values self, df: pd.DataFrame - pd.DataFrame: """Impute missing values appropriately.""" Numerical features: fill with median num imputer = SimpleImputer strategy='median' for col in self.config.NUMERICAL FEATURES: if col in df.columns: df col = num imputer.fit transform df col Categorical features: fill with mode cat imputer = SimpleImputer strategy='most frequent' for col in self.config.CATEGORICAL FEATURES: if col in df.columns: df col = cat imputer.fit transform df col return df php def create new features self, df: pd.DataFrame - pd.DataFrame: """Create new features from existing ones.""" Age calculations df 'HouseAge' = 2026 - df 'YearBuilt' df 'RemodAge' = 2026 - df 'YearRemodAdd' Area calculations df 'TotalSF' = df 'TotalBsmtSF' + df '1stFlrSF' + df '2ndFlrSF' df 'TotalPorchSF' = df 'OpenPorchSF' + df 'EnclosedPorch' + df '3SsnPorch' + df 'ScreenPorch' Bathroom count weighted df 'TotalBath' = df 'FullBath' + 0.5 df 'HalfBath' + df 'BsmtFullBath' + 0.5 df 'BsmtHalfBath' Binary features df 'HasPool' = df 'PoolArea' .apply lambda x: 1 if x 0 else 0 df 'HasFireplace' = df 'Fireplaces' .apply lambda x: 1 if x 0 else 0 df 'Has2ndFloor' = df '2ndFlrSF' .apply lambda x: 1 if x 0 else 0 Quality grouping df 'QualityGroup' = df 'OverallQual' .apply lambda x: 'Poor' if x <= 4 else 'Fair' if x <= 6 else 'Good' if x <= 8 else 'Excellent' return df php def encode categorical features self, df: pd.DataFrame - pd.DataFrame: """Encode categorical variables for modeling.""" Label encoding for ordinal features ordinal features = 'ExterQual', 'ExterCond', 'BsmtQual', 'BsmtCond', 'HeatingQC', 'KitchenQual', 'FireplaceQu', 'GarageQual', 'GarageCond', 'PoolQC' quality mapping = {'None': 0, 'Po': 1, 'Fa': 2, 'TA': 3, 'Gd': 4, 'Ex': 5} for feature in ordinal features: if feature in df.columns: df feature = df feature .map quality mapping .fillna 0 One-hot encoding for nominal features nominal features = col for col in self.config.CATEGORICAL FEATURES if col not in ordinal features and col = self.config.TARGET df = pd.get dummies df, columns=nominal features, drop first=True return df python from sklearn.preprocessing import RobustScaler def scale features self, df: pd.DataFrame, scaler: RobustScaler = None, fit: bool = True - Tuple pd.DataFrame, RobustScaler : """Scale numerical features using RobustScaler less sensitive to outliers .""" if fit: scaler = RobustScaler df self.config.NUMERICAL FEATURES = scaler.fit transform df self.config.NUMERICAL FEATURES else: df self.config.NUMERICAL FEATURES = scaler.transform df self.config.NUMERICAL FEATURES return df, scaler With properly engineered features, it's time to build models. I tested several approaches: python from sklearn.linear model import LinearRegression from sklearn.metrics import mean squared error from sklearn.model selection import train test split import numpy as np def train baseline model X train, y train, X val, y val : """Train a simple linear regression model as baseline.""" model = LinearRegression model.fit X train, y train y pred = model.predict X val rmse = np.sqrt mean squared error y val, y pred print f"Linear Regression RMSE: ${rmse:,.2f}" return model, rmse Result : ~$45,210 RMSE on validation set python from xgboost import XGBRegressor from sklearn.model selection import RandomizedSearchCV def train xgboost model X train, y train, X val, y val : """Train XGBoost model with hyperparameter tuning.""" param grid = { 'n estimators': 500, 1000, 1500 , 'learning rate': 0.01, 0.05, 0.1 , 'max depth': 3, 4, 5, 6 , 'subsample': 0.6, 0.8, 1.0 , 'colsample bytree': 0.6, 0.8, 1.0 , 'gamma': 0, 0.1, 0.2 , 'min child weight': 1, 3, 5 } base model = XGBRegressor objective='reg:squarederror', random state=self.config.RANDOM SEED, n jobs=-1 random search = RandomizedSearchCV estimator=base model, param distributions=param grid, n iter=50, scoring='neg root mean squared error', cv=5, verbose=2, random state=self.config.RANDOM SEED, n jobs=-1 random search.fit X train, y train best model = random search.best estimator y pred = best model.predict X val rmse = np.sqrt mean squared error y val, y pred print f"XGBoost RMSE: ${rmse:,.2f}" print f"Best parameters: {random search.best params }" return best model, rmse python from sklearn.ensemble import StackingRegressor from sklearn.linear model import RidgeCV def create ensemble model X train, y train, X val, y val : """Create a stacking ensemble of multiple models.""" estimators = 'xgb', XGBRegressor n estimators=1000, learning rate=0.05, max depth=6, subsample=0.8, colsample bytree=0.8, random state=self.config.RANDOM SEED , 'rf', RandomForestRegressor n estimators=500, max depth=10, random state=self.config.RANDOM SEED, n jobs=-1 , 'ridge', RidgeCV alphas= 0.1, 1.0, 10.0 stacking model = StackingRegressor estimators=estimators, final estimator=RidgeCV alphas= 0.1, 1.0, 10.0 , cv=5, n jobs=-1 stacking model.fit X train, y train y pred = stacking model.predict X val rmse = np.sqrt mean squared error y val, y pred print f"Stacking Ensemble RMSE: ${rmse:,.2f}" return stacking model, rmse Validation set performance after extensive experimentation: | Model | RMSE | R² Score | Training Time | |---|---|---|---| | Linear Regression | $45,210 | 0.82 | 0.5s | | Ridge Regression | $44,890 | 0.82 | 0.8s | | Random Forest | $28,950 | 0.92 | 15s | | XGBoost default | $26,120 | 0.93 | 20s | | XGBoost tuned | $23,450 | 0.94 | 45s | | LightGBM | $24,180 | 0.94 | 30s | | Stacking Ensemble | $22,890 | 0.95 | 60s | Kaggle Leaderboard Performance: Top 12% out of ~10,000 participants Understanding which features drive predictions is crucial for model interpretability and feature selection. python import xgboost as xgb fig, ax = plt.subplots figsize= 12, 8 xgb.plot importance best xgb model, max num features=20, height=0.8, ax=ax plt.title 'XGBoost Feature Importance', fontsize=16 plt.show Key Insight : The most important features are size-related and quality-related, which aligns with intuition — bigger, higher-quality houses tend to sell for more. A true production-level solution requires deployment consideration. php import joblib def save model self, model, model name: str - None: """Save trained model to disk.""" model path = self.config.MODELS DIR / f"{model name}.pkl" model path.parent.mkdir parents=True, exist ok=True joblib.dump model, model path print f"✅ Model saved to {model path}" def load model self, model name: str : """Load saved model from disk.""" model path = self.config.MODELS DIR / f"{model name}.pkl" model = joblib.load model path print f"✅ Model loaded from {model path}" return model class HousePricePredictor: """End-to-end prediction pipeline.""" def init self, config: Config : self.config = config self.data loader = DataLoader config self.feature engineer = FeatureEngineer config self.model = None self.scaler = None def train self : """Train the complete pipeline.""" train df, = self.data loader.load raw data Feature engineering train df = self.feature engineer.handle missing values train df train df = self.feature engineer.create new features train df train df = self.feature engineer.encode categorical features train df X = train df.drop columns= self.config.TARGET y = train df self.config.TARGET X train, X val, y train, y val = train test split X, y, test size=self.config.TEST SIZE, random state=self.config.RANDOM SEED X train, self.scaler = self.feature engineer.scale features X train, fit=True X val, = self.feature engineer.scale features X val, scaler=self.scaler, fit=False self.model, rmse = self.train xgboost model X train, y train, X val, y val self.save model self.model, 'xgboost house price predictor' joblib.dump self.scaler, self.config.MODELS DIR / 'scaler.pkl' return rmse def predict self, new data: pd.DataFrame - np.ndarray: """Make predictions on new data.""" if self.model is None: self.model = self.load model 'xgboost house price predictor' self.scaler = joblib.load self.config.MODELS DIR / 'scaler.pkl' new data = self.feature engineer.handle missing values new data new data = self.feature engineer.create new features new data new data = self.feature engineer.encode categorical features new data new data, = self.feature engineer.scale features new data, scaler=self.scaler, fit=False return self.model.predict new data python main.py import argparse def main : parser = argparse.ArgumentParser description='House Price Prediction Pipeline' parser.add argument '--train', action='store true', help='Train the model' parser.add argument '--predict', action='store true', help='Make predictions' parser.add argument '--input', type=str, help='Path to input CSV file' parser.add argument '--output', type=str, default='predictions.csv', help='Path to save predictions' args = parser.parse args config = Config predictor = HousePricePredictor config if args.train: rmse = predictor.train print f"Model trained with RMSE: ${rmse:,.2f}" if args.predict: if not args.input: print "Error: --input argument required" return test df = pd.read csv args.input predictions = predictor.predict test df pd.DataFrame {'Id': test df 'Id' , 'SalePrice': predictions} .to csv args.output, index=False print f"Predictions saved to {args.output}" if name == ' main ': main requirements.txt python =3.8 pandas =1.3.0 numpy =1.21.0 scikit-learn =0.24.2 xgboost =1.5.0 lightgbm =3.2.1 catboost =0.24.1 matplotlib =3.4.0 seaborn =0.11.0 joblib =1.0.0 jupyter =1.0.0 Install with: pip install -r requirements.txt The biggest performance improvements came from better features, not fancier models. The difference between good and great models often lies in feature quality. While XGBoost performed exceptionally well, the stacking ensemble achieved the best results. Model diversity leads to better generalization. Not all missing values are equal. Some indicate feature absence no pool, no fireplace , while others are genuinely missing. Context matters. The right-skewed sale price distribution benefited significantly from log transformation. Always use cross-validation. A single train-test split can yield misleading results, especially with smaller datasets. Setting random seeds everywhere random state=42 ensures reproducible results, crucial for debugging and production. This project provided invaluable experience. Building a production-ready machine learning pipeline from raw data taught me about: The model achieved a Top 12% ranking on Kaggle, but more importantly, it's structured for real-world deployment. For machine learning beginners, I highly recommend this competition. It's challenging enough to teach real skills yet approachable with fundamental techniques. I hope you found this helpful Questions or suggestions? Let's connect: GitHub: github.com/arghasarkar LinkedIn: linkedin.com/in/argha-sarkar Kaggle: kaggle.com/arghasarkar