{"slug": "house-prices-prediction-my-production-level-approach-to-advanced-regression", "title": "House Prices Prediction: My Production-Level Approach to Advanced Regression Techniques", "summary": "A developer built a production-level solution for the Kaggle House Prices: Advanced Regression Techniques competition, organizing the project into reusable modules with centralized configuration. The solution uses XGBoost with 79 features to predict house sale prices in Ames, Iowa, and is designed for real-world deployment rather than just leaderboard performance.", "body_md": "Imagine you're a real estate agent in Ames, Iowa. You have a list of 2,000+ houses, each with 79 features — from obvious ones like square footage and bedroom count to subtle details like basement height, garage quality, and proximity to amenities. Your task: predict the exact sale price of each house.\n\nThis is the essence of the Kaggle House Prices: Advanced Regression Techniques competition, one of the most popular machine learning challenges. It has helped thousands of data scientists develop practical regression skills.\n\nIn this blog post, I'll walk you through my production-level solution — not just a quick notebook to climb the leaderboard, but a robust, maintainable, and scalable approach suitable for real-world deployment.\n\nHouse price prediction extends far beyond academic exercises. Real-world applications include:\n\nThis dataset's richness — 79 features covering virtually every residential property aspect — provides an ideal environment for practicing advanced feature engineering, model selection, and hyperparameter tuning in realistic scenarios.\n\nThe dataset contains information about residential homes sold in Ames, Iowa, between 2006 and 2010.\n\nFeature Categories\n\n**Target Variable**\n\n**Dataset Statistics**\n\nMy solution stands out from typical Kaggle notebooks through its structure and rigor.\n\nInstead of a monolithic Jupyter notebook, the project is organized into reusable modules:\n\nHouse-Prices-Advanced-Regression-Techniques-Production-Level/\n\n├── data/\n\n│ ├── raw/\n\n│ ├── processed/\n\n│ └── external/\n\n├── notebooks/\n\n│ ├── 01_exploratory_data_analysis.ipynb\n\n│ ├── 02_feature_engineering.ipynb\n\n│ └── 03_model_experimentation.ipynb\n\n├── src/\n\n│ ├── __init__.py\n\n│ ├── data_loader.py\n\n│ ├── feature_engineer.py\n\n│ ├── model_trainer.py\n\n│ ├── evaluation.py\n\n│ └── config.py\n\n├── models/\n\n│ └── saved_models/\n\n├── requirements.txt\n\n├── README.md\n\n└── main.py\n\nAll paths, parameters, and settings are centralized:\n\n``` python\n# config.py\nfrom pathlib import Path\n\nclass Config:\n    # Paths\n    BASE_DIR = Path(__file__).parent.parent\n    DATA_DIR = BASE_DIR / 'data'\n    RAW_DATA = DATA_DIR / 'raw'\n    PROCESSED_DATA = DATA_DIR / 'processed'\n    MODELS_DIR = BASE_DIR / 'models'\n\n    # Data files\n    TRAIN_FILE = RAW_DATA / 'train.csv'\n    TEST_FILE = RAW_DATA / 'test.csv'\n\n    # Model settings\n    RANDOM_SEED = 42\n    TEST_SIZE = 0.2\n    TARGET = 'SalePrice'\n\n    # Feature engineering\n    NUMERICAL_FEATURES = []\n    CATEGORICAL_FEATURES = []\n    DATE_FEATURES = ['YearBuilt', 'YearRemodAdd', 'GarageYrBlt']\n\n    # Model parameters\n    XGBOOST_PARAMS = {\n        'objective': 'reg:squarederror',\n        'n_estimators': 1000,\n        'learning_rate': 0.05,\n        'max_depth': 6,\n        'subsample': 0.8,\n        'colsample_bytree': 0.8\n    }\npython\n# data_loader.py\nimport pandas as pd\nfrom pathlib import Path\nfrom typing import Tuple\nfrom .config import Config\n\nclass DataLoader:\n    \"\"\"Handles all data loading and initial preprocessing.\"\"\"\n\n    def __init__(self, config: Config):\n        self.config = config\n\n    def load_raw_data(self) -> Tuple[pd.DataFrame, pd.DataFrame]:\n        \"\"\"Load training and test datasets.\"\"\"\n        try:\n            train_df = pd.read_csv(self.config.TRAIN_FILE)\n            test_df = pd.read_csv(self.config.TEST_FILE)\n            print(f\"✅ Data loaded: {train_df.shape[0]} train, {test_df.shape[0]} test samples\")\n            return train_df, test_df\n        except FileNotFoundError as e:\n            print(f\"❌ Error loading data: {e}\")\n            raise\n\n    def save_processed_data(self, df: pd.DataFrame, filename: str) -> None:\n        \"\"\"Save processed dataframe to disk.\"\"\"\n        output_path = self.config.PROCESSED_DATA / filename\n        output_path.parent.mkdir(parents=True, exist_ok=True)\n        df.to_csv(output_path, index=False)\n        print(f\"✅ Saved processed data to {output_path}\")\n```\n\nUnderstanding your data is crucial before modeling. Here's what I discovered:\n\n``` python\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n\nplt.figure(figsize=(12, 6))\nsns.histplot(train_df['SalePrice'], bins=50, kde=True)\nplt.title('Distribution of Sale Prices', fontsize=16)\nplt.xlabel('Sale Price ($)', fontsize=12)\nplt.ylabel('Frequency', fontsize=12)\nplt.grid(True, alpha=0.3)\nplt.show()\n```\n\nKey Insight: Sale prices are right-skewed, with most houses priced between $100K-$200K and a few luxury properties pulling the average up. This suggests applying a log transformation to normalize the distribution.\n\n```\n# Select numerical features and compute correlation with SalePrice\nnumerical_cols = train_df.select_dtypes(include=['int64', 'float64']).columns\ncorrelation_matrix = train_df[numerical_cols].corr()\n\nplt.figure(figsize=(14, 10))\nsns.heatmap(correlation_matrix[['SalePrice']].sort_values(by='SalePrice', ascending=False),\n            annot=True, cmap='coolwarm', center=0, fmt='.2f')\nplt.title('Feature Correlation with SalePrice', fontsize=16)\nplt.show()\n# Calculate percentage of missing values for each feature\nmissing_percent = train_df.isnull().mean() * 100\nmissing_df = pd.DataFrame({\n    'Feature': train_df.columns,\n    'Missing_Percent': missing_percent\n}).sort_values('Missing_Percent', ascending=False)\n\n# Features with missing values > 10%\nprint(missing_df[missing_df['Missing_Percent'] > 10])\n# Calculate percentage of missing values for each feature\nmissing_percent = train_df.isnull().mean() * 100\nmissing_df = pd.DataFrame({\n    'Feature': train_df.columns,\n    'Missing_Percent': missing_percent\n}).sort_values('Missing_Percent', ascending=False)\n\n# Features with missing values > 10%\nprint(missing_df[missing_df['Missing_Percent'] > 10])\n```\n\nFeatures with Significant Missing Values:\n\nHandling Strategy: Drop features with >80% missing values. For others, use appropriate imputation based on feature type.\n\nRaw data rarely performs well in models. We must transform, create, and select features to extract maximum predictive power.\n\n``` python\n# feature_engineer.py\nimport numpy as np\nfrom sklearn.impute import SimpleImputer\n\nclass FeatureEngineer:\n    \"\"\"Handles all feature engineering tasks.\"\"\"\n\n    def __init__(self, config: Config):\n        self.config = config\n\n    def handle_missing_values(self, df: pd.DataFrame) -> pd.DataFrame:\n        \"\"\"Impute missing values appropriately.\"\"\"\n        # Numerical features: fill with median\n        num_imputer = SimpleImputer(strategy='median')\n        for col in self.config.NUMERICAL_FEATURES:\n            if col in df.columns:\n                df[col] = num_imputer.fit_transform(df[[col]])\n\n        # Categorical features: fill with mode\n        cat_imputer = SimpleImputer(strategy='most_frequent')\n        for col in self.config.CATEGORICAL_FEATURES:\n            if col in df.columns:\n                df[col] = cat_imputer.fit_transform(df[[col]])\n\n        return df\nphp\ndef create_new_features(self, df: pd.DataFrame) -> pd.DataFrame:\n    \"\"\"Create new features from existing ones.\"\"\"\n\n    # Age calculations\n    df['HouseAge'] = 2026 - df['YearBuilt']\n    df['RemodAge'] = 2026 - df['YearRemodAdd']\n\n    # Area calculations\n    df['TotalSF'] = df['TotalBsmtSF'] + df['1stFlrSF'] + df['2ndFlrSF']\n    df['TotalPorchSF'] = df['OpenPorchSF'] + df['EnclosedPorch'] + df['3SsnPorch'] + df['ScreenPorch']\n\n    # Bathroom count (weighted)\n    df['TotalBath'] = df['FullBath'] + 0.5 * df['HalfBath'] + df['BsmtFullBath'] + 0.5 * df['BsmtHalfBath']\n\n    # Binary features\n    df['HasPool'] = df['PoolArea'].apply(lambda x: 1 if x > 0 else 0)\n    df['HasFireplace'] = df['Fireplaces'].apply(lambda x: 1 if x > 0 else 0)\n    df['Has2ndFloor'] = df['2ndFlrSF'].apply(lambda x: 1 if x > 0 else 0)\n\n    # Quality grouping\n    df['QualityGroup'] = df['OverallQual'].apply(\n        lambda x: 'Poor' if x <= 4 else ('Fair' if x <= 6 else ('Good' if x <= 8 else 'Excellent'))\n    )\n\n    return df\nphp\ndef encode_categorical_features(self, df: pd.DataFrame) -> pd.DataFrame:\n    \"\"\"Encode categorical variables for modeling.\"\"\"\n\n    # Label encoding for ordinal features\n    ordinal_features = ['ExterQual', 'ExterCond', 'BsmtQual', 'BsmtCond', \n                       'HeatingQC', 'KitchenQual', 'FireplaceQu', 'GarageQual', \n                       'GarageCond', 'PoolQC']\n\n    quality_mapping = {'None': 0, 'Po': 1, 'Fa': 2, 'TA': 3, 'Gd': 4, 'Ex': 5}\n    for feature in ordinal_features:\n        if feature in df.columns:\n            df[feature] = df[feature].map(quality_mapping).fillna(0)\n\n    # One-hot encoding for nominal features\n    nominal_features = [col for col in self.config.CATEGORICAL_FEATURES \n                       if col not in ordinal_features and col != self.config.TARGET]\n\n    df = pd.get_dummies(df, columns=nominal_features, drop_first=True)\n\n    return df\npython\nfrom sklearn.preprocessing import RobustScaler\n\ndef scale_features(self, df: pd.DataFrame, scaler: RobustScaler = None, fit: bool = True) -> Tuple[pd.DataFrame, RobustScaler]:\n    \"\"\"Scale numerical features using RobustScaler (less sensitive to outliers).\"\"\"\n\n    if fit:\n        scaler = RobustScaler()\n        df[self.config.NUMERICAL_FEATURES] = scaler.fit_transform(df[self.config.NUMERICAL_FEATURES])\n    else:\n        df[self.config.NUMERICAL_FEATURES] = scaler.transform(df[self.config.NUMERICAL_FEATURES])\n\n    return df, scaler\n```\n\nWith properly engineered features, it's time to build models. I tested several approaches:\n\n``` python\nfrom sklearn.linear_model import LinearRegression\nfrom sklearn.metrics import mean_squared_error\nfrom sklearn.model_selection import train_test_split\nimport numpy as np\n\ndef train_baseline_model(X_train, y_train, X_val, y_val):\n    \"\"\"Train a simple linear regression model as baseline.\"\"\"\n\n    model = LinearRegression()\n    model.fit(X_train, y_train)\n\n    y_pred = model.predict(X_val)\n    rmse = np.sqrt(mean_squared_error(y_val, y_pred))\n    print(f\"Linear Regression RMSE: ${rmse:,.2f}\")\n\n    return model, rmse\n```\n\n**Result**: ~$45,210 RMSE on validation set\n\n``` python\nfrom xgboost import XGBRegressor\nfrom sklearn.model_selection import RandomizedSearchCV\n\ndef train_xgboost_model(X_train, y_train, X_val, y_val):\n    \"\"\"Train XGBoost model with hyperparameter tuning.\"\"\"\n\n    param_grid = {\n        'n_estimators': [500, 1000, 1500],\n        'learning_rate': [0.01, 0.05, 0.1],\n        'max_depth': [3, 4, 5, 6],\n        'subsample': [0.6, 0.8, 1.0],\n        'colsample_bytree': [0.6, 0.8, 1.0],\n        'gamma': [0, 0.1, 0.2],\n        'min_child_weight': [1, 3, 5]\n    }\n\n    base_model = XGBRegressor(\n        objective='reg:squarederror',\n        random_state=self.config.RANDOM_SEED,\n        n_jobs=-1\n    )\n\n    random_search = RandomizedSearchCV(\n        estimator=base_model,\n        param_distributions=param_grid,\n        n_iter=50,\n        scoring='neg_root_mean_squared_error',\n        cv=5,\n        verbose=2,\n        random_state=self.config.RANDOM_SEED,\n        n_jobs=-1\n    )\n\n    random_search.fit(X_train, y_train)\n    best_model = random_search.best_estimator_\n\n    y_pred = best_model.predict(X_val)\n    rmse = np.sqrt(mean_squared_error(y_val, y_pred))\n\n    print(f\"XGBoost RMSE: ${rmse:,.2f}\")\n    print(f\"Best parameters: {random_search.best_params_}\")\n\n    return best_model, rmse\npython\nfrom sklearn.ensemble import StackingRegressor\nfrom sklearn.linear_model import RidgeCV\n\ndef create_ensemble_model(X_train, y_train, X_val, y_val):\n    \"\"\"Create a stacking ensemble of multiple models.\"\"\"\n\n    estimators = [\n        ('xgb', XGBRegressor(\n            n_estimators=1000, learning_rate=0.05, max_depth=6,\n            subsample=0.8, colsample_bytree=0.8, random_state=self.config.RANDOM_SEED\n        )),\n        ('rf', RandomForestRegressor(\n            n_estimators=500, max_depth=10,\n            random_state=self.config.RANDOM_SEED, n_jobs=-1\n        )),\n        ('ridge', RidgeCV(alphas=[0.1, 1.0, 10.0]))\n    ]\n\n    stacking_model = StackingRegressor(\n        estimators=estimators,\n        final_estimator=RidgeCV(alphas=[0.1, 1.0, 10.0]),\n        cv=5,\n        n_jobs=-1\n    )\n\n    stacking_model.fit(X_train, y_train)\n    y_pred = stacking_model.predict(X_val)\n    rmse = np.sqrt(mean_squared_error(y_val, y_pred))\n\n    print(f\"Stacking Ensemble RMSE: ${rmse:,.2f}\")\n\n    return stacking_model, rmse\n```\n\nValidation set performance after extensive experimentation:\n\n| Model | RMSE | R² Score | Training Time |\n|---|---|---|---|\n| Linear Regression | $45,210 | 0.82 | 0.5s |\n| Ridge Regression | $44,890 | 0.82 | 0.8s |\n| Random Forest | $28,950 | 0.92 | 15s |\n| XGBoost (default) | $26,120 | 0.93 | 20s |\n| XGBoost (tuned) | $23,450 | 0.94 | 45s |\n| LightGBM | $24,180 | 0.94 | 30s |\n| Stacking Ensemble | $22,890 | 0.95 | 60s |\n\n**Kaggle Leaderboard Performance:** Top 12% (out of ~10,000 participants)\n\nUnderstanding which features drive predictions is crucial for model interpretability and feature selection.\n\n``` python\nimport xgboost as xgb\n\nfig, ax = plt.subplots(figsize=(12, 8))\nxgb.plot_importance(best_xgb_model, max_num_features=20, height=0.8, ax=ax)\nplt.title('XGBoost Feature Importance', fontsize=16)\nplt.show()\n```\n\n**Key Insight**: The most important features are size-related and quality-related, which aligns with intuition — bigger, higher-quality houses tend to sell for more.\n\nA true production-level solution requires deployment consideration.\n\n``` php\nimport joblib\n\ndef save_model(self, model, model_name: str) -> None:\n    \"\"\"Save trained model to disk.\"\"\"\n    model_path = self.config.MODELS_DIR / f\"{model_name}.pkl\"\n    model_path.parent.mkdir(parents=True, exist_ok=True)\n    joblib.dump(model, model_path)\n    print(f\"✅ Model saved to {model_path}\")\n\ndef load_model(self, model_name: str):\n    \"\"\"Load saved model from disk.\"\"\"\n    model_path = self.config.MODELS_DIR / f\"{model_name}.pkl\"\n    model = joblib.load(model_path)\n    print(f\"✅ Model loaded from {model_path}\")\n    return model\nclass HousePricePredictor:\n    \"\"\"End-to-end prediction pipeline.\"\"\"\n\n    def __init__(self, config: Config):\n        self.config = config\n        self.data_loader = DataLoader(config)\n        self.feature_engineer = FeatureEngineer(config)\n        self.model = None\n        self.scaler = None\n\n    def train(self):\n        \"\"\"Train the complete pipeline.\"\"\"\n        train_df, _ = self.data_loader.load_raw_data()\n\n        # Feature engineering\n        train_df = self.feature_engineer.handle_missing_values(train_df)\n        train_df = self.feature_engineer.create_new_features(train_df)\n        train_df = self.feature_engineer.encode_categorical_features(train_df)\n\n        X = train_df.drop(columns=[self.config.TARGET])\n        y = train_df[self.config.TARGET]\n\n        X_train, X_val, y_train, y_val = train_test_split(\n            X, y, test_size=self.config.TEST_SIZE, random_state=self.config.RANDOM_SEED\n        )\n\n        X_train, self.scaler = self.feature_engineer.scale_features(X_train, fit=True)\n        X_val, _ = self.feature_engineer.scale_features(X_val, scaler=self.scaler, fit=False)\n\n        self.model, rmse = self.train_xgboost_model(X_train, y_train, X_val, y_val)\n\n        self.save_model(self.model, 'xgboost_house_price_predictor')\n        joblib.dump(self.scaler, self.config.MODELS_DIR / 'scaler.pkl')\n\n        return rmse\n\n    def predict(self, new_data: pd.DataFrame) -> np.ndarray:\n        \"\"\"Make predictions on new data.\"\"\"\n        if self.model is None:\n            self.model = self.load_model('xgboost_house_price_predictor')\n            self.scaler = joblib.load(self.config.MODELS_DIR / 'scaler.pkl')\n\n        new_data = self.feature_engineer.handle_missing_values(new_data)\n        new_data = self.feature_engineer.create_new_features(new_data)\n        new_data = self.feature_engineer.encode_categorical_features(new_data)\n        new_data, _ = self.feature_engineer.scale_features(new_data, scaler=self.scaler, fit=False)\n\n        return self.model.predict(new_data)\npython\n# main.py\nimport argparse\n\ndef main():\n    parser = argparse.ArgumentParser(description='House Price Prediction Pipeline')\n    parser.add_argument('--train', action='store_true', help='Train the model')\n    parser.add_argument('--predict', action='store_true', help='Make predictions')\n    parser.add_argument('--input', type=str, help='Path to input CSV file')\n    parser.add_argument('--output', type=str, default='predictions.csv', help='Path to save predictions')\n\n    args = parser.parse_args()\n    config = Config()\n    predictor = HousePricePredictor(config)\n\n    if args.train:\n        rmse = predictor.train()\n        print(f\"Model trained with RMSE: ${rmse:,.2f}\")\n\n    if args.predict:\n        if not args.input:\n            print(\"Error: --input argument required\")\n            return\n\n        test_df = pd.read_csv(args.input)\n        predictions = predictor.predict(test_df)\n        pd.DataFrame({'Id': test_df['Id'], 'SalePrice': predictions}).to_csv(args.output, index=False)\n        print(f\"Predictions saved to {args.output}\")\n\nif __name__ == '__main__':\n    main()\n```\n\n`# requirements.txt`\n\npython>=3.8\n\npandas>=1.3.0\n\nnumpy>=1.21.0\n\nscikit-learn>=0.24.2\n\nxgboost>=1.5.0\n\nlightgbm>=3.2.1\n\ncatboost>=0.24.1\n\nmatplotlib>=3.4.0\n\nseaborn>=0.11.0\n\njoblib>=1.0.0\n\njupyter>=1.0.0\n\nInstall with:\n\n```\npip install -r requirements.txt\n```\n\nThe biggest performance improvements came from better features, not fancier models. The difference between good and great models often lies in feature quality.\n\nWhile XGBoost performed exceptionally well, the stacking ensemble achieved the best results. Model diversity leads to better generalization.\n\nNot all missing values are equal. Some indicate feature absence (no pool, no fireplace), while others are genuinely missing. Context matters.\n\nThe right-skewed sale price distribution benefited significantly from log transformation.\n\nAlways use cross-validation. A single train-test split can yield misleading results, especially with smaller datasets.\n\nSetting random seeds everywhere (random_state=42) ensures reproducible results, crucial for debugging and production.\n\nThis project provided invaluable experience. Building a production-ready machine learning pipeline from raw data taught me about:\n\nThe model achieved a Top 12% ranking on Kaggle, but more importantly, it's structured for real-world deployment.\n\nFor machine learning beginners, I highly recommend this competition. It's challenging enough to teach real skills yet approachable with fundamental techniques.\n\nI hope you found this helpful! Questions or suggestions? Let's connect:\n\nGitHub: github.com/arghasarkar\n\nLinkedIn: linkedin.com/in/argha-sarkar\n\nKaggle: kaggle.com/arghasarkar", "url": "https://wpnews.pro/news/house-prices-prediction-my-production-level-approach-to-advanced-regression", "canonical_source": "https://dev.to/argha_sarkar/house-prices-prediction-my-production-level-approach-to-advanced-regression-techniques-34l7", "published_at": "2026-07-20 14:38:09+00:00", "updated_at": "2026-07-20 14:50:15.523976+00:00", "lang": "en", "topics": ["machine-learning", "developer-tools"], "entities": ["Kaggle", "Ames", "Iowa"], "alternates": {"html": "https://wpnews.pro/news/house-prices-prediction-my-production-level-approach-to-advanced-regression", "markdown": "https://wpnews.pro/news/house-prices-prediction-my-production-level-approach-to-advanced-regression.md", "text": "https://wpnews.pro/news/house-prices-prediction-my-production-level-approach-to-advanced-regression.txt", "jsonld": "https://wpnews.pro/news/house-prices-prediction-my-production-level-approach-to-advanced-regression.jsonld"}}