{"slug": "building-a-production-grade-end-to-end-mlops-pipeline-from-scratch", "title": "Building a Production-Grade End-to-End MLOps Pipeline from Scratch", "summary": "A developer published a walkthrough for building a production-grade end-to-end MLOps pipeline around the Titanic survival prediction dataset, covering data versioning with DVC, validation, scikit-learn training, MLflow experiment tracking, Pytest testing, Docker containerization, GitHub Actions CI/CD, FastAPI serving, Prometheus monitoring, and Evidently AI drift detection. The project, released on GitHub as End-to-end-MLOPS-Pipeline, emphasizes that model training is a small fraction of production ML work.", "body_md": "Most machine learning tutorials end at `model.fit()`. But in the real world, training a model is barely 10% of the work. The remaining 90% is everything that makes ML actually work in production — data versioning, validation, experiment tracking, API serving, automated testing, containerization, CI/CD, monitoring, and drift detection.\n\nIn this walkthrough, I'll take you through building a **complete, production-grade MLOps pipeline** using the classic Titanic survival prediction problem. The focus isn't on building a complex model — it's on implementing every production practice that separates a Jupyter notebook experiment from a deployable ML system.\n\n**GitHub Repository:** [End-to-end-MLOPS-Pipeline](https://github.com/naman-0804/End-to-end-MLOPS-Pipeline)\n\nHere's the complete system we're building:\n\n```\nData (Titanic CSV)\n │\n ▼\nDVC (Data Versioning)\n │\n ▼\nData Validation (Schema & Constraint Checks)\n │\n ▼\nScikit-learn Training Pipeline\n │\n ▼\nMLflow (Experiment Tracking & Model Registry)\n │\n ▼\nPytest (Automated Testing)\n │\n ▼\nDocker (Containerization)\n │\n ▼\nGitHub Actions (CI/CD)\n │\n ▼\nFastAPI (Inference Service)\n │\n ├── Prometheus (Monitoring)\n │\n └── Evidently AI (Drift Detection)\n```\n\n| Component | Tool | \n|---|---|\n| Version Control | Git, GitHub | \n| Data Versioning | DVC | \n| Model Training | Scikit-learn | \n| Experiment Tracking | MLflow | \n| API Framework | FastAPI | \n| Testing | Pytest | \n| Containerization | Docker | \n| CI/CD | GitHub Actions | \n| Monitoring | Prometheus | \n| Drift Detection | Evidently AI | \n\n```\nEnd-to-end-MLOPS-Pipeline/\n│\n├── data/\n│   └── raw/\n│       └── titanic.csv\n│\n├── models/\n│   └── model.joblib\n│\n├── src/\n│   ├── data_ingestion.py\n│   ├── data_validation.py\n│   ├── data_preprocess.py\n│   ├── train.py\n│   └── drift_detection.py\n│\n├── api/\n│   ├── __init__.py\n│   └── main.py\n│\n├── tests/\n│   ├── __init__.py\n│   ├── test_api.py\n│   └── test_model.py\n│\n├── monitoring/\n│   ├── prometheus.yml\n│   └── drift_report.html\n│\n├── .github/\n│   └── workflows/\n│       └── ci_cd.yml\n│\n├── Dockerfile\n├── docker-compose.yml\n├── requirements.txt\n├── .gitignore\n├── .dockerignore\n└── .dvc/\n```\n\nStart by initializing Git and creating a virtual environment:\n\n```\ngit init\npython -m venv venv\n\n# Activate the virtual environment\n# Windows PowerShell:\n.\\venv\\Scripts\\Activate.ps1\n# Linux/macOS:\nsource venv/bin/activate\n```\n\n`.gitignore`\nCreate a `.gitignore` that keeps our repository clean:\n\n```\n# Virtual environments\nvenv/\nenv/\n.env\n\n# Python cache\n__pycache__/\n*.pyc\n\n# Data (tracked by DVC, not Git)\ndata/raw/*\n!data/raw/*.dvc\n!data/raw/*.csv\ndata/processed/\n\n# Models and MLflow\nmodels/\nmlruns/\n```\n\n`requirements.txt`\n\n```\n# Data & ML\npandas>=2.0.0\nnumpy>=1.24.0\nscikit-learn>=1.3.0\njoblib>=1.3.0\n\n# Data Validation & Tracking\ndvc>=3.0.0\nmlflow>=2.10.0\n\n# API\nfastapi>=0.100.0\nuvicorn[standard]>=0.22.0\npydantic>=2.0.0\n\n# Testing\npytest>=7.4.0\nhttpx>=0.24.0\nprometheus-client>=0.17.0\nevidently\n```\n\nInstall everything:\n\n```\npip install -r requirements.txt\n```\n\nGit was designed for source code — small text files. Datasets can be hundreds of megabytes or gigabytes. Committing them directly to Git makes your repository bloated, slow to clone, and impossible to manage at scale.\n\n**DVC (Data Version Control)** solves this by:\n\n`.dvc` metadata files in Git (a few bytes).`git checkout`.\nDownload the [Titanic dataset](https://raw.githubusercontent.com/datasciencedojo/datasets/master/titanic.csv) and place it in `data/raw/titanic.csv`.\n\n```\n# Initialize DVC\ndvc init\n\n# Track the dataset\ndvc add data/raw/titanic.csv\n\n# Commit the metadata\ngit add .\ngit commit -m \"Initialize project and track dataset with DVC\"\n```\n\nDVC creates a `data/raw/titanic.csv.dvc` file — a tiny metadata pointer that Git tracks instead of the actual CSV.\n\n`src/data_ingestion.py`)\nThe data ingestion module is responsible for loading raw data from disk with defensive checks — ensuring the file exists and isn't empty before passing it downstream.\n\n``` python\nimport os\nimport logging\nimport pandas as pd\n\nlogging.basicConfig(\n    level=logging.INFO,\n    format=\"%(asctime)s - %(levelname)s - %(message)s\"\n)\n\ndef load_data(file_path: str) -> pd.DataFrame:\n    \"\"\"\n    Load raw CSV data and validate it.\n    \"\"\"\n    if not os.path.exists(file_path):\n        logging.error(f\"File not found at path: {file_path}\")\n        raise FileNotFoundError(f\"File does not exist: {file_path}\")\n\n    logging.info(f\"Loading data from {file_path}\")\n    df = pd.read_csv(file_path)\n\n    if df.empty:\n        logging.error(\"Loaded dataframe is empty\")\n        raise ValueError(\"Dataframe is empty\")\n\n    logging.info(f\"Successfully loaded dataset with shape: {df.shape}\")\n    return df\n\nif __name__ == \"__main__\":\n    raw_data_path = os.path.join(\"data\", \"raw\", \"titanic.csv\")\n    df = load_data(raw_data_path)\n    print(df.head())\n```\n\n`logging` instead of `print()`?\nIn production MLOps, `print()` statements disappear into the void. `logging` gives you:\n\n`INFO`, `WARNING`, `ERROR`) to filter noise.` src/data_validation.py`)\nBefore touching any ML model, we validate that the incoming data matches our expectations. This is the **first line of defense** against silent model failures caused by upstream data changes.\n\n``` python\nimport os\nimport sys\nimport logging\nimport pandas as pd\n\nsys.path.append(os.path.abspath(os.path.join(os.path.dirname(__file__), \"..\")))\nfrom src.data_ingestion import load_data\n\nlogging.basicConfig(level=logging.INFO, format=\"%(asctime)s - %(levelname)s - %(message)s\")\n\n# Define expected schema and column rules\nREQUIRED_COLUMNS = [\n    \"Survived\", \"Pclass\", \"Name\", \"Sex\", \"Age\", \n    \"SibSp\", \"Parch\", \"Ticket\", \"Fare\", \"Embarked\"\n]\nVALID_PCLASS = {1, 2, 3}\nVALID_SEX = {\"male\", \"female\"}\n\ndef validate_data(df: pd.DataFrame) -> bool:\n    \"\"\"\n    Validates dataset integrity, schema, and column value constraints.\n    \"\"\"\n    logging.info(\"Starting data validation checks...\")\n\n    # 1. Check for required columns\n    missing_cols = [col for col in REQUIRED_COLUMNS if col not in df.columns]\n    if missing_cols:\n        logging.error(f\"Validation failed: Missing columns: {missing_cols}\")\n        raise ValueError(f\"Missing required columns: {missing_cols}\")\n\n    # 2. Check categorical domains\n    invalid_pclass = set(df[\"Pclass\"].dropna().unique()) - VALID_PCLASS\n    if invalid_pclass:\n        logging.error(f\"Validation failed: Unexpected Pclass values: {invalid_pclass}\")\n        raise ValueError(f\"Invalid values in Pclass: {invalid_pclass}\")\n\n    invalid_sex = set(df[\"Sex\"].dropna().unique()) - VALID_SEX\n    if invalid_sex:\n        logging.error(f\"Validation failed: Unexpected Sex values: {invalid_sex}\")\n        raise ValueError(f\"Invalid values in Sex: {invalid_sex}\")\n\n    # 3. Check target column for missing values\n    if df[\"Survived\"].isnull().any():\n        logging.error(\"Validation failed: Target column 'Survived' contains null values.\")\n        raise ValueError(\"Target column 'Survived' cannot contain null values.\")\n\n    # 4. Range checks\n    if (df[\"Fare\"] < 0).any():\n        logging.error(\"Validation failed: Found negative values in 'Fare'.\")\n        raise ValueError(\"Fare values cannot be negative.\")\n\n    logging.info(\"All data validation checks passed successfully!\")\n    return True\n\nif __name__ == \"__main__\":\n    raw_data_path = os.path.join(\"data\", \"raw\", \"titanic.csv\")\n    df = load_data(raw_data_path)\n    validate_data(df)\n```\n\n| Check | Why It Matters | \n|---|---|\n| Required columns exist | If someone upstream renames `Sex` to`Gender` , your pipeline fails*here* — not deep inside model training with a cryptic error. | \n| `Pclass` ∈ {1, 2, 3} | Unknown categories would crash one-hot encoding or produce garbage features. | \n| `Survived` has no nulls | Your target variable for supervised learning must be complete. | \n| `Fare >= 0` | Negative fares are logically impossible and indicate data corruption. | \n\n**Run it:**\n\n```\npython src/data_validation.py\n# Output: All data validation checks passed successfully!\n```\n\n`src/data_preprocess.py`)\n\n``` python\nimport os\nimport sys\nimport logging\nimport pandas as pd\nfrom sklearn.model_selection import train_test_split\n\nsys.path.append(os.path.abspath(os.path.join(os.path.dirname(__file__), \"..\")))\nfrom src.data_ingestion import load_data\nfrom src.data_validation import validate_data\n\nlogging.basicConfig(level=logging.INFO, format=\"%(asctime)s - %(levelname)s - %(message)s\")\n\ndef preprocess_data(df: pd.DataFrame) -> pd.DataFrame:\n    \"\"\"\n    Cleans raw titanic data using simple pandas operations:\n    1. Fills missing values (Age with median, Embarked with mode).\n    2. Drops columns that aren't useful (PassengerId, Name, Ticket, Cabin).\n    3. Converts Sex and Embarked to numbers using one-hot encoding (get_dummies).\n    \"\"\"\n    df = df.copy()\n\n    # 1. Fill missing values\n    df[\"Age\"] = df[\"Age\"].fillna(df[\"Age\"].median())\n    df[\"Fare\"] = df[\"Fare\"].fillna(df[\"Fare\"].median())\n    df[\"Embarked\"] = df[\"Embarked\"].fillna(df[\"Embarked\"].mode()[0])\n\n    # 2. Drop columns not needed for modeling\n    drop_cols = [\"PassengerId\", \"Name\", \"Ticket\", \"Cabin\"]\n    df = df.drop(columns=[col for col in drop_cols if col in df.columns])\n\n    # 3. Convert text/categorical columns to numbers (One-Hot Encoding)\n    df = pd.get_dummies(df, columns=[\"Sex\", \"Embarked\"], drop_first=True, dtype=int)\n\n    return df\n\ndef split_data(df: pd.DataFrame, test_size: float = 0.2, random_state: int = 42):\n    \"\"\"\n    Splits the cleaned dataframe into train and test sets.\n    \"\"\"\n    X = df.drop(columns=[\"Survived\"])\n    y = df[\"Survived\"]\n\n    X_train, X_test, y_train, y_test = train_test_split(\n        X, y, test_size=test_size, random_state=random_state, stratify=y\n    )\n\n    return X_train, X_test, y_train, y_test\n\nif __name__ == \"__main__\":\n    raw_data_path = os.path.join(\"data\", \"raw\", \"titanic.csv\")\n    df = load_data(raw_data_path)\n    validate_data(df)\n\n    cleaned_df = preprocess_data(df)\n    logging.info(f\"Cleaned data shape: {cleaned_df.shape}\")\n    print(\"\\nCleaned Data Preview:\")\n    print(cleaned_df.head())\n\n    X_train, X_test, y_train, y_test = split_data(cleaned_df)\n    logging.info(f\"Train samples: {len(X_train)}, Test samples: {len(X_test)}\")\n```\n\n`fillna` with median/mode`PassengerId` is a unique identifier (no predictive value). `Name` and `Ticket` are high-cardinality text. `Cabin` is >70% null.`pd.get_dummies(drop_first=True)`` Sex` (male/female) → `Sex_male` (1/0) and `Embarked` (C/Q/S) → `Embarked_Q`, `Embarked_S`. The `drop_first` avoids the `stratify=y`\n\n```\npython src/data_preprocess.py\n# Output:\n# Cleaned data shape: (891, 9)\n# Train samples: 712, Test samples: 179\n```\n\n`src/train.py`)\nThis is where most tutorials stop. But we're not just training a model — we're **logging every experiment** so we can compare runs, reproduce results, and promote the best model to production.\n\n``` python\nimport os\nimport sys\nimport logging\nimport joblib\nimport mlflow\nfrom sklearn.ensemble import RandomForestClassifier\nfrom sklearn.metrics import (\n    accuracy_score,\n    precision_score,\n    recall_score,\n    f1_score\n)\n\nsys.path.append(\n    os.path.abspath(\n        os.path.join(os.path.dirname(__file__), \"..\")\n    )\n)\nfrom src.data_ingestion import load_data\nfrom src.data_preprocess import preprocess_data, split_data\n\nlogging.basicConfig(\n    level=logging.INFO,\n    format=\"%(asctime)s - %(levelname)s - %(message)s\"\n)\n\ndef train_model(\n    n_estimators: int = 100,\n    max_depth: int = 5,\n    random_state: int = 42\n):\n    \"\"\"\n    Train a Random Forest model and log results to MLflow.\n    \"\"\"\n    # Load raw data\n    raw_data_path = os.path.join(\"data\", \"raw\", \"titanic.csv\")\n    df = load_data(raw_data_path)\n\n    # Preprocess data\n    cleaned_df = preprocess_data(df)\n\n    # Train-test split\n    X_train, X_test, y_train, y_test = split_data(\n        cleaned_df,\n        random_state=random_state\n    )\n\n    # Configure MLflow\n    mlflow.set_tracking_uri(\"sqlite:///mlflow.db\")\n    mlflow.set_experiment(\"Titanic-Survival-Prediction\")\n\n    with mlflow.start_run():\n        logging.info(\"Training Random Forest Classifier...\")\n\n        # Train model\n        model = RandomForestClassifier(\n            n_estimators=n_estimators,\n            max_depth=max_depth,\n            random_state=random_state\n        )\n        model.fit(X_train, y_train)\n\n        # Predictions\n        y_pred = model.predict(X_test)\n\n        # Metrics\n        acc = accuracy_score(y_test, y_pred)\n        prec = precision_score(y_test, y_pred)\n        rec = recall_score(y_test, y_pred)\n        f1 = f1_score(y_test, y_pred)\n\n        logging.info(\n            f\"Accuracy={acc:.4f}, \"\n            f\"Precision={prec:.4f}, \"\n            f\"Recall={rec:.4f}, \"\n            f\"F1={f1:.4f}\"\n        )\n\n        # Log parameters\n        mlflow.log_param(\"n_estimators\", n_estimators)\n        mlflow.log_param(\"max_depth\", max_depth)\n        mlflow.log_param(\"random_state\", random_state)\n\n        # Log metrics\n        mlflow.log_metric(\"accuracy\", acc)\n        mlflow.log_metric(\"precision\", prec)\n        mlflow.log_metric(\"recall\", rec)\n        mlflow.log_metric(\"f1_score\", f1)\n\n        # Save model locally\n        os.makedirs(\"models\", exist_ok=True)\n        model_path = os.path.join(\"models\", \"model.joblib\")\n        joblib.dump(model, model_path)\n        logging.info(f\"Model saved successfully at: {model_path}\")\n\n        # Log model artifact to MLflow\n        mlflow.log_artifact(model_path, artifact_path=\"model\")\n        logging.info(\"Experiment successfully logged to MLflow.\")\n\n    return model, acc\n\nif __name__ == \"__main__\":\n    train_model(n_estimators=100, max_depth=5)\n```\n\nEvery time `train_model()` runs, MLflow creates a unique **Run ID** and stores:\n\n| What | Why | \n|---|---|\n| **Parameters** (`n_estimators` ,`max_depth` ) | So you can compare what configuration produced each result. | \n| **Metrics** (`accuracy` ,`f1_score` ) | Track model performance across experiments. | \n| **Artifacts** (`model.joblib` ) | The actual trained binary, versioned and downloadable. | \n\n```\npython src/train.py\n# Output: Accuracy=0.7821, Precision=0.7885, Recall=0.5942, F1=0.6777\n\n# Launch the MLflow dashboard:\npython -m mlflow ui\n# Open http://localhost:5000 in your browser\n```\n\nYou can now visually compare multiple training runs, their hyperparameters, and metrics side-by-side in the MLflow UI.\n\n`api/main.py`)\nEnd-users don't run Python scripts. They send HTTP requests. Our FastAPI service:\n\n``` python\nimport os\nimport time\nimport joblib\nimport pandas as pd\nfrom fastapi import FastAPI, HTTPException, Response\nfrom pydantic import BaseModel, Field\nfrom prometheus_client import Counter, Histogram, generate_latest, CONTENT_TYPE_LATEST\n\n# 1. Initialize FastAPI app\napp = FastAPI(\n    title=\"Titanic Survival Prediction API\",\n    description=\"Production-ready inference service with Prometheus monitoring.\",\n    version=\"1.0.0\"\n)\n\n# 2. Prometheus Metrics definitions\nREQUEST_COUNT = Counter(\n    \"api_request_count_total\", \n    \"Total HTTP requests received\", \n    [\"method\", \"endpoint\", \"status\"]\n)\nREQUEST_LATENCY = Histogram(\n    \"api_request_latency_seconds\", \n    \"Histogram of request latencies in seconds\", \n    [\"endpoint\"]\n)\nPREDICTION_COUNT = Counter(\n    \"model_prediction_count_total\", \n    \"Total predictions generated by the model\", \n    [\"prediction\"]\n)\n\n# 3. Path to trained model\nMODEL_PATH = os.path.join(\"models\", \"model.joblib\")\nif not os.path.exists(MODEL_PATH):\n    raise FileNotFoundError(f\"Trained model not found at {MODEL_PATH}!\")\n\nmodel = joblib.load(MODEL_PATH)\n\n# 4. Pydantic Schemas\nclass PassengerInput(BaseModel):\n    Pclass: int = Field(..., ge=1, le=3, description=\"Ticket class (1 = 1st, 2 = 2nd, 3 = 3rd)\")\n    Sex: str = Field(..., description=\"Gender: 'male' or 'female'\")\n    Age: float = Field(..., ge=0, le=120, description=\"Age in years\")\n    SibSp: int = Field(..., ge=0, description=\"Number of siblings/spouses aboard\")\n    Parch: int = Field(..., ge=0, description=\"Number of parents/children aboard\")\n    Fare: float = Field(..., ge=0.0, description=\"Passenger fare\")\n    Embarked: str = Field(..., description=\"Port of Embarkation: 'C', 'Q', or 'S'\")\n\nclass PredictionResponse(BaseModel):\n    survived: bool\n    survival_probability: float\n\n# 5. Prometheus Scrape Endpoint\n@app.get(\"/metrics\")\ndef get_metrics():\n    \"\"\"Exposes Prometheus application metrics.\"\"\"\n    return Response(content=generate_latest(), media_type=CONTENT_TYPE_LATEST)\n\n@app.get(\"/\")\ndef health_check():\n    return {\"status\": \"healthy\", \"service\": \"titanic-survival-prediction\"}\n\n@app.post(\"/predict\", response_model=PredictionResponse)\ndef predict(passenger: PassengerInput):\n    start_time = time.time()\n    try:\n        input_data = pd.DataFrame([{\n            \"Pclass\": passenger.Pclass,\n            \"Age\": passenger.Age,\n            \"SibSp\": passenger.SibSp,\n            \"Parch\": passenger.Parch,\n            \"Fare\": passenger.Fare,\n            \"Sex_male\": 1 if passenger.Sex.lower() == \"male\" else 0,\n            \"Embarked_Q\": 1 if passenger.Embarked.upper() == \"Q\" else 0,\n            \"Embarked_S\": 1 if passenger.Embarked.upper() == \"S\" else 0,\n        }])\n\n        prediction = model.predict(input_data)[0]\n        probability = model.predict_proba(input_data)[0][1]\n\n        # Record metrics\n        PREDICTION_COUNT.labels(prediction=str(int(prediction))).inc()\n        REQUEST_COUNT.labels(method=\"POST\", endpoint=\"/predict\", status=\"200\").inc()\n        REQUEST_LATENCY.labels(endpoint=\"/predict\").observe(time.time() - start_time)\n\n        return PredictionResponse(\n            survived=bool(prediction == 1),\n            survival_probability=round(float(probability), 4)\n        )\n\n    except Exception as e:\n        REQUEST_COUNT.labels(method=\"POST\", endpoint=\"/predict\", status=\"500\").inc()\n        raise HTTPException(status_code=500, detail=str(e))\n```\n\nNotice the manual construction of `Sex_male`, `Embarked_Q`, and `Embarked_S`. These **must exactly match** the dummy columns created during training by `pd.get_dummies(..., drop_first=True)`. If you forget `Embarked_Q` or add `Sex_female` instead of `Sex_male`, the model will silently produce garbage predictions — no errors, just wrong numbers.\n\n**Request:**\n\n```\n{\n  \"Pclass\": 1,\n  \"Sex\": \"female\",\n  \"Age\": 29.0,\n  \"SibSp\": 0,\n  \"Parch\": 0,\n  \"Fare\": 100.0,\n  \"Embarked\": \"S\"\n}\n```\n\n**Response:**\n\n```\n{\n  \"survived\": true,\n  \"survival_probability\": 0.9200\n}\nuvicorn api.main:app --reload --port 8000\n# Open http://localhost:8000/docs for interactive Swagger UI\n```\n\nTests are CI/CD guardrails. If any test fails, deployment stops automatically.\n\n`tests/test_model.py` — Model Integrity Tests\n\n``` python\nimport os\nimport joblib\nimport pandas as pd\n\ndef test_model_file_exists():\n    \"\"\"Verify the trained model artifact exists on disk.\"\"\"\n    model_path = os.path.join(\"models\", \"model.joblib\")\n    assert os.path.exists(model_path), \"Model artifact 'model.joblib' is missing!\"\n\ndef test_model_prediction_output():\n    \"\"\"Verify model can take a sample input row and output 0 or 1.\"\"\"\n    model_path = os.path.join(\"models\", \"model.joblib\")\n    model = joblib.load(model_path)\n\n    sample = pd.DataFrame([{\n        \"Pclass\": 3,\n        \"Age\": 22.0,\n        \"SibSp\": 1,\n        \"Parch\": 0,\n        \"Fare\": 7.25,\n        \"Sex_male\": 1,\n        \"Embarked_Q\": 0,\n        \"Embarked_S\": 1\n    }])\n\n    pred = model.predict(sample)\n    assert pred[0] in [0, 1], f\"Unexpected prediction value: {pred[0]}\"\n```\n\n`tests/test_api.py` — API Contract Tests\n\n``` python\nfrom fastapi.testclient import TestClient\nfrom api.main import app\n\nclient = TestClient(app)\n\ndef test_health_check():\n    \"\"\"Test GET / returns healthy status.\"\"\"\n    response = client.get(\"/\")\n    assert response.status_code == 200\n    assert response.json() == {\"status\": \"healthy\", \"service\": \"titanic-survival-prediction\"}\n\ndef test_predict_endpoint_valid_input():\n    \"\"\"Test POST /predict with valid passenger JSON.\"\"\"\n    payload = {\n        \"Pclass\": 1,\n        \"Sex\": \"female\",\n        \"Age\": 29.0,\n        \"SibSp\": 0,\n        \"Parch\": 0,\n        \"Fare\": 100.0,\n        \"Embarked\": \"S\"\n    }\n    response = client.post(\"/predict\", json=payload)\n    assert response.status_code == 200\n\n    data = response.json()\n    assert \"survived\" in data\n    assert \"survival_probability\" in data\n    assert isinstance(data[\"survived\"], bool)\n    assert 0.0 <= data[\"survival_probability\"] <= 1.0\n\ndef test_predict_endpoint_invalid_input():\n    \"\"\"Test POST /predict fails when invalid data is passed.\"\"\"\n    invalid_payload = {\n        \"Pclass\": 1,\n        \"Sex\": \"female\",\n        \"Age\": -5.0,  # Negative age should be rejected by Pydantic ge=0\n        \"SibSp\": 0,\n        \"Parch\": 0,\n        \"Fare\": 50.0,\n        \"Embarked\": \"S\"\n    }\n    response = client.post(\"/predict\", json=invalid_payload)\n    assert response.status_code == 422  # Unprocessable Entity\n```\n\n**Run:**\n\n```\npytest tests/ -v\n# tests/test_model.py::test_model_file_exists PASSED\n# tests/test_model.py::test_model_prediction_output PASSED\n# tests/test_api.py::test_health_check PASSED\n# tests/test_api.py::test_predict_endpoint_valid_input PASSED\n# tests/test_api.py::test_predict_endpoint_invalid_input PASSED\n```\n\nYour model works on your machine with Python 3.13 on Windows. But it might fail on an Ubuntu cloud server because of missing C++ build tools, conflicting package versions, or different file paths. Docker packages your Python version, dependencies, model artifact, and FastAPI server into an isolated, reproducible Linux container.\n\n`Dockerfile`\n\n```\n# 1. Use an official lightweight Python runtime\nFROM python:3.11-slim\n\n# 2. Set environment variables\nENV PYTHONDONTWRITEBYTECODE=1 \\\n    PYTHONUNBUFFERED=1\n\n# 3. Set working directory inside container\nWORKDIR /app\n\n# 4. Install dependencies\nCOPY requirements.txt .\nRUN pip install --no-cache-dir -r requirements.txt\n\n# 5. Copy necessary application code and models\nCOPY api/ ./api/\nCOPY src/ ./src/\nCOPY models/ ./models/\n\n# 6. Expose the port FastAPI runs on\nEXPOSE 8000\n\n# 7. Command to run the application using Uvicorn\nCMD [\"uvicorn\", \"api.main:app\", \"--host\", \"0.0.0.0\", \"--port\", \"8000\"]\n```\n\nNotice that `COPY requirements.txt` and `RUN pip install` come **before** `COPY api/` and `COPY src/`. Docker caches layers. If you only change your source code (not dependencies), Docker skips the entire pip install step and rebuilds in ~2 seconds instead of 2 minutes.\n\n`.dockerignore`\n\n```\nvenv/\n.env\n__pycache__/\n*.pyc\n.git/\n.dvc/\ndata/raw/\nmlruns/\nmlflow.db\ntests/\nnotebooks/\n```\n\n`docker-compose.yml`\n\n```\nservices:\n  titanic-api:\n    image: titanic-mlops\n    container_name: titanic-mlops-api\n    ports:\n      - \"8000:8000\"\n    restart: unless-stopped\n\n  prometheus:\n    image: prom/prometheus:latest\n    container_name: titanic-prometheus\n    ports:\n      - \"9090:9090\"\n    volumes:\n      - ./monitoring/prometheus.yml:/etc/prometheus/prometheus.yml\n    restart: unless-stopped\n# Build the image\ndocker build -t titanic-mlops .\n\n# Run with Docker Compose (API + Prometheus)\ndocker compose up -d\n```\n\nNow visit:\n\nEvery push to `main` triggers an automated pipeline that validates data, trains the model, runs tests, and builds the Docker image — all on GitHub's free cloud runners.\n\n`.github/workflows/ci_cd.yml`\n\n```\nname: Titanic MLOps Pipeline\n\non:\n  push:\n    branches: [ \"main\", \"master\" ]\n  pull_request:\n    branches: [ \"main\", \"master\" ]\n\njobs:\n  mlops_pipeline:\n    runs-on: ubuntu-latest\n\n    steps:\n    - name: Checkout Code\n      uses: actions/checkout@v4\n\n    - name: Set up Python 3.11\n      uses: actions/setup-python@v5\n      with:\n        python-version: \"3.11\"\n        cache: 'pip'\n\n    - name: Install Dependencies\n      run: |\n        python -m pip install --upgrade pip\n        pip install -r requirements.txt\n\n    - name: Run Data Validation\n      run: |\n        python src/data_validation.py\n\n    - name: Train Model\n      run: |\n        python src/train.py\n\n    - name: Run Pytest Suite\n      env:\n        PYTHONPATH: .\n      run: |\n        pytest tests/ -v\n\n    - name: Build Docker Image\n      run: |\n        docker build -t titanic-mlops-api:latest .\n```\n\n`PYTHONPATH: .` Fix\nOn Linux CI runners, the project root isn't automatically in Python's import search path. Without `PYTHONPATH: .`, pytest can't find the `api` module and fails with `ModuleNotFoundError: No module named 'api'`.\n\n**Push and watch it run:**\n\n```\ngit add .\ngit commit -m \"Add CI/CD pipeline\"\ngit push origin main\n```\n\nCheck the **Actions** tab on your GitHub repository to see the pipeline execute live.\n\nIn our `api/main.py`, we defined three metric types:\n\n| Metric | Type | What It Measures | \n|---|---|---|\n| `api_request_count_total` | Counter | Total HTTP requests by endpoint and status code | \n| `api_request_latency_seconds` | Histogram | Inference response time in seconds | \n| `model_prediction_count_total` | Counter | How many survived (1) vs died (0) predictions | \n\n`monitoring/prometheus.yml`\n\n```\nglobal:\n  scrape_interval: 5s\n\nscrape_configs:\n  - job_name: \"titanic-api\"\n    metrics_path: \"/metrics\"\n    static_configs:\n      - targets: [\"titanic-api:8000\"]\n```\n\nPrometheus pulls from the FastAPI `/metrics` endpoint every 5 seconds and stores time-series data. \n\nImagine your model is deployed and serving 10,000 requests per day. Prometheus helps you detect:\n\n`survived=0` when it's normally 38% → something is wrong with the incoming data.\nAfter running some predictions through the API, visit `http://localhost:9090` and query:\n\n```\napi_request_count_total\nmodel_prediction_count_total\nrate(api_request_latency_seconds_sum[5m]) / rate(api_request_latency_seconds_count[5m])\n```\n\nA model trained on 1912 Titanic passengers assumes certain demographic distributions — median age ~28, mostly 3rd class, mostly male. If the model is deployed live and incoming traffic suddenly changes (e.g., all passengers are age 60+ with fares >$300), the model won't crash — it will silently produce **unreliable, inaccurate predictions**.\n\nThis is called **Data Drift** (or Covariate Shift), and it's the #1 silent killer of production ML systems.\n\n`src/drift_detection.py`\n\n``` python\nimport os\nimport sys\nimport logging\nimport pandas as pd\n\n# Support both new and older Evidently versions\ntry:\n    from evidently.report import Report\n    from evidently.metric_preset import DataDriftPreset, DataQualityPreset\nexcept ModuleNotFoundError:\n    from evidently.legacy.report import Report\n    from evidently.legacy.metric_preset import DataDriftPreset, DataQualityPreset\n\nsys.path.append(os.path.abspath(os.path.join(os.path.dirname(__file__), \"..\")))\nfrom src.data_ingestion import load_data\nfrom src.data_preprocess import preprocess_data\n\nlogging.basicConfig(level=logging.INFO, format=\"%(asctime)s - %(levelname)s - %(message)s\")\n\ndef generate_drift_report(\n    reference_data: pd.DataFrame, \n    current_data: pd.DataFrame, \n    output_html_path: str = \"monitoring/drift_report.html\"\n):\n    \"\"\"\n    Compares reference (training) data with current (production) data\n    and generates an interactive HTML drift dashboard.\n    \"\"\"\n    logging.info(\"Building Evidently Data Drift and Data Quality Report...\")\n\n    features = [col for col in reference_data.columns if col != \"Survived\"]\n    ref_df = reference_data[features]\n    curr_df = current_data[features]\n\n    report = Report(metrics=[\n        DataDriftPreset(),\n        DataQualityPreset()\n    ])\n\n    report.run(reference_data=ref_df, current_data=curr_df)\n\n    os.makedirs(os.path.dirname(output_html_path), exist_ok=True)\n    report.save_html(output_html_path)\n\n    logging.info(f\"Drift report successfully generated at: {output_html_path}\")\n    return output_html_path\n\nif __name__ == \"__main__\":\n    # Load baseline reference data\n    raw_path = os.path.join(\"data\", \"raw\", \"titanic.csv\")\n    df = load_data(raw_path)\n    reference_df = preprocess_data(df)\n\n    # Simulate \"Current\" production data with intentional drift!\n    current_df = reference_df.copy()\n    current_df[\"Age\"] = current_df[\"Age\"] + 25.0  # Simulated drift in Age\n    current_df[\"Fare\"] = current_df[\"Fare\"] * 3.5  # Simulated drift in Fare\n\n    # Generate drift report\n    report_file = generate_drift_report(reference_df, current_df)\n    print(f\"\\nReport ready! Open in browser: {os.path.abspath(report_file)}\")\n```\n\nEvidently runs statistical tests on every feature:\n\n| Feature Type | Statistical Test | What It Measures | \n|---|---|---|\n| Numerical ( `Age` ,`Fare` ) | Kolmogorov-Smirnov test | Whether two distributions are from the same population | \n| Categorical ( `Pclass` ,`Sex_male` ) | Chi-Square test | Whether category proportions have changed | \n\nIf the p-value falls below the threshold (default 0.05), the feature is flagged as **drifted**.\n\n```\npython src/drift_detection.py\n# Drift report successfully generated at: monitoring/drift_report.html\n```\n\nOpen `monitoring/drift_report.html` in your browser to see the interactive Evidently dashboard with per-feature distribution charts, p-values, and drift flags.\n\nLet's step back and appreciate what we've built:\n\n```\n┌──────────────────────────────────────────────────────┐\n│                  Developer Pushes Code               │\n└────────────────────────┬─────────────────────────────┘\n                         │\n                         ▼\n┌──────────────────────────────────────────────────────┐\n│              GitHub Actions CI/CD Pipeline            │\n│                                                      │\n│  ✅ Install Dependencies                             │\n│  ✅ Validate Data Schema                             │\n│  ✅ Train Model + Log to MLflow                      │\n│  ✅ Run Pytest Suite (Model + API tests)             │\n│  ✅ Build Docker Image                               │\n└────────────────────────┬─────────────────────────────┘\n                         │\n                         ▼\n┌──────────────────────────────────────────────────────┐\n│            Docker Container (Production)             │\n│                                                      │\n│  FastAPI (:8000)                                     │\n│  ├── POST /predict  → Model Inference                │\n│  ├── GET  /metrics  → Prometheus Scrape Endpoint     │\n│  └── GET  /docs     → Interactive Swagger UI         │\n└────────────────────────┬─────────────────────────────┘\n                         │\n              ┌──────────┴──────────┐\n              ▼                     ▼\n┌────────────────────┐  ┌────────────────────┐\n│  Prometheus (:9090) │  │  Evidently AI       │\n│                    │  │                    │\n│  Request Counts    │  │  Data Drift        │\n│  Latency Tracking  │  │  Feature Drift     │\n│  Prediction Dist.  │  │  Quality Metrics   │\n└────────────────────┘  └────────────────────┘\n```\n\n**Data Versioning (DVC)**: Never commit raw data to Git. Track it with DVC for reproducibility and rollback.\n\n**Validate Before You Train**: Schema checks, constraint checks, and missing-value guardrails catch data problems *before* they silently corrupt your model.\n\n**Track Every Experiment (MLflow)**: Every training run should log parameters, metrics, and artifacts. \"I think I got 82% accuracy last Tuesday\" is not acceptable in production.\n\n**Serve via API, Not Scripts**: End-users send HTTP requests. FastAPI with Pydantic validation gives you type-safe, self-documenting endpoints.\n\n**Test Everything Automatically**: Model integrity tests + API contract tests in Pytest, executed automatically in CI.\n\n**Containerize for Reproducibility**: Docker eliminates \"works on my machine\" problems entirely.\n\n**Automate with CI/CD**: GitHub Actions ensures every push is validated, tested, trained, and container-built before reaching production.\n\n**Monitor in Production**: Prometheus gives you real-time visibility into traffic, latency, and prediction behavior. Without it, you're flying blind.\n\n**Detect Drift**: Evidently AI catches the silent killer of ML systems — when production data distribution shifts away from what the model was trained on.\n\n*If you found this walkthrough helpful, consider giving the [GitHub repo](https://github.com/naman-0804/End-to-end-MLOPS-Pipeline) a ⭐ and sharing this post with someone learning MLOps!*", "url": "https://wpnews.pro/news/building-a-production-grade-end-to-end-mlops-pipeline-from-scratch", "canonical_source": "https://dev.to/naman_2004/building-a-production-grade-end-to-end-mlops-pipeline-from-scratch-l9h", "published_at": "2026-09-18 18:21:06+00:00", "updated_at": "2026-09-18 18:53:08.235514+00:00", "lang": "en", "topics": ["mlops", "machine-learning", "ai-infrastructure", "developer-tools"], "entities": ["GitHub", "DVC", "MLflow", "FastAPI", "Docker", "Prometheus", "Evidently AI", "scikit-learn"], "alternates": {"html": "https://wpnews.pro/news/building-a-production-grade-end-to-end-mlops-pipeline-from-scratch", "markdown": "https://wpnews.pro/news/building-a-production-grade-end-to-end-mlops-pipeline-from-scratch.md", "text": "https://wpnews.pro/news/building-a-production-grade-end-to-end-mlops-pipeline-from-scratch.txt", "jsonld": "https://wpnews.pro/news/building-a-production-grade-end-to-end-mlops-pipeline-from-scratch.jsonld"}}