{"slug": "one-master-dataset-two-consumers-building-a-governed-pipeline-for-bi-ml", "title": "One Master Dataset, Two Consumers: Building a Governed Pipeline for BI + ML", "summary": "A developer built a governed data pipeline that produces a single master dataset from the UCI Adult Census Income data, serving both a Power BI dashboard and four ML models for income prediction. Missing values are preserved as 'Unknown' with boolean flags for full auditability, and permutation importance across models revealed that marital status, capital gains, and education are the strongest predictors. The project highlights the risks of dataset drift and the importance of treating model selection as a business decision rather than a purely technical one.", "body_md": "A recent project of mine called for two things that are easy to build separately and dangerous to build separately: a Power BI dashboard for exploring workforce demographics, and four tuned ML models predicting income bracket from the same population.\n\nThe tempting shortcut is two datasets — one imputed and encoded for modeling, one left more human-readable for BI. I deliberately avoided that.\n\nTwo datasets drift. A fix applied to one never reaches the other, and by the second refresh cycle, \"the data\" means two different things depending on who you ask. Six months in, someone's dashboard says one number and someone's model says another, and nobody can explain why without a forensic diff.\n\nSo instead, everything here traces back to one governed pipeline (`build_master_dataset.py`\n\n) that produces a single master table — `adult_master.csv`\n\n/ `.parquet`\n\n— 48,790 rows, 26 columns, built from the UCI Adult Census Income dataset. Power BI reads the human-readable `income_bracket`\n\nand categorical columns directly. The ML pipeline reads the numeric `income_target`\n\nflag and the same underlying predictors. One source of truth, two consumers.\n\nThe raw source data uses `?`\n\nfor missing categorical values — `workclass`\n\n(5.73% missing), `occupation`\n\n(5.75%), `native_country`\n\n(1.75%). The easy move is to impute these away with the mode or drop the rows. I didn't do either.\n\nEvery missing value is relabeled `Unknown`\n\nand paired with a boolean `*_was_missing`\n\nflag. That means the fact that a value was missing stays fully auditable downstream — in the BI layer, in the model, in any report — instead of being silently absorbed into \"most common category\" and forgotten.\n\nThe Power BI layer isn't a KPI wall. It's built around one DAX pattern — `High Income Rate vs Overall`\n\n— that drops onto any breakdown (occupation, education, region, work-hours category) and immediately shows whether that segment sits above or below the population baseline. Twenty measures total, plus three sort-key columns so categories like \"Part-time (<20)\" through \"Overtime (60+)\" render in logical rather than alphabetical order.\n\nRather than trust a single model's built-in feature importance (which can vary wildly by algorithm), I computed permutation importance across all four tuned models on a held-out sample and looked for agreement.\n\nThe signal was consistent: relationship/marital status dominates, followed by capital gains, education, age, occupation, and hours worked.\n\nTwo findings were worth calling out explicitly rather than leaving buried in a model file:\n\n`fnlwgt`\n\n, a U.S. Census sampling weight included in the raw data, has effectively zero correlation with income (Pearson r = −0.006). It was excluded from modeling entirely rather than left in to add noise — a reminder that \"more columns\" isn't automatically \"more signal.\"| Model | Accuracy | Precision | Recall | F1 | ROC-AUC |\n|---|---|---|---|---|---|\n| Logistic Regression | 81.1% | 0.567 | 0.845 | 0.679 | 0.907 |\n| Random Forest | 81.7% | 0.575 | 0.863 | 0.690 | 0.917 |\n| HistGradientBoosting | 87.3% | 0.776 | 0.652 | 0.708 | 0.927 |\n| XGBoost | 83.4% | 0.604 | 0.865 | 0.712 | 0.928 |\n\nXGBoost edges out HistGradientBoosting on ROC-AUC — 0.928 vs 0.927 — but that's the wrong reason to call it the clear winner. HistGB actually has better accuracy, precision, and error rate. XGBoost's class-rebalancing (`scale_pos_weight`\n\n) trades precision for recall, catching more true high earners at the cost of more false positives.\n\nWhich one is \"better\" depends on whether the downstream decision punishes missed opportunities or wasted outreach more — a business question, not a modeling one. So that's what I wrote, instead of picking a winner by a 0.001 margin because it looks cleaner in a table.\n\nNone of this is exotic. There's no novel architecture here — it's a gradient-boosted tree model on tabular census data, the kind of problem that's been solved a thousand times. What I think is worth sharing isn't the modeling, it's the discipline around it: one pipeline instead of two, missingness that stays visible instead of vanishing, importance from four models instead of one, and a model choice explained in terms of the decision it serves rather than a leaderboard.\n\nFull write-up, including the complete methodology and caveats, is on my site: [Workforce Income Analytics — Case Study](https://lohkahmeng-portfolio.netlify.app/project-5-workforce-income.html)", "url": "https://wpnews.pro/news/one-master-dataset-two-consumers-building-a-governed-pipeline-for-bi-ml", "canonical_source": "https://dev.to/lohkahmeng/one-master-dataset-two-consumers-building-a-governed-pipeline-for-bi-ml-43i9", "published_at": "2026-07-08 06:01:51+00:00", "updated_at": "2026-07-08 06:28:47.868298+00:00", "lang": "en", "topics": ["machine-learning", "mlops", "developer-tools"], "entities": ["Power BI", "UCI Adult Census Income", "XGBoost", "HistGradientBoosting", "Random Forest", "Logistic Regression"], "alternates": {"html": "https://wpnews.pro/news/one-master-dataset-two-consumers-building-a-governed-pipeline-for-bi-ml", "markdown": "https://wpnews.pro/news/one-master-dataset-two-consumers-building-a-governed-pipeline-for-bi-ml.md", "text": "https://wpnews.pro/news/one-master-dataset-two-consumers-building-a-governed-pipeline-for-bi-ml.txt", "jsonld": "https://wpnews.pro/news/one-master-dataset-two-consumers-building-a-governed-pipeline-for-bi-ml.jsonld"}}