{"slug": "what-400-powder-blends-taught-me-about-ml-under-messy-real-world-labels", "title": "What 400 powder blends taught me about ML under messy, real-world labels", "summary": "Satyam Pandey, a research assistant at NJIT and now a Senior Quantitative Analyst at Exelon, trained Random Forest and XGBoost models on 400 pharmaceutical powder blends to predict flowability, reaching up to 85 percent accuracy, and used a variational autoencoder to generate 163 candidate formulations, 12 of which were validated experimentally with 83 percent holding up. Pandey argues that noisy physical-measurement labels undermine reported accuracy and that auditing labels—cross-checking against physical plausibility and treating high-confidence model-label disagreements as signal—matters more than tuning models. His follow-up work, accepted at the LRLS workshop at IEEE BigData 2026, applies the same label-error detection approach to EIA-930 hourly electricity demand records.", "body_md": "Most machine learning tutorials assume the labels are right. Clean target column, balanced classes, a test set you can trust. The first real dataset I worked on seriously had none of that, and it changed how I evaluate every model since.\n\nThe project was pharmaceutical powder blends, work I did as a research assistant at NJIT and later co-authored at the AIChE 2024 Annual Meeting. The dataset covered 400 powder blends. The goal was to predict flowability, a property that decides whether a blend will move through manufacturing equipment cleanly or clog it. Flowability sounds simple. In practice it depends on particle size, moisture, composition, and how the blend was handled, and the measured labels carried all the noise of a real lab.\n\nWe trained Random Forest and XGBoost models on the blend properties and reached up to 85 percent accuracy on flowability prediction. That number came with an asterisk I will get to in a moment.\n\nWe also trained a variational autoencoder to generate new candidate formulations. The VAE produced 163 generated formulations, and 12 of them were validated experimentally, with 83 percent holding up against what the model expected. Generative models get a lot of hype, but the part that mattered was the validation loop: a generated formulation is a hypothesis, not a result, until a lab bench says otherwise.\n\nHere is what the accuracy figure hides. When labels come from physical measurements, some of them are wrong. Not maliciously wrong, just wrong: a misread instrument, a sample that sat too long, a borderline case recorded as a clean pass. If a few percent of your training labels are flipped, three things happen:\n\nWe could not relabel 400 blends. Nobody had the budget for that. So the practical moves were unglamorous: cross-check labels against physical plausibility, look hard at the examples the model got \"wrong\" with high confidence, and treat disagreement between model and label as information instead of failure. Some of our most useful findings started as confident predictions that contradicted the recorded label.\n\nI now spend my days as a Senior Quantitative Analyst at Exelon, and I am still working on the same problem in a different costume. My paper accepted at the LRLS workshop at IEEE BigData 2026 is about detecting controlled errors in EIA-930 hourly electricity demand records under limited labels. Grid demand data, like lab data, contains records that are quietly wrong, and downstream forecasts inherit those errors unless someone builds the detection step.\n\nDifferent domain, same lesson: before you tune the model, audit the labels. A benchmark with verified error cases tells you more than a leaderboard score on data nobody has checked.\n\nIf you work with messy real-world data, I would like to hear how you handle label quality. It is the least glamorous part of ML and, in my experience, the part that decides whether the model survives contact with production.\n\nSatyam Pandey\n\nFounder of Brandsap, building Rankelo. Senior Quantitative Analyst at Exelon. MS in Computer Science, NJIT.\n\n[https://pandeysatyam.com](https://pandeysatyam.com)\n\nORCID: [https://orcid.org/0009-0002-9127-6960](https://orcid.org/0009-0002-9127-6960)\n\nPaper: [https://proceedings.aiche.org/conferences/aiche-annual-meeting/2024/proceeding/paper/39e-machine-learning-and-deep-learning-models-analysis-and-prediction-pharmaceutical-powder-blend-properties](https://proceedings.aiche.org/conferences/aiche-annual-meeting/2024/proceeding/paper/39e-machine-learning-and-deep-learning-models-analysis-and-prediction-pharmaceutical-powder-blend-properties)", "url": "https://wpnews.pro/news/what-400-powder-blends-taught-me-about-ml-under-messy-real-world-labels", "canonical_source": "https://dev.to/pandeysatyam/what-400-powder-blends-taught-me-about-ml-under-messy-real-world-labels-49eh", "published_at": "2026-11-15 08:16:46+00:00", "updated_at": "2026-11-15 08:20:50.614566+00:00", "lang": "en", "topics": ["machine-learning", "ai-research", "generative-ai"], "entities": ["Satyam Pandey", "NJIT", "Exelon", "AIChE", "IEEE BigData 2026", "EIA-930", "Brandsap", "Rankelo"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/what-400-powder-blends-taught-me-about-ml-under-messy-real-world-labels", "markdown": "https://wpnews.pro/news/what-400-powder-blends-taught-me-about-ml-under-messy-real-world-labels.md", "text": "https://wpnews.pro/news/what-400-powder-blends-taught-me-about-ml-under-messy-real-world-labels.txt", "jsonld": "https://wpnews.pro/news/what-400-powder-blends-taught-me-about-ml-under-messy-real-world-labels.jsonld"}}