Breath-Dx is a new open benchmark for electronic-nose (eNose) disease screening that forces models to survive sensor drift, not just random cross-validation. Authors released the S-OH dataset on Hugging Face, code on GitHub, and arXiv paper 2611.15348 so teams can reproduce ten baseline families on 1,234 patients.
If you build health ML, the headline is simple: every method they tested lost 0.05–0.12 ROC AUC when evaluation respected time, exposing lab scores that would fail in clinics.
TL;DR #
| Item | Detail |
|---|---|
| Data | S-OH :1,234 patients ,9 diagnostic groups,17-channel series |
| Methods | 10 models across tabular, CNN, GNN, and TS architectures |
| Drift effect | −0.05 to −0.12 AUC vs random CV forall families |
| Peak temporal AUC | ~0.74–0.76 on somelung conditions |
| Release | HF dataset ,GitHub splits + eval protocols |
| Award | Authors cite IEEE ICDM 2026 Applied Track runner-up |
Why eNose research needed a harder benchmark #
Electronic noses promise portable, non-invasive screening from exhaled breath. Most papers use small private cohorts and random splits, which hide drift: sensors age, environments shift, and batch effects correlate with time.
Breath-Dx standardizes temporal train/test partitions on public data so ROC AUC reflects deployment reality, the same discipline InclusionMed pushes for healthcare AI indices.
What the S-OH dataset contains #
Scent of Health (S-OH) is described in the paper as the largest public clinical eNose collection to date:
- 1,234 patients
- Nine diagnostic groups
- 17-channel multivariate time series per participant
- Explicit temporal splits accounting forsensor drift
Download via ivanpodd/S-OH and read the dataset card for collection protocol and label definitions before fine-tuning frontier models.
Methods and results under temporal validation #
Breath-Dx evaluates four families:
- Feature-based tabular learners
- Convolutional models
- Graph-based models
- Time series architectures
Under temporal validation:
- All ten methods lose0.05–0.12 AUC versusrandom CV
- Recurrent models show themost consistent performance across diseases
- Specialized TS models achievehighest peaks on some tasks
- Best temporal AUC ~0.74–0.76 forsome lung conditions ; othersnear chance
That spread matters for product roadmaps: screening UX must communicate uncertainty per condition, not a single "AI accuracy" banner.
Monitoring deployment drift when sensor baselines shift over calendar time
Demographics and confounding #
Authors emphasize demographic confounders:
- Breath signals add independent value forsome diseases
- For others , signal addslittle beyond age
Benchmark protocols in the GitHub repo document how to report stratified metrics. Copy that pattern when you ship internal health agents audited like GPT-6 healthcare pilots—** subgroup tables** are not optional.
How to run the benchmark #
1. Clone **[github.com/ivanpodd/Breath-Dx](https://github.com/ivanpodd/Breath-Dx?ref=explainx)** .
2. Pull **[ivanpodd/S-OH](https://huggingface.co/datasets/ivanpodd/S-OH?ref=explainx)** with the**published temporal splits** .
- Reproduce baselines before swapping incustom transformers .
- Log drift curves overcalendar time , not onlyaggregate AUC .
For time-series craft, cross-read multi-branch transformer KV work when you design long-context breath windows—different domain, same leakage pitfalls.
Implications for AI-for-health builders #
Do not ship random-CV numbers in investor decks if temporal splits collapse performance. Breath-Dx gives citeable evidence that drift is universal in their study, not an edge case.
Regulatory path stays long. Benchmark success does not clear FDA/EMA pathways; treat Breath-Dx as R&D hygiene, like Sakana JFinBench for finance agents.
Open data helps audits. Public splits enable third-party replication, reducing vendor-only claims—aligned with Meta Brain2Qwerty style reproducibility expectations.
Conference recognition #
In the Hugging Face paper thread, authors note IEEE ICDM 2026 Best Paper Runner-Up (Applied Track). That signals peer review interest, not clinical endorsement.
Limitations to state plainly #
- Nine groups stillexclude many diseases.
- eNose hardware varies;S-OH protocols may not transfer toconsumer gadgets .
- Temporal splits punishdrift butdo not simulate fullprospective trials .
Comparison with other health ML evaluation habits #
Many Kaggle-style medical competitions still publish random patient splits. Breath-Dx argues that practice is especially toxic for eNose because sensor aging correlates with calendar time. The 0.05–0.12 AUC penalty is not a rounding error; for screening at fixed sensitivity, it can flip cost-effectiveness models used in hospital procurement.
If you previously cited non-temporal AUC in decks, rerun tables with Breath-Dx splits before regulatory meetings. Align messaging with InclusionMed gaps on demographic reporting.
Integration with modern time-series foundation models #
Teams experimenting with Chronos, Moirai, or custom transformers should treat Breath-Dx as a cheap sanity check before million-parameter finetunes. Pretraining on generic series does not immunize models against drift; the paper shows universal degradation under temporal validation.
Start with released recurrent baselines to establish floor metrics, then swap in your architecture while holding splits fixed. Log calibration curves—** AUC alone** hides overconfidence when prevalence shifts.
Data card fields to read before training #
On ivanpodd/S-OH, verify label definitions per diagnostic group, acquisition dates tied to each patient row, sensor firmware revisions if documented, and missing channel handling rules.
Mis-reading label granularity (staging versus binary screening) is a common silent bug in health RAG pipelines, similar to JFinBenchcredential-level mistakes in finance agents.
What clinicians will ask in pilot meetings #
Prepare answers for: (1) what happens when the sensor is recalibrated mid-study—point to temporal split design; (2) whether breath signal is independent of age—cite demographic confounder analysis; (3) which diseases are near chance even in lab data—quote per-condition AUC bands from the paper.
Open science citation hygiene #
When you write about Breath-Dx, cite arXiv 2611.15348, the GitHub repository, and the Hugging Face dataset card—not aggregator rewrites. That supports reproducibility norms we use in Casey Handmer’s physics literature audit and other open benchmark coverage.
Source citation discipline for clinical AI benchmark posts that link datasets and code releases
Model family details builders should reproduce #
Breath-Dx spans feature engineering pipelines and end-to-end neural models. Tabular baselines extract hand-crafted statistics from 17-channel traces—mean, slope, spectral bins—then feed gradient boosted trees or logistic heads. Convolutional baselines treat each breath sample as a 2D map (time × channel). Graph models connect sensor nodes with learned edges, useful when cross-channel correlation shifts under drift.
Time-series transformers and State-space variants chase long-range dependencies across entire breath profiles. The paper’s point is not “** transformers always win**”; it is “** every family overfits random splits**.” When you port Hy4-class multimodal models or small SLMs to S-OH, keep the temporal split as your primary metric, with random CV labeled diagnostic only.
Designing a drift monitoring stack for eNose products #
Production eNose programs need three layers Breath-Dx implies but does not ship as software:
- Reference gas standards on a schedule tied tocalendar time , not sample count alone.
- Population shift dashboards comparingembedding centroids week over week.
- Human override whenAUC drops belowpre-registered thresholds.
Map layer 3 to Americans’ negative AI sentiment polls: consumers tolerate assistive screening only when false positives are explained and appealable.
Regulatory and ethics framing #
EU AI Act high-risk framing may apply if breath models inform triage without clinician sign-off. Breath-Dx does not solve classification under law; it supplies evidence that uncalibrated lab AUC misleads.
Document informed consent for biometric breath collection, especially when workplace wellness programs also collect camera or wearable telemetry that employees may read as surveillance.
Extending the benchmark responsibly #
Authors invite reproducible research. If you contribute new models:
- Publish seeded splits unchanged from GitHub.
- Report per-disease tables, not onemacro AUC .
- Disclose hardware calibration if youre-record supplemental patients.
Avoid merging S-OH with private cohorts without domain adaptation ablations—drift will reappear.
Pairing Breath-Dx with agentic health workflows #
If you wire eNose scores into LLM triage agents, treat model output as features, not diagnoses. Agents should surface confidence, collection timestamp, and sensor batch ID in every summary—metadata Breath-Dx implies but agents often drop. Read AWS responsible agent roadmaps before automating patient-facing messages from breath classifiers. Hospital IT should segment research laptops running Breath-Dx baselines from EHR-connected production VLANs, the same blast-radius discipline we recommend after Java4Shell patching sprints.
Future cohort and hardware directions #
Authors point toward multi-site sensors and prospective drift correction algorithms. Until those land, treat Breath-Dx as the reference implementation for honest eNose metrics, not as proof that any single architecture is clinic-ready.
When you present results to hospital innovation boards, lead with temporal AUC and per-disease tables. Add a one-slide drift story explaining why random CV overstated past pilots. That framing prevents procurement teams from comparing Breath-Dx numbers to vendor brochures trained on leaky splits.
Workshop path for health ML teams #
Structured AI workshops help teams adopt leakage-safe validation before clinical pilots. Pair Breath-Dx exercises with InclusionMed gap analysis for equity reporting.