{"slug": "tabpfn-vs-gradient-boosting-tiny-medical-datasets-showdown", "title": "TabPFN vs Gradient Boosting: Tiny Medical Datasets Showdown", "summary": "A developer benchmarked TabPFN against tuned XGBoost and LightGBM on a 142-patient, 31-feature clinical biomarker classification task using 5-fold stratified cross-validation repeated 10 times. TabPFN achieved a mean AUC of 0.81 (std 0.07) and mean average precision of 0.64, outperforming tuned XGBoost (0.77 AUC, 0.58 AP) and LightGBM (0.76 AUC, 0.57 AP), while requiring no hyperparameter tuning and running a single fold in under 0.3 seconds on CPU versus several minutes per fold for the boosted-tree baselines.", "body_md": "If you've worked on a clinical ML project, you know the dataset size everyone pretends doesn't exist: 80 patients. 150 patients. Maybe 300 if the hospital is generous and the IRB is fast. Meanwhile every tabular ML tutorial on the internet assumes you have 100k+ rows and can afford to tune a gradient boosted tree for three days on a GPU cluster.\n\nI recently had to pick a model for a biomarker classification task — 142 patients, 31 features (labs, vitals, a few engineered ratios), binary outcome, heavily class-imbalanced, and noisy because half the labs were drawn under different protocols. The usual suspects were XGBoost and LightGBM. But TabPFN kept showing up in my feed as the thing you're supposed to try on small tabular data now. So I actually ran the comparison instead of trusting a benchmark table.\n\nThis post is what I found, with the caveats that actually matter for a healthcare dataset instead of a Kaggle leaderboard.\n\nTabPFN (Tabular Prior-Data Fitted Network) isn't trained on your data in the traditional sense. It's a transformer pretrained once on millions of synthetic datasets, and at inference time it does in-context learning: you feed it your train set and test set together, and it does a single forward pass to produce predictions. No gradient descent on your data, no hyperparameter search, no early stopping.\n\nThe catch (as of the versions most people are using): it was designed for datasets up to a few thousand rows and a few hundred features. That's not a limitation for us — that's literally our use case.\n\nbash\n\npip install tabpfn xgboost lightgbm scikit-learn shap\n\nDataset: 142 rows, 31 numeric features, binary label, ~22% positive class. 5-fold stratified CV, repeated 10 times to get stable estimates given how small the test folds are (roughly 28 rows per fold — variance is brutal at this scale).\n\npython\n\nimport numpy as np\n\nfrom sklearn.model_selection import RepeatedStratifiedKFold\n\nfrom sklearn.metrics import roc_auc_score, average_precision_score\n\nfrom tabpfn import TabPFNClassifier\n\nimport xgboost as xgb\n\nimport lightgbm as lgb\n\nX, y = load_clinical_data() # (142, 31), binary y\n\ncv = RepeatedStratifiedKFold(n_splits=5, n_repeats=10, random_state=42)\n\ndef evaluate(model_fn):\n\n    aucs, aps = [], []\n\n    for train_idx, test_idx in cv.split(X, y):\n\n        model = model_fn()\n\n        model.fit(X[train_idx], y[train_idx])\n\n        preds = model.predict_proba(X[test_idx])[:, 1]\n\n        aucs.append(roc_auc_score(y[test_idx], preds))\n\n        aps.append(average_precision_score(y[test_idx], preds))\n\n    return np.mean(aucs), np.std(aucs), np.mean(aps)\n\nTabPFN has no real \"training\" step — it's a forward pass. On this dataset size, fit+predict for a single fold took under 0.3 seconds on CPU. XGBoost and LightGBM, once you include a proper hyperparameter search (which you need, because default params on 140 rows will overfit or underfit depending on the day), took several minutes per fold with a reasonable Optuna search.\n\npython\n\ndef tabpfn_model():\n\n    return TabPFNClassifier(device=\"cpu\", N_ensemble_configurations=32)\n\ndef xgb_model():\n\n    return xgb.XGBClassifier(\n\n        n_estimators=150, max_depth=3, learning_rate=0.05,\n\n        subsample=0.7, colsample_bytree=0.7,\n\n        reg_lambda=2.0, eval_metric=\"logloss\"\n\n    )\n\ntabpfn_auc = evaluate(tabpfn_model)\n\nxgb_auc = evaluate(xgb_model)\n\nThe time savings aren't just convenience. On small clinical datasets, your \"hyperparameter tuning\" is really a search over an enormous space of near-identical local optima, and the result is unstable across random seeds. TabPFN sidesteps that instability entirely because there's nothing to tune — which is either a feature or a limitation depending on how much you value control.\n\nAcross 50 CV splits:\n\n| Model | Mean AUC | Std AUC | Mean AP | \n|---|---|---|---|\n| TabPFN | 0.81 | 0.07 | 0.64 | \n| XGBoost (tuned) | 0.77 | 0.11 | 0.58 | \n| LightGBM (tuned) | 0.76 | 0.12 | 0.57 | \n| XGBoost (default params) | 0.69 | 0.15 | 0.49 | \n\nTwo things stood out. First, TabPFN's variance across folds was noticeably lower — it's less sensitive to which 28 patients ended up in the test fold, which matters a lot when your confidence interval is the whole story. Second, the gap between tuned and untuned XGBoost was huge, which is the real danger of gradient boosting on tiny data: if you don't have the discipline (or the held-out budget) to tune properly, you'll ship a model that's quietly overfit to noise in the lab values.\n\nThis isn't universal — on a second dataset I tested (180 rows, mostly categorical features with a lot of missingness), tuned LightGBM edged out TabPFN by about 2 AUC points, likely because TabPFN's synthetic pretraining prior doesn't model heavy categorical missingness patterns as well as it models continuous clinical variables.\n\nThis is where the decision usually actually gets made, not on the leaderboard. Clinicians and IRBs want to know *why* the model flagged a patient. With gradient boosting, you get mature tooling:\n\npython\n\nimport shap\n\nmodel = xgb_model().fit(X, y)\n\nexplainer = shap.TreeExplainer(model)\n\nshap_values = explainer.shap_values(X)\n\nshap.summary_plot(shap_values, X, feature_names=feature_names)\n\nTreeExplainer is exact, fast, and well understood by reviewers at this point — it's practically a checkbox item in clinical ML papers now. TabPFN interpretability is much rougher: you're stuck with KernelSHAP or permutation importance, both of which are slow and noisy precisely because the model itself has no stable internal structure to exploit, and both become even noisier on 142 rows. I got usable but visibly less stable SHAP plots out of TabPFN — rerunning with a different background sample shifted the top-5 features more than I was comfortable with.\n\nIf your deliverable includes a feature-importance figure that a clinical co-author needs to defend in a meeting, that instability is a real cost, not a nitpick.\n\nThe other practical issue: TabPFN needs your training set at inference time, every time, because prediction is still an in-context forward pass over train+test. That means:\n\n`model.pkl` you hand to a hospital IT team the way you would an XGBoost booster file.\nXGBoost/LightGBM, by contrast, produce a single portable artifact, run in milliseconds, and have first-class support in basically every deployment stack (ONNX, Treelite, PMML if you're in a particularly regulated shop). For a model going into an EHR-integrated pipeline, that maturity isn't a nice-to-have — it's often a hard requirement from the infra team.\n\nI used TabPFN for the initial exploratory phase and as a cross-check against overfitting — if tuned XGBoost dramatically outperforms TabPFN on a dataset this small, that's usually a signal the tuning found noise, not signal. For the actual deployed model, I went with a conservatively tuned LightGBM, because the interpretability tooling and deployment story were non-negotiable for this particular clinical partner.\n\nThat won't be the right call for every project. If you're doing a quick feasibility study, a hackathon, or a paper where inference-time compute isn't constrained, TabPFN's speed and low-variance performance on sub-1000-row datasets is genuinely hard to beat.\n\nHave you shipped TabPFN in an actual production or clinical pipeline, or has it stayed in the notebook for you too? I'd like to know if anyone's solved the \"carry the training set to inference\" problem cleanly.", "url": "https://wpnews.pro/news/tabpfn-vs-gradient-boosting-tiny-medical-datasets-showdown", "canonical_source": "https://dev.to/renato_silva_71eef0fc385f/tabpfn-vs-gradient-boosting-tiny-medical-datasets-showdown-jgc", "published_at": "2026-10-19 16:07:03+00:00", "updated_at": "2026-10-19 16:14:50.456689+00:00", "lang": "en", "topics": ["machine-learning", "ai-research", "ai-tools"], "entities": ["TabPFN", "XGBoost", "LightGBM", "Optuna", "scikit-learn"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/tabpfn-vs-gradient-boosting-tiny-medical-datasets-showdown", "markdown": "https://wpnews.pro/news/tabpfn-vs-gradient-boosting-tiny-medical-datasets-showdown.md", "text": "https://wpnews.pro/news/tabpfn-vs-gradient-boosting-tiny-medical-datasets-showdown.txt", "jsonld": "https://wpnews.pro/news/tabpfn-vs-gradient-boosting-tiny-medical-datasets-showdown.jsonld"}}