TabPFN vs Gradient Boosting: Tiny Medical Datasets Showdown

A developer benchmarked TabPFN against tuned XGBoost and LightGBM on a 142-patient, 31-feature clinical biomarker classification task using 5-fold stratified cross-validation repeated 10 times. TabPFN achieved a mean AUC of 0.81 (std 0.07) and mean average precision of 0.64, outperforming tuned XGBoost (0.77 AUC, 0.58 AP) and LightGBM (0.76 AUC, 0.57 AP), while requiring no hyperparameter tuning and running a single fold in under 0.3 seconds on CPU versus several minutes per fold for the boosted-tree baselines.

If you've worked on a clinical ML project, you know the dataset size everyone pretends doesn't exist: 80 patients. 150 patients. Maybe 300 if the hospital is generous and the IRB is fast. Meanwhile every tabular ML tutorial on the internet assumes you have 100k+ rows and can afford to tune a gradient boosted tree for three days on a GPU cluster. I recently had to pick a model for a biomarker classification task — 142 patients, 31 features labs, vitals, a few engineered ratios , binary outcome, heavily class-imbalanced, and noisy because half the labs were drawn under different protocols. The usual suspects were XGBoost and LightGBM. But TabPFN kept showing up in my feed as the thing you're supposed to try on small tabular data now. So I actually ran the comparison instead of trusting a benchmark table. This post is what I found, with the caveats that actually matter for a healthcare dataset instead of a Kaggle leaderboard. TabPFN Tabular Prior-Data Fitted Network isn't trained on your data in the traditional sense. It's a transformer pretrained once on millions of synthetic datasets, and at inference time it does in-context learning: you feed it your train set and test set together, and it does a single forward pass to produce predictions. No gradient descent on your data, no hyperparameter search, no early stopping. The catch as of the versions most people are using : it was designed for datasets up to a few thousand rows and a few hundred features. That's not a limitation for us — that's literally our use case. bash pip install tabpfn xgboost lightgbm scikit-learn shap Dataset: 142 rows, 31 numeric features, binary label, ~22% positive class. 5-fold stratified CV, repeated 10 times to get stable estimates given how small the test folds are roughly 28 rows per fold — variance is brutal at this scale . python import numpy as np from sklearn.model selection import RepeatedStratifiedKFold from sklearn.metrics import roc auc score, average precision score from tabpfn import TabPFNClassifier import xgboost as xgb import lightgbm as lgb X, y = load clinical data 142, 31 , binary y cv = RepeatedStratifiedKFold n splits=5, n repeats=10, random state=42 def evaluate model fn : aucs, aps = , for train idx, test idx in cv.split X, y : model = model fn model.fit X train idx , y train idx preds = model.predict proba X test idx :, 1 aucs.append roc auc score y test idx , preds aps.append average precision score y test idx , preds return np.mean aucs , np.std aucs , np.mean aps TabPFN has no real "training" step — it's a forward pass. On this dataset size, fit+predict for a single fold took under 0.3 seconds on CPU. XGBoost and LightGBM, once you include a proper hyperparameter search which you need, because default params on 140 rows will overfit or underfit depending on the day , took several minutes per fold with a reasonable Optuna search. python def tabpfn model : return TabPFNClassifier device="cpu", N ensemble configurations=32 def xgb model : return xgb.XGBClassifier n estimators=150, max depth=3, learning rate=0.05, subsample=0.7, colsample bytree=0.7, reg lambda=2.0, eval metric="logloss" tabpfn auc = evaluate tabpfn model xgb auc = evaluate xgb model The time savings aren't just convenience. On small clinical datasets, your "hyperparameter tuning" is really a search over an enormous space of near-identical local optima, and the result is unstable across random seeds. TabPFN sidesteps that instability entirely because there's nothing to tune — which is either a feature or a limitation depending on how much you value control. Across 50 CV splits: | Model | Mean AUC | Std AUC | Mean AP | |---|---|---|---| | TabPFN | 0.81 | 0.07 | 0.64 | | XGBoost tuned | 0.77 | 0.11 | 0.58 | | LightGBM tuned | 0.76 | 0.12 | 0.57 | | XGBoost default params | 0.69 | 0.15 | 0.49 | Two things stood out. First, TabPFN's variance across folds was noticeably lower — it's less sensitive to which 28 patients ended up in the test fold, which matters a lot when your confidence interval is the whole story. Second, the gap between tuned and untuned XGBoost was huge, which is the real danger of gradient boosting on tiny data: if you don't have the discipline or the held-out budget to tune properly, you'll ship a model that's quietly overfit to noise in the lab values. This isn't universal — on a second dataset I tested 180 rows, mostly categorical features with a lot of missingness , tuned LightGBM edged out TabPFN by about 2 AUC points, likely because TabPFN's synthetic pretraining prior doesn't model heavy categorical missingness patterns as well as it models continuous clinical variables. This is where the decision usually actually gets made, not on the leaderboard. Clinicians and IRBs want to know why the model flagged a patient. With gradient boosting, you get mature tooling: python import shap model = xgb model .fit X, y explainer = shap.TreeExplainer model shap values = explainer.shap values X shap.summary plot shap values, X, feature names=feature names TreeExplainer is exact, fast, and well understood by reviewers at this point — it's practically a checkbox item in clinical ML papers now. TabPFN interpretability is much rougher: you're stuck with KernelSHAP or permutation importance, both of which are slow and noisy precisely because the model itself has no stable internal structure to exploit, and both become even noisier on 142 rows. I got usable but visibly less stable SHAP plots out of TabPFN — rerunning with a different background sample shifted the top-5 features more than I was comfortable with. If your deliverable includes a feature-importance figure that a clinical co-author needs to defend in a meeting, that instability is a real cost, not a nitpick. The other practical issue: TabPFN needs your training set at inference time, every time, because prediction is still an in-context forward pass over train+test. That means: model.pkl you hand to a hospital IT team the way you would an XGBoost booster file. XGBoost/LightGBM, by contrast, produce a single portable artifact, run in milliseconds, and have first-class support in basically every deployment stack ONNX, Treelite, PMML if you're in a particularly regulated shop . For a model going into an EHR-integrated pipeline, that maturity isn't a nice-to-have — it's often a hard requirement from the infra team. I used TabPFN for the initial exploratory phase and as a cross-check against overfitting — if tuned XGBoost dramatically outperforms TabPFN on a dataset this small, that's usually a signal the tuning found noise, not signal. For the actual deployed model, I went with a conservatively tuned LightGBM, because the interpretability tooling and deployment story were non-negotiable for this particular clinical partner. That won't be the right call for every project. If you're doing a quick feasibility study, a hackathon, or a paper where inference-time compute isn't constrained, TabPFN's speed and low-variance performance on sub-1000-row datasets is genuinely hard to beat. Have you shipped TabPFN in an actual production or clinical pipeline, or has it stayed in the notebook for you too? I'd like to know if anyone's solved the "carry the training set to inference" problem cleanly.