TabPFN vs Gradient Boosting: Tiny Medical Datasets Showdown

If you've worked on a clinical ML project, you know the dataset size everyone pretends doesn't exist: 80 patients. 150 patients. Maybe 300 if the hospital is generous and the IRB is fast. Meanwhile every tabular ML tutorial on the internet assumes you have 100k+ rows and can afford to tune a gradient boosted tree for three days on a GPU cluster. I recently had to pick a model for a biomarker classification task — 142 patients, 31 features (labs, vitals, a few engineered ratios), binary outcome, heavily class-imbalanced, and noisy because half the labs were drawn under different protocols. The usual suspects were XGBoost and LightGBM. But TabPFN kept showing up in my feed as the thing you're supposed to try on small tabular data now. So I actually ran the comparison instead of trusting a benchmark table.

This post is what I found, with the caveats that actually matter for a healthcare dataset instead of a Kaggle leaderboard.

TabPFN (Tabular Prior-Data Fitted Network) isn't trained on your data in the traditional sense. It's a transformer pretrained once on millions of synthetic datasets, and at inference time it does in-context learning: you feed it your train set and test set together, and it does a single forward pass to produce predictions. No gradient descent on your data, no hyperparameter search, no early stopping.

The catch (as of the versions most people are using): it was designed for datasets up to a few thousand rows and a few hundred features. That's not a limitation for us — that's literally our use case.

bash

pip install tabpfn xgboost lightgbm scikit-learn shap

Dataset: 142 rows, 31 numeric features, binary label, ~22% positive class. 5-fold stratified CV, repeated 10 times to get stable estimates given how small the test folds are (roughly 28 rows per fold — variance is brutal at this scale). python

import numpy as np

from sklearn.model_selection import RepeatedStratifiedKFold

from sklearn.metrics import roc_auc_score, average_precision_score

from tabpfn import TabPFNClassifier

import xgboost as xgb

import lightgbm as lgb

X, y = load_clinical_data() # (142, 31), binary y

cv = RepeatedStratifiedKFold(n_splits=5, n_repeats=10, random_state=42)

def evaluate(model_fn):

    aucs, aps = [], []

    for train_idx, test_idx in cv.split(X, y):

        model = model_fn()

        model.fit(X[train_idx], y[train_idx])

        preds = model.predict_proba(X[test_idx])[:, 1]

        aucs.append(roc_auc_score(y[test_idx], preds))

        aps.append(average_precision_score(y[test_idx], preds))

    return np.mean(aucs), np.std(aucs), np.mean(aps)

TabPFN has no real "training" step — it's a forward pass. On this dataset size, fit+predict for a single fold took under 0.3 seconds on CPU. XGBoost and LightGBM, once you include a proper hyperparameter search (which you need, because default params on 140 rows will overfit or underfit depending on the day), took several minutes per fold with a reasonable Optuna search.

python

def tabpfn_model():

    return TabPFNClassifier(device="cpu", N_ensemble_configurations=32)

def xgb_model():

    return xgb.XGBClassifier(

        n_estimators=150, max_depth=3, learning_rate=0.05,

        subsample=0.7, colsample_bytree=0.7,

        reg_lambda=2.0, eval_metric="logloss"

    )

tabpfn_auc = evaluate(tabpfn_model)

xgb_auc = evaluate(xgb_model)

The time savings aren't just convenience. On small clinical datasets, your "hyperparameter tuning" is really a search over an enormous space of near-identical local optima, and the result is unstable across random seeds. TabPFN sidesteps that instability entirely because there's nothing to tune — which is either a feature or a limitation depending on how much you value control.

Across 50 CV splits:

Two things stood out. First, TabPFN's variance across folds was noticeably lower — it's less sensitive to which 28 patients ended up in the test fold, which matters a lot when your confidence interval is the whole story. Second, the gap between tuned and untuned XGBoost was huge, which is the real danger of gradient boosting on tiny data: if you don't have the discipline (or the held-out budget) to tune properly, you'll ship a model that's quietly overfit to noise in the lab values.

This isn't universal — on a second dataset I tested (180 rows, mostly categorical features with a lot of missingness), tuned LightGBM edged out TabPFN by about 2 AUC points, likely because TabPFN's synthetic pretraining prior doesn't model heavy categorical missingness patterns as well as it models continuous clinical variables.

This is where the decision usually actually gets made, not on the leaderboard. Clinicians and IRBs want to know why the model flagged a patient. With gradient boosting, you get mature tooling:

python

import shap

model = xgb_model().fit(X, y)

explainer = shap.TreeExplainer(model)

shap_values = explainer.shap_values(X)

shap.summary_plot(shap_values, X, feature_names=feature_names)

TreeExplainer is exact, fast, and well understood by reviewers at this point — it's practically a checkbox item in clinical ML papers now. TabPFN interpretability is much rougher: you're stuck with KernelSHAP or permutation importance, both of which are slow and noisy precisely because the model itself has no stable internal structure to exploit, and both become even noisier on 142 rows. I got usable but visibly less stable SHAP plots out of TabPFN — rerunning with a different background sample shifted the top-5 features more than I was comfortable with.

If your deliverable includes a feature-importance figure that a clinical co-author needs to defend in a meeting, that instability is a real cost, not a nitpick. The other practical issue: TabPFN needs your training set at inference time, every time, because prediction is still an in-context forward pass over train+test. That means:

model.pkl you hand to a hospital IT team the way you would an XGBoost booster file. XGBoost/LightGBM, by contrast, produce a single portable artifact, run in milliseconds, and have first-class support in basically every deployment stack (ONNX, Treelite, PMML if you're in a particularly regulated shop). For a model going into an EHR-integrated pipeline, that maturity isn't a nice-to-have — it's often a hard requirement from the infra team.

I used TabPFN for the initial exploratory phase and as a cross-check against overfitting — if tuned XGBoost dramatically outperforms TabPFN on a dataset this small, that's usually a signal the tuning found noise, not signal. For the actual deployed model, I went with a conservatively tuned LightGBM, because the interpretability tooling and deployment story were non-negotiable for this particular clinical partner.

That won't be the right call for every project. If you're doing a quick feasibility study, a hackathon, or a paper where inference-time compute isn't constrained, TabPFN's speed and low-variance performance on sub-1000-row datasets is genuinely hard to beat.

Have you shipped TabPFN in an actual production or clinical pipeline, or has it stayed in the notebook for you too? I'd like to know if anyone's solved the "carry the training set to inference" problem cleanly.