{"slug": "are-llms-good-in-context-tabular-classifiers", "title": "Are LLMs good in-context tabular classifiers?", "summary": "TabBench-LLM, a benchmark introduced by the TabArena team, evaluates large language models as few-shot in-context tabular classifiers against Random Forest and TabPFN v2, using 19 synthetic tasks and 38 public TabArena-v0.1 datasets. The benchmark reports that LLMs are rarely used for tabular prediction due to high cost, unstable outputs, and memorization risks, with one headline model requiring ~9,500 requests versus baselines fitting in seconds on a laptop.", "body_md": "TabBench-LLM evaluates large language models as few-shot, **in-context tabular classifiers**, head-to-head with two baselines. It adapts [TabArena](https://huggingface.co/spaces/TabArena/leaderboard), a general benchmark for tabular ML, to the LLM setting.\n\nThe primary suite is 19 deterministic synthetic tasks whose feature and class names carry no real-world meaning, so the decision rule can only be inferred from the rows in the prompt, never recalled. A secondary suite of 38 public [TabArena-v0.1](https://huggingface.co/spaces/TabArena/leaderboard) datasets checks whether that carries over to real tables. Untuned Random Forest and TabPFN v2 fit the identical splits.\n\n**Elo** is a pairwise rating over shared datasets, calibrated so Random Forest = 1000. It\ncompares models only on targets they both attempted, but cannot correct selection bias when\nthe missing targets are systematically harder — so the headline is released only once every\nmodel has completed the same scheduled synthetic block.\n\n## Methodology\n\nPer dataset we draw a stratified training table of 10, 20, 50 or 100 rows and a fixed held-out test set, shared by every model. Each LLM request carries the labelled table and exactly one unlabelled test row, and asks for a single class token. Real datasets additionally compare visible class names with opaque tokens; synthetic targets are meaningless already, so only their opaque arm is run.\n\n## Discussion\n\nLLMs are rarely used for tabular prediction tasks, nor are they often evaluated on them. The two reinforce each other and neither is based on much published evidence. There are several things standing in the way.\n\nFirstly, consider what a caller actually gets back. **The output is a token, not a probability**.\nThreshold tuning, ranking metrics and cost-sensitive decisions require a calibrated score. Verbalised confidences are poorly calibrated, logprobs are not always exposed and answers must be parsed before scoring. Predictions are also unstable: shuffling demonstrations, renaming classes or resampling can change them. There are no feature importances or reproducible artefacts, deployment sends rows to a third-party API, and hosted models can change or disappear.\n\nThen the input side. **Numbers arrive as text**: a tokeniser splits 0.4931 into fragments and must reconstruct scale, ordering and distance, while a tree uses the value directly. CSV, JSON, Markdown, feature order and precision can all affect the score. A baseline sees the matrix; the LLM sees one arbitrary rendering, fixed here across models but not across the literature.\n\nThe **regime also does not scale**: in-context learning is limited by prompt length, inference needs one request per test row and one headline model takes ~9,500 requests, while Random Forest and TabPFN v2 fit the same splits in seconds on a laptop. The 100-example setting is therefore the practical comparison. At this cost, an LLM must be clearly better, not merely comparable, to earn a place in a pipeline.\n\nUnderneath it all lies a measurement problem. Standard tabular **datasets have been online for years**, including targets, so recall is indistinguishable from inference. The stronger results on TabArena datasets are therefore likely helped by memorisation. The synthetic suite and opaque class tokens reduce that risk, although TabPFN has a slightly unfair advantage there: the data is generated from priors similar to those used in its training.\n\nNone of this required anyone to decide that LLMs should be excluded. Weak expected results, high cost and an ambiguous protocol were enough on their own. Running the benchmark replaces that inference with a number: how far off they are and on which kinds of structure.\n\nReading the tables:\n\n**Elo**- pairwise rating, calibrated so Random Forest = 1000.** Score**- min–max normalized macro-F1 per dataset (best model = 1, worst = 0).** Improvability %**- mean relative gap to the best model per dataset; lower is better.\n\nRankings can use any recorded metric via the selector.\n\n## 🎁 Donate tokens\n\nThe code and the experiment design are ready; hosted frontier models still need API quota. Credits, higher rate limits, or GPU hours let us add models without shrinking the common evaluation block.\n\n### What your tokens buy\n\n19 datasets × 5 folds × 100 held-out rows, opaque labels, reasoning off.\n\nAll four few-shot sizes (10 / 20 / 50 / 100) on the synthetic suite.\n\nSynthetic and real suites at every sample size, plus the feature-cap runs.\n\n### Additional evaluation targets\n\nProvider access can add further frontier systems. Evaluation targets only; no endorsement or affiliation implied.\n\n[Offer tokens →](https://github.com/not-a-feature/TabBench-LLM/issues/new?title=Token%20donation&body=Which%20provider%2Fmodel%2C%20and%20roughly%20how%20much%20quota%3F%20Please%20do%20not%20paste%20API%20keys%20here.)Spare credits, a raised rate limit, an unused grant, or GPU hours all help. Open an issue to start;\n\n**never paste an API key in public**.\n\nThe numbers are regenerated by `tabbench-llm site`\n\nand uploaded as `files/tabbench-llm/leaderboard.json`\n\n. The charts are drawn in the browser with [Plotly](https://plotly.com/javascript/), loaded from [jsdelivr](https://www.jsdelivr.com); the figure export uses JSZip from the same CDN.", "url": "https://wpnews.pro/news/are-llms-good-in-context-tabular-classifiers", "canonical_source": "https://juleskreuer.eu/research/tabbench-llm", "published_at": "2026-08-17 07:45:17+00:00", "updated_at": "2026-08-17 08:10:55.559592+00:00", "lang": "en", "topics": ["large-language-models", "machine-learning", "ai-research"], "entities": ["TabBench-LLM", "TabArena", "Random Forest", "TabPFN v2"], "alternates": {"html": "https://wpnews.pro/news/are-llms-good-in-context-tabular-classifiers", "markdown": "https://wpnews.pro/news/are-llms-good-in-context-tabular-classifiers.md", "text": "https://wpnews.pro/news/are-llms-good-in-context-tabular-classifiers.txt", "jsonld": "https://wpnews.pro/news/are-llms-good-in-context-tabular-classifiers.jsonld"}}