[Submitted on 3 Aug 2026]
[View PDF](/pdf/2608.02412)
[HTML (experimental)](https://arxiv.org/html/2608.02412v1)
Abstract:Large language models (LLMs) have become the default tool for a remarkable range of tasks, yet they have had conspicuously little success at one of the most common machine learning workloads: predictive analytics over tabular data. This gap is the founding premise of the fast-growing field of tabular foundation models, but the question of why generic LLMs fail has remained open. We study a frontier LLM in its purest inference regime - a single generation pass over a prompt containing the full training and test data, with no tools, no agentic scaffolding, and no fine-tuning - and systematically evaluate five hypotheses for the failure: (a) an inability to handle noisy or non-linearly-separable data; (b) the linearised CSV format obscuring column structure; (c) the tokenisation of numeric values; (d) the number of test points classified per query; and (e) the dimensionality of the input. Controlled experiments falsify (a)-(d). Dimensionality, in contrast, is decisive: sweeping random linear projections of thirty-one benchmark datasets, the LLM is the only method among nine whose accuracy decreases as dimensionality grows, while every classical baseline stays flat or improves. A behavioural comparison against 252 configured classical models finds that in two dimensions the LLM predicts like a local, distance-based method (up to 91.6% grid agreement), but in higher dimensions no classical model - even when augmented with tuned, dimension-dependent noise - reproduces its predictions. We do not claim to have identified the internal mechanism; our results show, more modestly, that the LLM's capability dissolves with dimension in a way no noise-corrupted classical learner mimics - which explains why LLMs, so capable elsewhere, keep losing to fifty-year-old baselines on tables, while leaving the mechanism of the prediction as an open question.
References & Citations
...
Bibliographic Explorer
(What is the Explorer?) Connected Papers
(What is Connected Papers?) Litmaps
(What is Litmaps?) scite Smart Citations
(What are Smart Citations?)# Code, Data and Media Associated with this Article alphaXiv
(What is alphaXiv?) CatalyzeX Code Finder for Papers
(What is CatalyzeX?) DagsHub
(What is DagsHub?) Gotit.pub
(What is GotitPub?) Hugging Face
(What is Huggingface?) ScienceCast
(What is ScienceCast?)# Demos Influence Flower
(What are Influence Flowers?) CORE Recommender
(What is CORE?) IArxiv Recommender
(What is IArxiv?)# arXivLabs: experimental projects with community collaborators arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.