Why LLMs Still Struggle With Tabular Prediction

Most business prediction problems do not arrive as prose. They arrive as rows: account attributes, transactions, sensor readings, test results, and a target column. For this kind of data, gradient-boosted trees and other conventional methods remain hard to displace.

A new paper, Why Large Language Models Fail at Tabular Prediction, asks a much more useful question than “can an LLM classify a table?”: what, specifically, breaks as the task becomes more like ordinary tabular machine learning?

The answer from the authors’ controlled experiments is input dimensionality. Their result matters because it separates a real limitation from several explanations that sound plausible but did not hold up in their tests.

The experiment was about prediction, not table chat

The paper evaluates frontier LLMs in a pure inference setup: a model receives labeled examples and must predict labels for new rows in a single generation pass. There is no fine-tuning, retrieval pipeline, tool calling, or agent loop to compensate for the base model. This is deliberately narrow. It asks whether a general-purpose language model can act as a direct tabular learner.

Across 31 benchmark datasets, the authors compare nine methods and 252 configured classical models. That scope is important: a weak result on one CSV is easy to explain away as prompt design or a quirky dataset. A consistent trend across many tasks is harder to dismiss.

The headline is not simply that LLMs lose to established tabular baselines. It is that their accuracy declines as the number of input dimensions grows, while the classical baselines in the study stay stable or improve. The paper therefore treats dimensionality as the central failure mode rather than an incidental property of difficult datasets.

Four popular explanations did not survive testing

There are several standard reasons developers give for poor LLM performance on tables. The researchers turn these into falsifiable hypotheses.

“The classes overlap too much.” If the data are noisy or not cleanly separable, perhaps the model cannot infer a decision boundary. The experiments indicate this is not enough to explain the gap: performance problems remain even when the signal is made easier to separate.

“CSV destroys the table structure.” Turning columns and rows into a token sequence feels like an obvious mismatch. But changing the serialization format did not fix the issue. The model can access column information, yet it still fails on signals that should be simple.

“Numbers are tokenized badly.” Numeric strings split into tokens in awkward ways, so perhaps magnitude comparisons are the culprit. Lowering numeric precision and related interventions did not produce the expected recovery.

“There are too many test rows in one prompt.” A prompt with many predictions might spread the model’s compute too thin. Again, reducing that load was not the explanation.

None of these findings says formatting, numeric representation, or context length never matter in production. They do. The narrower point is that they do not account for the observed collapse in this direct-prediction setting. That distinction should change how we debug these systems: switching from CSV to JSON is unlikely to solve a high-dimensional learning problem.

Dimensionality is a different kind of obstacle

Tabular features are often heterogeneous and only weakly related: a customer’s region, account age, device type, transaction count, income band, and dozens of derived signals may each contribute a little. As features accumulate, useful local similarity becomes difficult to identify. This is one form of the familiar curse of dimensionality.

In two dimensions, the paper finds that LLM behavior resembles local methods such as low-k nearest neighbors or short-length-scale Gaussian processes. In other words, the model’s predictions can look like a neighborhood-based decision rule. The reported grid agreement reaches 91.6% for some such comparisons.

At higher dimensions, that resemblance disappears. The authors could not reproduce the LLM’s behavior with any of their classical models, even after adding tuned dimension-dependent noise. That is a valuable negative result. The model is not merely becoming an ordinary nearest-neighbor classifier with worse settings; its prediction behavior becomes hard to characterize and less accurate.

This also clarifies why fluent explanations can be misleading. An LLM can describe a dataset, identify likely feature meanings, and write a plausible analysis plan. Those abilities do not imply that the model has acquired the inductive bias needed to estimate a reliable decision function over many structured variables.

What should developers use instead?

For supervised prediction on a table, start with methods designed for tables. Gradient-boosted decision trees are still a strong default for mixed numeric and categorical features, moderate dataset sizes, missing values, and non-linear interactions. XGBoost’s original paper remains useful background for why boosted trees became such a practical baseline.

There is also active work on models that are explicitly built for tabular prediction rather than adapted from next-token prediction. TabPFN-2.5, for example, is a tabular foundation model evaluated against conventional approaches. Its existence reinforces the design lesson: “foundation model” is not synonymous with “general-purpose LLM.” Architecture, training distribution, and task framing all matter.

General-purpose LLMs still have useful roles around a tabular workflow:

• generate SQL, feature-engineering code, and experiment scaffolding;
• translate business questions into measurable targets;
• summarize data dictionaries and data-quality checks;
• assist with model documentation and error analysis;
• provide a natural-language interface to a model trained elsewhere.

Those are orchestration and interface tasks. They should not be confused with replacing the predictive model itself.

The practical takeaway

Do not treat a chat model’s ability to read a table as evidence that it can learn from the table. If the goal is accurate tabular prediction, establish a tree-based baseline first, measure it with a held-out evaluation protocol, and only then test an LLM-based approach against it.

The new paper is a preprint, so its conclusions deserve replication and extension. It also studies pure inference rather than fine-tuned or tool-augmented systems. But its experimental approach is worth adopting: isolate one explanation at a time, reject the convenient ones when the data rejects them, and focus on the structural constraint that remains.

For a broader map of this research area, see the survey on large language models for tabular data. The main message is restrained but useful: language models can make tabular workflows easier to operate, while the actual predictive core still needs methods with the right bias for the data.