BlockTabBench: Evaluating Tabular Models Across Block's Machine Learning Ecosystem

Block released BlockTabBench2026, an evaluation benchmark of over one hundred datasets from its production machine learning ecosystem, and found that tabular foundation models using in-context learning excel with limited training data but lose their edge as dataset size grows, while standard deep learning approaches like TabM outperform tree-based XGBoost and foundation models on larger datasets. The benchmark, which spans datasets from a few thousand to several million rows and uses out-of-time splits to reflect real-world drift, aims to assess tabular model performance across Block's financial domain, including triggers like Cash App transactions and concerns such as scams and account takeovers.

Introduction Tabular machine learning is entering a transition similar to what natural language processing experienced during the emergence of large language models, with foundation models becoming an increasingly important part of the research landscape 1, 2, 3, 4 . As a result, standardized evaluations such as TabArena 5 and BeyondArena 6 have become valuable tools for understanding the relative efficacy of different modeling approaches and for driving reproducible progress across the field. At Block, we have a large and diverse collection of datasets used to train models that identify fraudulent or high-risk activity across our products, accumulated across many years of operating production machine learning systems. These datasets are based on financial activity with very different characteristics than existing public datasets such as substantially higher row and column counts, varying amounts of imbalance, and temporal drift. While recent efforts such as BeyondArena represent a shift toward these more realistic properties, public evaluations typically aim to measure tabular machine learning performance in a domain general sense and thus do not fully capture the scale or characteristics of Block’s specific financial domain. We developed BlockTabBench to assess tabular machine learning performance across Block’s machine learning problems. We annotate individual datasets with metadata such as the business event of interest e.g., every Square payment, every Cash App deposit , the problem domain e.g., scams, account takeovers , and attributes from the dataset e.g., row and column count, imbalance, temporal drift . We benchmarked various models on each dataset to understand the strengths and weaknesses of each approach. Overall, we found: - Tabular foundation models that rely on in-context learning are highly performant when training data is limited but lose their advantage as the dataset size grows - Standard deep learning approaches for tabular data scale with millions of rows, outperforming both tree-based methods and tabular foundation models on larger datasets - The advantage of tree models is concentrated on problems where every approach performs well, and narrows as the achievable ceiling lowers Block’s Machine Learning Ecosystem At Block, we are often interested in scanning an event stream like Cash App transactions for a problem like a scam or account takeover. We call the former the trigger and the latter the concern. A single trigger typically fans out to many concerns, with the same event stream scored by several models trained with different labels. We sampled over one hundred datasets across all of our triggers and concerns and built BlockTabBench2026, an evaluation that is representative of Block’s ecosystem of production machine learning models as of 2026. BlockTabBench2026 contains datasets spanning approximately a few thousand rows to several million rows and column counts from a few hundred into the mid-thousands. Concerns vary in how often they occur, which gives us tasks ranging from rare-event detection to nearly balanced classification as well as in how quickly they can be confirmed, which determines how much labeled data is available. Every dataset is split out-of-time, matching how the corresponding production models were developed on historical data and deployed on unseen future data . Public evaluations have traditionally assumed random splits, whereas our results reflect a more realistic setting where there may be significant drift between the train and test distributions. Evaluating Tabular Models We benchmarked various modeling approaches on BlockTabBench2026 to understand how they performed overall as well as how dataset properties contributed to each model’s performance. We were interested in comparing tree models XGBoost 7 against modern deep learning approaches for tabular data TabM 8 and tabular foundation models that rely on in-context learning for classification TabFM 4 . We used the area under the Precision-Recall curve PR-AUC to determine a winner for each dataset. We chose PR-AUC because production models are deployed at a decision threshold that varies by concern, trading off precision and recall according to the cost of acting on a false positive versus missing a true positive. PR-AUC evaluates a model across all such thresholds rather than committing to a single operating point. For each dataset, we evaluated multiple random seeds with XGBoost and TabM undergoing hyperparameter tuning and TabFM using 50K sampled rows but otherwise fixed hyperparameters 32-member ensemble, top 500 features via XGBoost importance . Benchmarking was performed on de-identified data with engineered features in Block's internal infrastructure. Counting the datasets each model wins, we find XGBoost to perform the best overall 41% win rate , closely followed by TabM 37% win rate , and finally TabFM 22% win rate . Breaking down the wins by dataset properties like train rows and problem difficulty, measured as the best PR-AUC any model reaches, paints a more nuanced picture where different models excel in different regimes. For each model, we fit a logistic regression predicting its wins from these properties. TabFM’s in-context learning excels under data scarcity, similar to early findings in the large language modeling literature around the effectiveness of few-shot prompting 9 . This makes TabFM useful when labeled data is limited such as with newly emerging fraud patterns or other situations that require classification from few targeted examples. As the number of training rows increases, TabM’s standard gradient-based training overtakes other modeling approaches, consistent with deep learning scaling laws 10, 11 . In contrast to TabFM’s utility under label constraints, TabM is useful for problems with high label volume such as those where labels are continuously generated by a process like a manual review workforce investigating flagged transactions. We also find that XGBoost's wins are concentrated on the problems where every approach performs, but its advantage narrows as the ceiling falls. This finding suggests that XGBoost is a reasonable default if it performs near ceiling, but for problems far from ceiling, exploring alternatives could pay off. Summary Constructing a benchmark that captures the nature of Block’s machine learning ecosystem allows us to draw conclusions about the efficacy of different tabular models similar to how evaluations are used to benchmark the advances in large language models. We find that tabular foundation models – which have recently exceeded XGBoost and more standard tabular deep learning approaches on public benchmarks like TabArena – only maintain this lead in the limited data regime. While standard tabular deep learning approaches excelled when large amounts of training data were available, it is possible that tabular foundation models can maintain their advantage via fine-tuning 12 or training from scratch on large amounts of synthetic data that better represents the activity seen across Block’s services. References 1 Hollmann, Noah, et al. "Accurate predictions on small data with a tabular foundation model." Nature 637.8045 2025 : 319-326. 2 Grinsztajn, Léo, et al. "TabPFN-3: Technical report." arXiv preprint arXiv .13986 2026 . 3 Qu, Jingang, et al. "TabICLv2: A better, faster, scalable, and open tabular foundation model." Proceedings of the 43rd International Conference on Machine Learning 2026 . 4 Kong, Weihao, and Abhimanyu Das. "Introducing TabFM: a zero-shot foundation model for tabular data." https://research.google/blog/introducing-tabfm-a-zero-shot-foundation-model-for-tabular-data https://research.google/blog/introducing-tabfm-a-zero-shot-foundation-model-for-tabular-data 2026 . 5 Erickson, Nick, et al. "TabArena: A living benchmark for machine learning on tabular data." Advances in Neural Information Processing Systems 38 2026 . 6 Purucker, Lennart, et al. "Beyond IID: How general are tabular foundation models, really?" arXiv preprint arXiv .30410 2026 . 7 Chen, Tianqi, and Carlos Guestrin. "XGBoost: A scalable tree boosting system." Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining 2016 . 8 Gorishniy, Yury, et al. "TabM: Advancing tabular deep learning with parameter-efficient ensembling." Proceedings of the 13th International Conference on Learning Representations 2025 . 9 Brown, Tom B., et al. "Language models are few-shot learners." Advances in Neural Information Processing Systems 33 2020 . 10 Hestness, Joel, et al. "Deep learning scaling is predictable, empirically." arXiv preprint arXiv .00409 2017 . 11 Kaplan, Jared, et al. "Scaling laws for neural language models." arXiv preprint arXiv .08361 2020 . 12 Rubachev, Ivan, et al. "On finetuning tabular foundation models." arXiv preprint arXiv .08982 2025 .