What Predicts Correctness in Text-to-SQL? A Selective-Prediction Study

A study on text-to-SQL systems found that verification-based signals, such as LLM judges, outperform black-box signals like self-consistency in predicting query correctness, with a two-provider ensemble achieving 0.82 AUROC. Fine-tuned verifiers perform well in-distribution but fail to generalize across unseen schemas, indicating that correctness uncertainty relies on reasoning-based signals from large frozen models.

arXiv:2607.06799v1 Announce Type: cross Abstract: Evaluating uncertainty in AI-generated SQL queries requires estimating whether a query is correct, where correct means it executes to the same result as a human-written reference. We study which signals predict correctness on hard multi-table text-to-SQL, using AUROC to measure how well each ranks correct queries above incorrect ones. On BIRD and Spider, black-box signals such as string, structural, and execution self-consistency, a schema-relevance score, and query executability all fall between about 0.61 and 0.68 AUROC, with string self-consistency strongest at 0.675; white-box log-probability is similar 0.67 . The signals that move past this ceiling are verification-based: an LLM judge scores from 0.72 GPT-4o-mini to 0.78 Claude . Judges from different providers make different errors, so a two-provider ensemble reaches 0.82 AUROC with a well-calibrated probability expected calibration error 0.03 and supports useful abstention frontiers for example, answering 27% of questions at 24% selective risk where self-consistency offers no valid low-risk subset. The pattern holds across two benchmarks, two generators, and two judge providers. We also ask whether a verifier can be trained. Fine-tuned verifiers, both encoder and generative, reach about 0.77 to 0.79 AUROC in-distribution but fall to about 0.66 on unseen schemas; scaling to 7B, adding schema diversity, distilling a strong judge's rationales, and cross-benchmark training all fail to close that gap. Cross-schema transfer appears to track model scale and reasoning rather than fine-tuning. In practice, correctness uncertainty for text-to-SQL lives in reasoning-based signals: a fine-tuned verifier is a good in-domain tool, but a verifier that generalizes across schemas currently means a large frozen reasoning model.