TraceSQL: Traceable Answerability Estimation for Reference-Free Text-to-SQL Verification

Researchers propose TraceSQL, a lightweight verification model that uses 67 explicit diagnostic features to estimate whether generated SQL correctly answers a user question without ground-truth references. On BIRD development databases, TraceSQL achieves 66.47% F1 and 64.48% ROC-AUC, outperforming the GradeSQL-7B Outcome Reward Model baseline's 61.87% F1 and 58.26% ROC-AUC. The model's feature attribution enables traceable decisions, addressing the limited visibility of existing LLM-based judges and ORMs.

arXiv:2608.17795v1 Announce Type: new Abstract: Text-to-SQL systems are commonly evaluated using ground-truth SQL queries or reference execution results, but such supervision is unavailable at inference time in real-world deployments. This creates a critical verification problem: given only a user question, database context, and generated SQL, can a system estimate whether the generated query is likely to correctly answer the question? Recent approaches use LLMs as judge or specialized agents to inspect generated SQL, but their decisions can be difficult to trace. Outcome Reward Models ORMs address this by learning from execution-labeled candidate SQLs and assigning correctness scores to unseen queries, yet they still provide limited visibility into the signals behind each verification. To address this limitation, we propose TraceSQL, a lightweight and traceable verification model built on explicit diagnostic features. TraceSQL combines 67 features capturing question ambiguity, question requirements, question-schema-SQL consistency, SQL structure, and intent alignment. These signals remain available for examining which factors influence each prediction and for tracing decisions back to diagnostic evidence. On BIRD development databases, TraceSQL achieves 66.47% F1 and 64.48% ROC-AUC, compared with 61.87% F1 and 58.26% ROC-AUC for the GradeSQL-7B ORM baseline on the same generated-SQL evaluation. Feature attribution further shows that the model relies on both semantic grounding and deterministic SQL-structure signals. These results show that SQL verification can be performed with a lightweight learned model while retaining feature-level evidence for inspecting and diagnosing its predictions.