Every number I had published for schemagate came from my own tests. This month a SIGMOD'27 paper released a public benchmark for exactly the problem schemagate exists for: Benchmarking Text-to-SQL under Role-Based Access Control. So I ran it.
Each question comes with a role and a list of the columns that role may read. If the answer needs a column the role can't see, the right reply is "Sorry, I cannot answer." The authors score it with their own evaluator. I didn't change a line of it.
Their main finding is uncomfortable for tools like mine. If you just hide the restricted columns, models still answer. They patch something together from the columns that are left. On BIRD, their best "hide the schema" setup still answered 27% of the questions it should have refused.
Three things happen for every question:
Same model as the paper's best runs (Claude Sonnet 4.5), five trials, the authors' scoring:
| AC-F1 | answered when it should refuse | SQL that read forbidden data | |
|---|---|---|---|
| BIRD, schemagate | 90.1 | 5.4% | 0 |
| BIRD, paper's best | 88.1 | 7.4% | not reported |
| Spider, schemagate | 93.0 | 5.1% | 0 |
| Spider, paper's best | 93.0 | 4.6% | not reported |
On BIRD it beats the best configuration in the paper. On Spider it ties. In neither did any query read data the role wasn't allowed to see. The scope check makes that a guarantee rather than a measured rate.
And the paper's best setup shows the model the whole schema plus the policy. schemagate gets these numbers without the model ever seeing what is restricted.
Taking the scope check out doubled the bad answers on BIRD (5.4% to 11.3%). The one-line notice was tested on 100 questions: 16% bad answers without it, 5% with it.
One surprise: giving the model BIRD's column descriptions made things worse. Better notes on the visible columns helped it find workarounds for questions it should have refused.
The notice ships on by default in schemagate 1.6.0. To reproduce the runs: benchmarks/rbac_text2sql.py, plus the evaluator's reports in benchmarks/results/rbac/.
pip install schemagate · GitHub