{"slug": "i-ran-my-text-to-sql-access-control-on-a-benchmark-i-didn-t-write", "title": "I ran my text-to-SQL access control on a benchmark I didn't write", "summary": "A developer behind schemagate, an open-source text-to-SQL access-control tool, ran the project against a public role-based-access-control benchmark from a SIGMOD'27 paper and reported that it beat the paper's best configuration on BIRD (90.1 vs 88.1 AC-F1) and tied on Spider (93.0), with no query reading forbidden data in either run. The developer said removing schemagate's scope check doubled bad answers on BIRD from 5.4% to 11.3%, and that a one-line notice cut bad answers from 16% to 5% on a 100-question test. The notice ships by default in schemagate 1.6.0.", "body_md": "Every number I had published for schemagate came from my own tests. This month a SIGMOD'27 paper released a public benchmark for exactly the problem schemagate exists for: [Benchmarking Text-to-SQL under Role-Based Access Control](https://arxiv.org/abs/2607.22115). So I ran it.\n\nEach question comes with a role and a list of the columns that role may read. If the answer needs a column the role can't see, the right reply is \"Sorry, I cannot answer.\" The authors score it with their own evaluator. I didn't change a line of it.\n\nTheir main finding is uncomfortable for tools like mine. If you just hide the restricted columns, models still answer. They patch something together from the columns that are left. On BIRD, their best \"hide the schema\" setup still answered 27% of the questions it should have refused.\n\nThree things happen for every question:\n\nSame model as the paper's best runs (Claude Sonnet 4.5), five trials, the authors' scoring:\n\n|  | AC-F1 | answered when it should refuse | SQL that read forbidden data | \n|---|---|---|---|\n| BIRD, schemagate | **90.1** | **5.4%** | **0** | \n| BIRD, paper's best | 88.1 | 7.4% | not reported | \n| Spider, schemagate | 93.0 | 5.1% | **0** | \n| Spider, paper's best | 93.0 | 4.6% | not reported | \n\nOn BIRD it beats the best configuration in the paper. On Spider it ties. In neither did any query read data the role wasn't allowed to see. The scope check makes that a guarantee rather than a measured rate.\n\nAnd the paper's best setup shows the model the *whole* schema plus the policy. schemagate gets these numbers without the model ever seeing what is restricted.\n\nTaking the scope check out doubled the bad answers on BIRD (5.4% to 11.3%). The one-line notice was tested on 100 questions: 16% bad answers without it, 5% with it.\n\nOne surprise: giving the model BIRD's column descriptions made things **worse**. Better notes on the visible columns helped it find workarounds for questions it should have refused.\n\nThe notice ships on by default in schemagate 1.6.0. To reproduce the runs: `benchmarks/rbac_text2sql.py`, plus the evaluator's reports in `benchmarks/results/rbac/`.\n\n`pip install schemagate` · [GitHub](https://github.com/ashishsinha1602/schemagate)", "url": "https://wpnews.pro/news/i-ran-my-text-to-sql-access-control-on-a-benchmark-i-didn-t-write", "canonical_source": "https://dev.to/ashish_sinha_5241c7673d93/i-ran-my-text-to-sql-access-control-on-a-benchmark-i-didnt-write-23n4", "published_at": "2026-10-17 02:46:35+00:00", "updated_at": "2026-10-17 02:58:07.755329+00:00", "lang": "en", "topics": ["large-language-models", "ai-tools", "developer-tools", "ai-agents"], "entities": ["schemagate", "SIGMOD'27", "BIRD", "Spider", "Claude Sonnet 4.5", "arXiv"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/i-ran-my-text-to-sql-access-control-on-a-benchmark-i-didn-t-write", "markdown": "https://wpnews.pro/news/i-ran-my-text-to-sql-access-control-on-a-benchmark-i-didn-t-write.md", "text": "https://wpnews.pro/news/i-ran-my-text-to-sql-access-control-on-a-benchmark-i-didn-t-write.txt", "jsonld": "https://wpnews.pro/news/i-ran-my-text-to-sql-access-control-on-a-benchmark-i-didn-t-write.jsonld"}}