BIRD Mini-Dev is 500 text-to-SQL questions over 11 databases, in SQLite, PostgreSQL and MySQL versions, scored by whether the query returns the right rows. BIRD keeps the leaderboard and checks the submissions.
schemagate, the Apache-2.0 library I maintain, is now first in all three columns:
| schemagate | Next entry | |
|---|---|---|
| SQLite | 81.0 | 75.6 |
| PostgreSQL | 72.6 | 69.0 |
| MySQL | 72.4 | 49.2 |
The setup is plain: schemagate picks the tables a question needs, the prompt carries only those plus BIRD's column notes, and Claude Opus 5.5 writes one query. No candidate voting and no agent loop. If the query errors, the model sees the error and tries again.
schemagate's main job is access control. It removes the tables the caller may not read before the model sees the schema. A benchmark doesn't test that part, but it shows the filtering doesn't cost accuracy.
What it isn't: the main BIRD board, which uses a hidden test set. Our first test result there was 72.11, and a rerun is pending.
pip install schemagate