How to build a tiny 1.5B text-to-SQL model that beats a 7B

A developer built a 1.5B parameter text-to-SQL model that outperforms a 7B model on the Spider benchmark, achieving 71.5% accuracy through self-consistency voting. The project, starting with Qwen2.5-0.5B and progressing to Qwen2.5-Coder-1.5B, highlights the importance of evaluation harnesses and the trade-off between model size and compute.

I wanted to build something with an LLM using my own hands. Not wire an API into a wrapper, but take a base model, train it, measure it, break it, and serve it over HTTP. Why? Because I wanted to get my hands dirty working with an LLM, making and breaking things. Text to SQL fits exactly. You ask a question in plain English, the system writes SQL, and you can run the query to find out whether it was right. Correctness is whether the rows match the reference query's rows against the real database, which is a fact rather than a judgement. It also has a mature benchmark in Spider https://huggingface.co/datasets/xlangai/spider , 10,000 human written questions over 200 real SQLite databases, split so the test databases never appear in training. So I built it. Qwen2.5-0.5B as the base, LoRA adapters, one g5.xlarge with an NVIDIA A10G, about a dollar an hour. Continued pretraining on SQL text, then supervised fine-tuning on question and query pairs, then reinforcement learning with GRPO on top. The dashboards looked great. Reward climbing to 1.0, loss dropping cleanly through every stage. It scored 6.4%. Comically bad levels of accuracy. I didn't find that out for a while, because there was no held out evaluation. Every number I had came from data the model had trained on. The untrained base model scored 17.4%, so three stages of training had made things worse than no training at all. The fix wasn't a better model. It was building the thing that could tell me I was wrong, an evaluation harness that runs both queries against the real database and compares the rows that come back. Then I rebuilt. 6.4% to 44.6% on real schemas, then 49.7% with a proper RL reward, all still at 0.5B. Only then did I switch to Qwen2.5-Coder-1.5B, which landed at 68.1%. Sampling eight answers per question and returning whichever result most of them agreed on took it to 71.5%, against 71.2% for Qwen2.5-Coder-7B-Instruct. The 7B is still better at one attempt, 71.2% to 68.1%. The 1.5B gets there by answering eight times and voting, trading compute for the gap. That trade is the point though, and the point of this entire blog, and it's the one a lot of people are making now. Take a small model, aim it at one task, and build enough system around it that it beats something far larger at that one thing. The first version was three training stages run back to back on Qwen2.5-0.5B , a half billion param open model from Alibaba, all of them using LoRA Low-Rank Adaptation https://arxiv.org/abs/2106.09685 , which freezes the model's real weights and trains a tiny fraction of new, lightweight parameters. Continued PreTraining : keep training the base model on raw SQL text so it gets used to the shape of the language. Supervised FineTuning : show it thousands of question and correct-query pairs and have it imitate them. Reinforcement Learning with GRPO : Group Relative Policy Optimization. The model writes several answers to the same question, each is scored, and it gets pushed toward whichever ones scored above that group's average. It learns from its own attempts rather than from copied answers. Here is what those runs reported: | Stage | Steps | Loss | Data actually seen | |---|---|---|---| | Continued PreTraining | 500 | 2.46 to 0.81 | ~2.7 passes over the corpus | | Supervised FineTuning | 500 | 3.26 to 0.51 | 0.10 of one pass, about 8,000 of 78,577 examples | | Reinforcement Learning | 300 | reward 0.8 to 1.0 | 5,000 rows of the same training set | You have to understand that this was a learning project and my approach was flawed, so a few things went wrong. Every curve went the right way, and none of them meant anything, for three reasons. The data taught the wrong task : sql-create-context hands the model a schema already trimmed to exactly the columns the answer needs, so it never has to work out which table matters. Working that out is the entire job. It also saw only a tenth of that data, because the run was configured by step count rather than by passes over it. The reward could not teach anything : a formatting reward worth 1.0 fired for essentially every answer, and an exact string match reward worth 2.0 fired for almost none, so a group of four answers usually scored 1.0, 1.0, 1.0, 1.0 . GRPO scores each answer against its group's average, so when they all score the same the update is zero. The metric that tracks this hit 1.00 by step 160, meaning a large share of those 300 steps did nothing at all. And exact string matching is the wrong test anyway. It scores age 56 as zero against age 56 . There was no held out evaluation : the RL stage ran on a slice of the same data the model had been finetuned on, so every number I had measured how well it remembered its own training set. That last one is the real defect and the other two follow from it. Bad data and a broken reward are ordinary mistakes. What made them expensive is that nothing in the system could report them. The dashboards were green throughout. I stayed on the half billion param model for all of this. Not because it was good, but because a full fine tune plus a full evaluation there is about two GPU hours. When you are wrong five times in a row, that matters more than the ceiling does. The old training set handed the model schemas already trimmed to the columns the answer needed. The rebuild uses Spider with the real, complete database schemas, so the model has to find the right table among all the wrong ones. Three other things changed at the same time. Loss is computed on the answer only : A training example is the schema, then the question, then the correct query. With full schemas the schema part is roughly 75% of the tokens. If you score the model on reproducing the whole example, most of the training signal is teaching it to generate database schemas, which nobody asked for. Masking the prompt means every bit of the signal lands on the query. Two full passes over the data instead of a tenth of one : An epoch is one complete pass through the training set. The old run had covered 0.10 of one. A learning rate suited to what was actually being trained : The learning rate controls how big each update is. The old value was appropriate for nudging a model that already knew the task. LoRA starts its extra weights from scratch, so it needs a larger one. That is the single biggest change in the project. 6.4% to 44.6%. One thing I did not expect. Validation loss picked the wrong checkpoint. Epoch 2 looked slightly worse than epoch 1 on both validation loss and token accuracy, and it was 3.9 points better on real execution accuracy. Loss is a proxy for a proxy. Run the metric you actually care about, on held out data, at every checkpoint. In hindsight the old setup is a bit embarrassing, but every one of these fixes came from the failures, not from knowing better. Something that's true for most things that we do, not just machine learning. Now that the harness existed, the reward could use it. Instead of a black and white, all or nothing test, the reward is a ladder with partial credit. | Outcome | Reward | What it buys | |---|---|---| | No SQL, or it will not parse | 0.0 | nothing | | It parses as SQL | 0.2 | syntax is a solved sub-problem | | The database accepted and ran it | 0.5 | the tables and columns actually exist | The rows match the reference | 2.0 | correct | The point of the middle rungs is the zero gradient problem from before. A group of four answers where none is fully correct now scores something like 0.2, 0.5, 0.5, 0.2 instead of 1.0, 1.0, 1.0, 1.0 . There is real disagreement inside the group, so there is a real update. The 0.5 rung is aimed at one specific failure. Going into this, 88% of the model's remaining mistakes were invented column and table names. That rung pays for nothing except the columns existing. 44.6% to 49.7%. The mechanism moved in exactly the way the ladder was designed to move it. unknown column 647 → 458 down 29% unknown table 66 → 7 down 89% Two models answering the same 2,147 questions share most of their answers, and the shared ones tell you nothing. Only the questions where they disagree carry information. McNemar's test looks at exactly those. If the two models were equally good, the disagreements should split roughly evenly between them. They did not. RL fixed 251 questions and broke 142. The test says a split that lopsided happens by chance with probability below 0.001. The fixed and broken counts matter as much as the test. The headline is plus 5.1 points, but that is a net . It is not a clean sweep, and reporting it as one would hide 142 regressions. None of these throw an error. They just quietly change what the reward means, and nothing tells you it happened. Reward scaling flattens the ladder : the library divides each group's scores by how spread out that group was. The order of the rungs survives, the spacing does not, so a group whose best answer was genuinely correct ends up pushing exactly as hard as one whose best answer merely parsed. The ladder becomes a ranking. Switching that off keeps the 7.5 times gap between correct and parsed intact. The reference model is not the one you think : RL usually penalizes the model for drifting too far from a reference version of itself, measured as KL divergence. Under LoRA that reference is your adapter switched off, which is the raw base model, not your finetuned checkpoint. So the penalty was measuring how far supervised finetuning had already moved, not what RL was doing. I set it to zero rather than pull the model back toward something I had deliberately trained it away from. Row caps must not be able to change a verdict : the reward stops reading rows once it has one more than the correct answer has. Anything longer could not have matched anyway, so the cap can never turn a right answer into a wrong one. It only stops a hallucinated cross join from materializing a million rows inside the training loop. Reward hacking is when the model finds a way to score well without doing the task. It is usually discovered from a reward curve that climbs while the benchmark stays flat, which is to say after the GPU hours are gone. So before spending the GPU hours, I attacked my own reward. I wrote eleven policies that never look at the question at all, and measured how much reward each one could collect. SELECT 1 always parses and never matches. SELECT FROM <first table always runs. WHERE 1 = 0 goes after the empty result loophole. Degenerate cross joins test the row cap and the parser guards. The best of them collected 26.8% of what a real answer gets, and got 1.8% of questions outright, which is the empty answer slice and exactly where I expected it to score. So the reward is not easy to cheat, and that is a number now rather than an assumption. This probe is also what caught the reward scaling default above. The original training data told the model to think before answering. The problem was that the "thinking" was one hardcoded sentence repeated across all 5,378 examples . As scratch paper it is blank. It carries no information about the specific question, so all the model learns is to recite a preamble before answering exactly as it would have anyway. The proper version of this is rejection sampling , sometimes called STaR Self-Taught Reasoner . Take a larger model that can actually reason, have it solve each of your training questions eight times while showing its work, then run every attempt against the database and keep only the ones whose SQL was correct . That filter is the entire technique. A large model is wrong confidently and fluently, and training a small model on fluent wrong reasoning is worse than not training it at all. The harness from earlier is what makes the filter possible, which is the third job that one component ended up doing. It worked as a data pipeline. 4,823 verified traces, covering 90.8% of the training questions, with a median of 114 words of reasoning that named real tables and real columns. And it made the model worse. 44.6% down to 41.8%. | Training data | Tokens the model writes | Of which are SQL | Share of the signal landing on the SQL | |---|---|---|---| | Canned sentence | 62.6 | 33.6 | 53.7% | | Real traces | 195.1 | 31.1 | 15.9% | Training only teaches the model on the tokens it writes. With the canned sentence, it wrote about 63 tokens per answer and 34 of them were the SQL, so more than half its practice was on the query itself. With real traces it wrote about 195 tokens and the SQL was still only 31 of them. The query had not got any longer. It was just buried under 114 words of reasoning. So the model went from spending half its practice on the query to spending a sixth of it. Same number of passes over the data, same learning rate, 3.4 times less practice on the only part that gets marked. Running it for twice as many passes gave that practice back, and it came out at 47.1% , above where it started. Running reinforcement learning on top of the trace trained model was a wash. Plus 0.1 points, 179 questions fixed and 177 broken. Statistically that is nothing. The reason is that both interventions attack the same failure. Traces help the model name real tables and columns because the reasoning it copied named real tables and columns. The execution reward helps with exactly the same thing, and it is better at it. So the traces had already done a partial version of RL's job and left it nothing to work on. The best 0.5B model in the project stayed the straightforward one, supervised finetuning plus RL, at 49.7% . It cost real time and it did not produce the headline number, so the temptation is to leave it out. Two reasons not to. First, a negative result with a measured cause is more useful than a positive one without. "Reasoning traces did not help" is folklore. "Reasoning traces diluted the training signal on the answer by 3.4 times, and doubling the passes over the data fixed it" is something you can act on. Second, it is a clean example of the thing this whole project is about. The pipeline ran, the data was verified, the traces were genuinely good, and the model got worse. Nothing about that was visible from the training curves. The only reason I know any of it is that there was a held out number to check. At 49.7% the 0.5B model had run out of road with this training setup. The failure breakdown said the remaining errors were not the kind a better reward fixes, so I changed the base model to Qwen2.5-Coder-1.5B . Three times the parameters, and pretrained on code rather than general text, which should help with a task whose output is code. Same pipeline, nothing else changed. 67.9% , and 68.1% after RL. That is a large jump and it invites the obvious objection. Maybe none of the previous work mattered and I just needed a bigger model. That is an empirical question, so I answered it with an experiment instead of an opinion. Run the old pipeline and the new one on both base models. Four runs, every combination. | Pipeline | 0.5B | 1.5B | What scale bought | |---|---|---|---| | Old | 4.6% | 46.9% | +42.4 | New | 44.6% | 67.9% | +23.2 | What technique bought | +40.1 | +20.9 | Both effects are large and both hold up statistically. Technique at 1.5B fixed 559 questions and broke 110. Scale under the new pipeline fixed 577 and broke 78. Both matter, and they eat into each other : Technique is worth 20.9 points even on a bigger, code pretrained base, so the work is not just compensating for a weak model. But that is half what it was worth at 0.5B, because the bigger model had already solved some of what the technique was fixing. The corners of the table show the same thing. Measured on its own, technique is worth 40.1 points and the bigger model is worth 42.4, so you would expect 82 from doing both. Going from the worst combination to the best actually gets you 63.3, from 4.6% to 67.9%. The missing 19 points are questions both changes would have fixed on their own, and you only get paid for those once. The old pipeline at 0.5B also carried a tokenizer bug that stopped generation from terminating properly, while the 1.5B version of it used a working stop token. So the +42.4 in the top row is partly "a bug was also fixed". The two numbers I actually rely on are the ones underneath it, where everything else is held fixed. And the scale column is not purely scale. The new base is both larger and code pretrained, so those two things are tangled together in the +23.2 and I cannot separate them without a fourth base model I did not train. The technique column is clean, since that is the same base with two pipelines. The scale column should be read as "bigger and code pretrained", not "bigger". Worth saying that the old pipeline was set up to lose as gracefully as possible. Its destructive pretraining stage was left out, the library that corrupted the vocabulary was left out, the working stop token was used. It is the strongest fair version of the thing it loses to. Same reward, same configuration, run on the new base. RL on the 0.5B base: +5.1 points 251 fixed, 142 broken p < 0.001 RL on the 1.5B base: +0.2 points 122 fixed, 117 broken not significant "Not significant" here means the fixed and broken counts are close enough that the difference is indistinguishable from chance. 122 against 117 is a coin flip. The strange part is that the mechanism still worked perfectly. Hallucinated columns fell 23%. The share of queries the database would actually run rose from 88.2% to 90.6%. Syntax errors dropped from 5 to 2. Everything improved except the score. The reason is that the reward pays for executability, and executability had stopped being the bottleneck. At 0.5B, invented columns dominated the failures, so fixing them moved the number. At 1.5B the model already runs 90% of its queries, and what remains are queries that execute perfectly and answer the wrong question. This reward cannot see that. A query that runs and returns wrong rows scores 0.5, exactly the same as one that runs and returns wrong rows for a completely different reason. Two diagnostics from that run are worth carrying to any RL project: 56% of training groups produced no gradient at all : That is the original v0 failure arriving for the opposite reason. Back then the rollouts were uniformly wrong, so nothing separated them. Now they are uniformly right. Reinforcement learning gets harder as your policy gets better , because it runs on disagreement inside a group, and a good model disagrees with itself less. Training reward rose 24% while held out accuracy moved 0.2 points : With entropy falling steadily the whole time. Entropy here is how spread out the model's choices are, and it falling means the model is getting more confident and less varied. That is precisely the shape of the dashboards that started this whole project. The only difference is that this time there was a held out number sitting next to it saying the gain was not real. Everything so far changes the model's weights. At 68.1% it is worth asking what the remaining 32% actually looks like, because the answer decides what you can do about it without training anything. 1,462 correct 68.1% 483 runs perfectly, answers the wrong question 22.5% no error exists 202 the database rejects it outright 9.4% an error message exists Two completely different populations. The second one has a signal attached, because when SQLite refuses a query it tells you why, in words, like no such column: customer name . The first one has nothing. The query runs, rows come back, and everything looks fine. The obvious move for the 202 is to show the model its own error and let it try again, up to three rounds. Two things make that a measurement rather than a demo. The loop never sees the correct answer . Its only stopping signal is whether the database accepted the query, which is all that exists at inference time in the real world. And the rounds are batched rather than per question . Round one generates all 2,147, and only the roughly 200 rejections go into round two. Looping question by question is about 50 times slower for an identical answer. It bought 1.1 points. I had predicted 4 and written that down beforehand. The reason is the useful part. 202 rejected on the first attempt 63 became something the database would run 31% of rejections 23 were actually correct 37% of those repairs 40 moved from "database refused it" to "runs fine, wrong rows" There are two multiplications there, not one. An error message tells you that you are wrong. It never tells you what is right. So the model fixes the complaint rather than the answer, and two thirds of its repairs migrate from the visible failure bucket into the invisible one. At 0.5B it does essentially nothing. 500 rejections, 12 repaired. A weak model told "no such column: X" still has no idea which column does exist. I do not report a significance test for retry, on purpose. The test works by weighing how many questions got fixed against how many got broken, and retry cannot break any. A query the database rejected was already wrong, so replacing it either helps or changes nothing. The broken count was zero in all four runs, which I checked rather than assumed. With nothing on the losing side, the test would call any gain significant at all, even a gain of one question. What actually means something here is the size of the gain against what it costs. The 483 queries that run cleanly and return wrong rows produce no error, so retry is structurally blind to them. But they do disagree with each other , and disagreement is an indicator/signal. So instead of taking the model's single most likely answer, sample eight different ones, run all eight, group them by the rows they return rather than by the text of the query , and answer with the largest group. Two queries written completely differently that return identical rows are probably both right. A hallucinated one usually returns something nobody else got. The grouping has to go through the same comparison function the benchmark uses, not a hash of the rows. That comparison ignores column order, so SELECT age, name and SELECT name, age count as one answer. Hashing would be faster and would split exactly the groups voting exists to merge. Two things are built in as checks rather than assumed. The first of the eight candidates is always the model's ordinary single answer, the one it would have given without voting, so setting k to 1 has to reproduce the plain score exactly. If it ever doesn't, something is wired wrong and I find out straight away. And voting at any k only looks at the first k candidates, so generating 16 once lets me score every budget from 1 to 16 without generating anything again. Every query that returns nothing looks identical to every other query that returns nothing. A broken filter, a made up condition matching no rows, and a genuinely empty answer all land in the same group and vote together. In every other group, agreement means several differently written queries arrived at the same rows, which is real evidence. In the empty group it only means several queries failed to return anything, which is not. Three people shrugging is not a consensus. So an empty group loses to any group that came back with actual rows. Both halves of that trade are measured. It is worth 1.3 points , and it costs 6 questions where the correct answer really was empty. | k | vote@k | pass@k, the ceiling | Gap | Gain from doubling | |---|---|---|---|---| | 1 | 68.1% | 68.1% | 0.0 | | | 2 | 69.2% | 70.1% | 0.9 | +1.2 | | 4 | 70.6% | 72.8% | 2.2 | +1.4 | | 8 | 71.7% | 75.4% | 3.7 | +1.1 | 16 | 72.1% | 76.9% | 4.8 | +0.4 | vote@k is what the system actually answers with. pass@k is whether any of the k candidates was correct, which you can only know by checking the answer key afterwards. It is the ceiling a perfect chooser could reach. Voting saturates and the ceiling does not. Each doubling buys 1.2, then 1.4, then 1.1, then 0.4. The last doubling returns a third of the one before it. But pass@k keeps climbing, so the gap between them widens from 0.0 to 4.8 points . That gap is right answers the model generated and threw away. At 16 samples the bottleneck is no longer producing a correct query. It is recognising the one already in hand. Measured at k = 8 alone this looks like a technique with room left in it, and it is not. greedy result → vote result correct → correct 1452 ran, wrong → ran, wrong 453 rejected → rejected 106 rejected → correct 45 rejected → ran, wrong 45 ran, wrong → correct 38 ← invisible to retry correct → ran, wrong 8 fixed 83, broke 8 p < 0.001 Significance testing is legitimate here, because voting can and did break things. Eight regressions makes the comparison two directional again. There are two separate results in that table. Voting reaches the bucket retry cannot see at all, 38 questions. And it beat retry on retry's own ground , repairing 45 rejected queries against retry's 23. Seven more samples turns out to be a better repair mechanism than one error message, which is worth knowing before building anything cleverer than sampling. retry voting 1.5B +1.1 +3.5 0.5B +0.2 +3.0 Retry needs a model good enough to act on feedback. Voting only needs one that is right sometimes , and then fishes that answer out. The second is a far weaker requirement, which is why voting survives at a model size where retry collapses entirely. That asymmetry is only visible because both techniques were measured at both sizes. Told as a 1.5B story, it would have been thrown away. Voting requires running eight or sixteen candidates anyway. Once you have done that, you already have something else for free, and I think it is the most useful output of the whole project. How much did the candidates agree with each other? | Agreement across 16 samples | Questions | Share | How often correct | |---|---|---|---| All 16 agree | 1,405 | 65.4% | 85.9% | | 12 to 15 agree | 274 | 12.8% | 60.6% | | 1 to 11 agree | 377 | 17.6% | 46.2% | | Nothing ran at all | 91 | 4.2% | 0.0% | When the model agrees with itself it is right 86% of the time, and that covers two thirds of all questions. When it does not, it is close to a coin flip 46.2% . In a product that distinction is worth more than the 3.5 points voting adds to the score. It is the difference between a system that silently returns a wrong number and one that can say "I am not confident about this one, check it". A wrong SQL answer does not look wrong. It looks like a number in a cell. 91 questions where not one of sixteen attempts produced a query the database would run. Not a wrong answer among them, because there was never an answer at all. Accuracy there is 0.0%, and it is 0.0% by definition rather than by bad luck. At eight samples these hide inside the low agreement band, indistinguishable from questions the model merely found hard. At sixteen they separate cleanly, and that separation is the whole value of them. The service can recognize these before it answers rather than after. Saying nothing on 4.2% of questions costs a great deal less than being confidently wrong on them. Each band is scored on its own members : An earlier version of this table lumped 7-of-8 agreement in with 6-of-8 and reported the pair as 62.8%. Scored separately, 7-of-8 was 62.8% and 6-of-8 was 47.9%. Lumping them made the weaker half look 15 points better than it was, and anyone trusting that number would have trusted answers they should not have. The bands are deliberately wide : The middle of the chart is a mess. 5-of-16 scores 33% while 1-of-16 scores 55%, but those are only 27 and 33 questions each, so the gap between them is chance rather than anything real. The order is wrong too, with less agreement sometimes scoring better than more. Splitting that middle into finer bands would be reporting a precision that is not there. One answer gets no confidence at all : If the model only answered once, there is nothing to compare it against, so the service returns "unmeasured" instead of a number. A confidence figure nobody has checked is worse than no figure. The same pattern shows up on the 0.5B model, just lower, with unanimous answers correct 80.7% of the time instead of 85.9%. That is what makes this a property of the approach rather than a fluke of one particular model. The vote vs ceiling gap from the last section has a specific shape. Voting picks a wrong answer while holding a correct one 3.6% of the time. That looked like the cheapest remaining win in the project. No training, no GPU, the candidates are already sitting on disk. So I looked at the failures before building anything. 87% of them are cases where the wrong answer won six votes to two. The model is confidently and consistently wrong, and no rule based on counting votes can override a 6-2 majority. I built two smarter selectors anyway and measured them at +0.1 and −0.1 points. Dead end, and worth the hour it took to prove rather than the week it would have taken to build. Closing that gap needs a model trained to score candidates against the question, not a tiebreak rule. If I had to throw away everything in this project except one idea, it would be this one. Accuracy tells you how often a system is right on average. It tells a user nothing about the answer currently on their screen. Self agreement tells you about this answer, not the average one, it costs nothing once you are sampling, it needs no extra model, and it works well enough to act on. The more valuable output of a system like this is not being right more often. It is knowing when to shut up. Every score so far has handed the model the complete, correct schema for the exact database the question is about. Ask about students, get the student database. Nothing else on screen. No real deployment works like that. A company's warehouse has hundreds or thousands of tables, nobody knows in advance which database a question belongs to, and the whole thing does not remotely fit in a model's context window. Something has to search the schema first, pick out the handful of tables that look relevant, and pass only those to the model. If that search picks wrong, the model never had a chance. This section takes the gift away and measures what it was worth. My first design was wrong, and the reason is worth a paragraph. Spider's databases are tiny. The median one has 4 tables and 19 columns , and writing it into a prompt as CREATE TABLE statements costs about 102 tokens. Searching for the right table among four is not a search problem. Any method scores near perfect and you learn nothing. The obvious fix is to throw all 206 databases into one big pile and search that. This fails badly. 125 table names appear in more than one database 436 of 1,053 tables affected 41% customers → 22 databases student → 12 addresses → 15 staff → 11 There are 22 different tables called customers in that pile, belonging to 22 unrelated databases. A question asking about customers genuinely does not say which one it means. No search method can resolve that , however good, because the information needed to resolve it is not in the question. A test built this way would measure an impossible task and then blame the retriever for failing it. So I built a collision free pool instead. Take databases one at a time, in order of how many benchmark questions they carry, and skip any database whose table names clash with one already taken. That leaves: Of those 300 tables, only 107 belong to databases that carry any questions. The other 193 are pure noise , sitting there to be wrong answers. And the pool genuinely does not fit. Written out as CREATE TABLE statements it comes to 8,262 tokens, against the 3,072 token prompt limit these models were evaluated under. Nearly three times over. Search is not an optimization here, it is the only way the prompt fits at all. One thing to carry through the rest of this section: every number below is over 1,457 questions, not 2,147 . Putting a retrieval number next to a number from earlier in this post is comparing two different tests. Each table becomes one short searchable document, just its name and its column names: student: id, name, age, dept id The search query is the question text and nothing else . Never the database name. That is deliberate and important: if the retriever knew which database a question came from, it could just look up that database's tables directly, and every number here would be measuring an answer key rather than a search. BM25 is keyword matching, the classic search engine approach. A table scores higher when more of the question's words appear in it, and rare words count for more than common ones. Matching the word "hangar" tells you a lot, matching "the" tells you nothing. One detail carries a surprising amount of weight. Schema names are written like city code or singerID , and people ask questions using words like "city" and "singer". A normal tokenizer treats city code as one indivisible thing, so a question about cities can never match it. Mine splits on both snake case and camelCase and keeps the joined form as well, so city code becomes city code , city and code . Bridging that gap between English and identifiers is most of what this job actually is. Dense retrieval matches on meaning instead of words. A small embedding model turns each table description into a list of numbers, positioned so that texts meaning similar things sit near each other. Do the same to the question, and the nearest tables win. It can match "how many people are enrolled" to a student table without either phrase sharing a single word with the other. Both methods return the top k tables, where k is just how many you decide to hand the model. Here is how often each finds what the question needs. | k | BM25 recall | BM25 coverage | Dense recall | Dense coverage | |---|---|---|---|---| | 1 | 33.9% | 25.1% | 58.3% | 44.0% | | 5 | 63.9% | 53.8% | 88.1% | 79.2% | 10 | 73.0% | 64.1% | 92.6% | 86.3% | | 20 | 78.6% | 70.7% | 95.1% | 90.3% | The column that matters is coverage , not recall, and the difference between them is not a technicality. Recall is the share of needed tables that were found. Coverage is the share of questions where every needed table was found. Say a question needs to join two tables and the search finds one of them. Recall calls that 50%, a decent partial score. But the question is exactly as unanswerable as if the search had found nothing at all, because you cannot write half a join. Coverage scores it zero, which is the truth. Dense wins clearly, which is not surprising given that questions are English and schemas are identifiers. BM25 is in here as the honest baseline, since it is free and needs no GPU, and if it had come close that would have been worth knowing. Now run the actual model on what each search method hands it. Four conditions, identical in every respect except which schema text goes into the prompt, all over the same 1,457 questions. | Condition | Schema shown | Accuracy | Queries that run | |---|---|---|---| Oracle | exactly the tables the answer uses | 68.4% | 92.1% | | Gold | the whole correct database | 63.5% | 89.7% | | Dense, top 10 | 10 tables out of 300 | 45.2% | 66.8% | | BM25, top 10 | 10 tables out of 300 | 37.1% | 57.5% | Two findings, and I predicted neither. Showing less schema helps, if it is the right schema : Oracle beats gold by 4.9 points, and gold is only showing tables from the correct database. Every irrelevant table costs something, even a related one. That is the exact opposite of what was true in training, where trimmed schemas were actively harmful because the model never learned to pick a table out of a crowd. Training on clutter is necessary. Having no clutter at question time is pure upside. Most of the loss is distraction, not absence : This is the one that surprised me. Split the dense results by whether the search actually found everything the question needed. all needed tables present 1,257 questions 86.3% accuracy 51.9% at least one missing 200 questions 13.7% accuracy 3.0% The missing group behaves exactly as you would expect. 3.0% is effectively zero, and those questions were unanswerable the moment the search missed. Fine. But look at the other group. Every table the question needed was sitting right there in the prompt, and accuracy was still only 51.9% against oracle's 68.4%. Sixteen and a half points destroyed purely by the nine irrelevant tables sitting next to the right ones. The share of queries the database will even run drops from 92.1% to 66.8%, because the model keeps reaching for a plausible looking table that belongs to some completely unrelated database. Stacking those two costs together: oracle 68.4% minus distraction 16.5 points → 51.9% right tables present, plus nine more minus absence 6.7 points → 45.2% 13.7% of questions miss a table Distraction costs two and a half times what absence does. I had predicted 58 to 61% before running this, by reasoning that accuracy would be roughly coverage multiplied by the baseline. In other words, I assumed that finding the tables was the whole problem, and that once found, the model would perform as it always had. The gap between that prediction and the real 45.2% is exactly the distraction cost, which my mental model of the system had no room for at all. If distraction is the dominant cost, then handing the model fewer tables should reduce it. I wrote that prediction down before testing it. Six more runs, both search methods at three different values of k, say it is wrong. | Method | k | Coverage | Accuracy when covered | Accuracy when missing | Accuracy overall | |---|---|---|---|---|---| | Dense | 5 | 79.2% | 55.6% | 1.0% | 44.3% | Dense | 10 | 86.3% | 51.9% | 3.0% | 45.2% | | Dense | 20 | 90.3% | 49.4% | 2.1% | 44.8% | | BM25 | 5 | 53.8% | 63.6% | 2.2% | 35.3% | | BM25 | 10 | 64.1% | 57.1% | 1.3% | 37.1% | | BM25 | 20 | 70.7% | 54.1% | 2.6% | 39.0% | Look at the dense rows first. Overall accuracy reads 44.3, 45.2, 44.8 across a fourfold change in how many tables get retrieved. Flat enough that you would reasonably conclude k does not matter much. That conclusion would be wrong, and the last two columns show why. Two opposing forces are cancelling out. Coverage climbs from 79% to 90%, because searching wider finds the right tables more often. Accuracy on questions that were already covered falls from 56% to 49%, because every extra table is another distraction. One goes up, the other goes down, and the total sits still. BM25 starts so starved of recall, only 54% coverage at k=5, that the coverage gain outweighs everything else and its overall number climbs from 35.3 to 39.0. Its covered accuracy falls on exactly the same schedule though, 64% to 57% to 54%. Whether raising k appears to help depends entirely on whether your search method still had recall left to gain. It never depends on the distraction going away, which got worse in every single condition tested. So the honest reading is narrower than "retrieve less". The number of tables you retrieve is not a dial worth tuning. It trades coverage against distraction at roughly one for one, and the 16.5 point distraction penalty survives every setting I tried. What would actually move this is making the model itself robust to irrelevant tables, by training it on cluttered schemas instead of clean ones. That is a training change, not a search change. A project can report 68.1% accuracy and say nothing at all about whether an answer takes 200 milliseconds or 30 seconds. So the last piece is an HTTP service, and a benchmark of it. The service reuses the voting and retry code directly rather than reimplementing it. That is the point. A second copy of the clustering or selection logic could drift away from the numbers those techniques were measured at, and nothing would tell you. A request returns the SQL, the rows, a confidence band from the agreement table, and a timing breakdown. Measured on one A10G handling one request at a time , which is what a service does, unlike the evaluation harness that batches sixteen together. Over the full test split, so the accuracy column is real evidence rather than a spot check. | Mode | Accuracy | p50 | p95 | p99 | Mean | Queries/sec | |---|---|---|---|---|---|---| | Greedy | 67.9% | 2,652 ms | 5,100 ms | 7,162 ms | 3,088 ms | 0.32 | Vote, 8 samples | 71.3% | 5,650 ms | 10,789 ms | 15,272 ms | 6,522 ms | 0.15 | | Retry, 3 rounds | 68.9% | 2,714 ms | 9,739 ms | 16,042 ms | 3,777 ms | 0.26 | p50 is the median request. p95 is the slowest one in twenty. p99 is the slowest one in a hundred. That accuracy column is doing quiet work. It reproduces the harness from a completely different code path with different batching, giving +3.4 for voting and +1.0 for retry against the harness's +3.5 and +1.1. Retry's median is indistinguishable from plain greedy , 2,714 ms against 2,652, because most requests never enter the loop at all. Every bit of its cost sits in the tail. Its mean of 3,777 ms hides that from both directions simultaneously. It is 40% above a median that describes most requests, and 76% below a p99 that describes the requests a user actually complains about. It describes neither. Voting is the opposite shape. It costs 2.1 times more at the median, but its p99 is only 2.7 times its own median, so what you measure in staging is what you get in production. The mean says retry is cheaper. The tail says voting is more predictable. Users experience the tail. That last one is the whole system running. Three questions against a test database, answered by the service. The first two are correct and unanimous. The third asks for distinct types of planes owned by pilots, and returns four plausible plane names from the wrong table. Nothing in the SQL or the output reveals that. The only sign is one sample out of eight disagreeing, which drops the confidence badge from high to medium. All of this is one g5.xlarge in us-east-1. A single NVIDIA A10G with 24GB, four vCPUs, $1.006 an hour on demand. The three serving benchmarks above took ~1.8, ~4 and ~2.5 hours respectively, which is most of a working day of GPU time spent purely on measuring something I had already built. That ratio is the honest summary of the project. Far more compute went into finding out whether things worked than into making them work. Four pieces of infrastructure earned their place: Automatic shutdown when idle : A cron job checks GPU utilization every minute and shuts the box down after 30 minutes below 5%. A forgotten g5.xlarge is about $170 a week, and this is the single highest value cost control in the project. But the first version measured the wrong thing. GPU utilization alone is not "is this machine in use". An evaluation run spends real minutes loading weights, building prompts, executing SQL against SQLite and scoring results, all at 0% GPU. That script shut down a live run twice before I fixed it. Idleness is now graded. If a process is still holding GPU memory, or a human is still logged in, the limit becomes 180 minutes instead of 30. Not immunity, because "forgot to log out" must not cost $170 a week either. Rescuing work from spot instances : Spot instances are 50 to 60% cheaper, and AWS can reclaim them with a two minute warning delivered through the instance metadata service. Reclaiming means terminate, not stop, so the disk goes with it. A service polls for that warning and syncs checkpoints to S3 when it arrives. Two minutes is enough to do one thing quickly and not enough to do anything clever. It is not the primary mechanism though. A missed notice or a kernel panic gives no warning at all, so checkpoints also go to S3 on every save. Resuming is not restarting the same machine, it is a fresh machine pulling the last checkpoint from S3 and carrying on from that step, which I tested by killing a run and deleting its disk. One thing to note is that I never actually ran on spot. AWS took forever to process my quota request and by the time it came through, most of the project was already done. So this is built and tested but not battle worn. It is a genuinely good feature and half the price is half the price, so if you use it, let me know how it holds up. Never mirror a checkpoint directory to S3. The tidy way to stop S3 filling up with old checkpoints is aws s3 sync --delete , which mirrors the local directory. It is also one fresh instance away from deleting the only copy of a run, because a new machine's checkpoint directory is empty and mirroring an empty directory deletes everything. Old checkpoints are about 110MB each. Letting them pile up is far cheaper than the failure they prevent. Also, I originally set everything up in us-east-2, because GPU instances ran about 30% cheaper there. Then I found out I could not launch an instance at all. AWS caps how many of each instance type you can run, per region, and those caps start at zero for GPU instances until you ask for an increase. Mine in us-east-2 was zero and stayed zero, so the cheaper price was for something I was not allowed to buy. Everything moved to us-east-1, where I already had approval for a few. Check your limit in a region before you plan around its pricing. Also worth knowing: a stopped instance still charges you for its disk. Mine was about $16 a month for a 200GB volume doing nothing at all. If you are finished with a machine, terminate it rather than leaving it stopped. Something trained to pick the answer : At sixteen tries the model writes a correct query for 77% of questions and picks it only 72% of the time. Those 4.8 points are right answers it already produced and threw away. I tried two hand written rules for choosing better and they scored +0.1 and −0.1, so this needs a model trained to judge candidates against the question, not another rule. Training on messy schemas : The model learned on tidy, correct database descriptions, then gets handed cluttered ones full of irrelevant tables and loses 16.5 points to the clutter. That is fixable in training, by showing it messy schemas while it learns, rather than in the search step. Faster serving : 2.7 seconds for a typical question is slow, and the service currently handles one request at a time. Processing several at once on the GPU is the standard fix and the single biggest weakness in these numbers. Not multi turn reinforcement learning , which was my original plan. The idea was to train the model on the whole back and forth of the retry loop rather than on single answers. Two things killed it. Only 37% of the queries it repairs end up correct, so there is not much there to learn from. And at 1.5B, over half of training groups already teach the model nothing, because all its attempts agree with each other and it needs disagreement to learn. Training on longer conversations makes that worse. The headline is that a 1.5B model matched a 7B one. That is true, and it is also the least interesting thing here. The model did not get good because it got bigger. It went from 6.4% to 49.7% at half a billion parameters, before I touched the model size at all, and every one of those gains came from something outside the weights. A way to check answers by running them. A reward that had a gradient in it. Training data that did not hide the hard part. Then, once the model was as good as that size allowed, the same measuring tools were pointed at a bigger base and the gains transferred intact. And the last 3.4 points, the ones that actually drew level with the 7B, came from no training whatsoever. Sample eight answers, run all eight, keep whichever result most of them agreed on. That is a system decision, not a model decision. Look at what the pieces actually are, stripped of SQL. Only the first is SQL specific, and only barely. Anywhere you can mechanically decide whether an output is correct, all six apply. Code that has to compile and pass tests. Structured extraction validated against a schema. Maths with a checker. An API call that either succeeds or does not. In each case you can build the same loop, and in each case a small model wrapped in it will go further than its parameter count suggests. This is why the small model + good system approach works, and why so many people are converging on it right now. You are not trying to beat a frontier model at everything. You are picking one task, building the infrastructure that knows what correct looks like, and letting a cheap model take as many attempts as it needs. The expensive part was never the model. It was building the thing that could tell me when I was wrong. Thank you for reading my blog, this was fun to build, and I learnt a lot of things too. Happy to receive your thoughts in the comments Connect with me on LinkedIn https://www.linkedin.com/in/pradhyumna-n-holla/ and X https://x.com/PradHolla . The benchmark The techniques Tools GRPOConfig defaults carefully, since two of them silently changed the meaning of my reward. AWS EC2 G5 Instances https://aws.amazon.com/ec2/instance-types/g5/ — High performance GPU based instances for graphics intensive applications and ML inference Spot instance interruption notices https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-interruptions.html — the two minute warning, and how to poll for it from the instance metadata service. This is what the checkpoint rescue in this post is built on, and it is worth reading before you put a training run on spot rather than after.