{"slug": "how-to-build-a-tiny-1-5b-text-to-sql-model-that-beats-a-7b", "title": "How to build a tiny 1.5B text-to-SQL model that beats a 7B", "summary": "A developer built a 1.5B parameter text-to-SQL model that outperforms a 7B model on the Spider benchmark, achieving 71.5% accuracy through self-consistency voting. The project, starting with Qwen2.5-0.5B and progressing to Qwen2.5-Coder-1.5B, highlights the importance of evaluation harnesses and the trade-off between model size and compute.", "body_md": "I wanted to build something with an LLM using my own hands. Not wire an API into a wrapper, but take a base model, train it, measure it, break it, and serve it over HTTP. Why? Because I wanted to get my hands dirty working with an LLM, making and breaking things.\n\nText to SQL fits exactly. You ask a question in plain English, the system writes SQL, and you can run the query to find out whether it was right. Correctness is whether the rows match the reference query's rows against the real database, which is a fact rather than a judgement. It also has a mature benchmark in [Spider](https://huggingface.co/datasets/xlangai/spider), 10,000 human written questions over 200 real SQLite databases, split so the test databases never appear in training.\n\nSo I built it. Qwen2.5-0.5B as the base, LoRA adapters, one g5.xlarge with an NVIDIA A10G, about a dollar an hour. Continued pretraining on SQL text, then supervised fine-tuning on question and query pairs, then reinforcement learning with GRPO on top. The dashboards looked great. Reward climbing to 1.0, loss dropping cleanly through every stage.\n\nIt scored 6.4%. Comically bad levels of accuracy.\n\nI didn't find that out for a while, because there was no held out evaluation. Every number I had came from data the model had trained on. The untrained base model scored 17.4%, so three stages of training had made things worse than no training at all.\n\nThe fix wasn't a better model. It was building the thing that could tell me I was wrong, an evaluation harness that runs both queries against the real database and compares the rows that come back.\n\nThen I rebuilt. 6.4% to 44.6% on real schemas, then 49.7% with a proper RL reward, all still at 0.5B. Only then did I switch to Qwen2.5-Coder-1.5B, which landed at 68.1%. Sampling eight answers per question and returning whichever result most of them agreed on took it to 71.5%, against 71.2% for Qwen2.5-Coder-7B-Instruct.\n\nThe 7B is still better at one attempt, 71.2% to 68.1%. The 1.5B gets there by answering eight times and voting, trading compute for the gap.\n\nThat trade is the point though, and the point of this entire blog, and it's the one a lot of people are making now. Take a small model, aim it at one task, and build enough system around it that it beats something far larger at that one thing.\n\nThe first version was three training stages run back to back on `Qwen2.5-0.5B`\n\n, a half billion param open model from Alibaba, all of them using LoRA([Low-Rank Adaptation](https://arxiv.org/abs/2106.09685)), which freezes the model's real weights and trains a tiny fraction of new, lightweight parameters.\n\n**Continued PreTraining**: keep training the base model on raw SQL text so it gets used to the shape of the language.\n\n**Supervised FineTuning**: show it thousands of question and correct-query pairs and have it imitate them.\n\n**Reinforcement Learning with GRPO**: Group Relative Policy Optimization. The model writes several answers to the same question, each is scored, and it gets pushed toward whichever ones scored above that group's average. It learns from its own attempts rather than from copied answers.\n\nHere is what those runs reported:\n\n| Stage | Steps | Loss | Data actually seen |\n|---|---|---|---|\n| Continued PreTraining | 500 | 2.46 to 0.81 | ~2.7 passes over the corpus |\n| Supervised FineTuning | 500 | 3.26 to 0.51 |\n0.10 of one pass, about 8,000 of 78,577 examples |\n| Reinforcement Learning | 300 | reward 0.8 to 1.0 | 5,000 rows of the same training set |\n\nYou have to understand that this was a learning project and my approach was flawed, so a few things went wrong. Every curve went the right way, and none of them meant anything, for three reasons.\n\n**The data taught the wrong task**: `sql-create-context`\n\nhands the model a schema already trimmed to exactly the columns the answer needs, so it never has to work out which table matters. Working that out is the entire job. It also saw only a tenth of that data, because the run was configured by step count rather than by passes over it.\n\n**The reward could not teach anything**: a formatting reward worth 1.0 fired for essentially every answer, and an exact string match reward worth 2.0 fired for almost none, so a group of four answers usually scored `[1.0, 1.0, 1.0, 1.0]`\n\n. GRPO scores each answer against its group's average, so when they all score the same the update is zero. The metric that tracks this hit 1.00 by step 160, meaning a large share of those 300 steps did nothing at all. And exact string matching is the wrong test anyway. It scores `age>56`\n\nas zero against `age > 56`\n\n.\n\n**There was no held out evaluation**: the RL stage ran on a slice of the same data the model had been finetuned on, so every number I had measured how well it remembered its own training set.\n\nThat last one is the real defect and the other two follow from it. Bad data and a broken reward are ordinary mistakes. What made them expensive is that nothing in the system could report them. The dashboards were green throughout.\n\nI stayed on the half billion param model for all of this. Not because it was good, but because a full fine tune plus a full evaluation there is about two GPU hours. When you are wrong five times in a row, that matters more than the ceiling does.\n\nThe old training set handed the model schemas already trimmed to the columns the answer needed. The rebuild uses Spider with the real, complete database schemas, so the model has to find the right table among all the wrong ones. Three other things changed at the same time.\n\n**Loss is computed on the answer only**: A training example is the schema, then the question, then the correct query. With full schemas the schema part is roughly 75% of the tokens. If you score the model on reproducing the whole example, most of the training signal is teaching it to generate database schemas, which nobody asked for. Masking the prompt means every bit of the signal lands on the query.\n\n**Two full passes over the data instead of a tenth of one**: An epoch is one complete pass through the training set. The old run had covered 0.10 of one.\n\n**A learning rate suited to what was actually being trained**: The learning rate controls how big each update is. The old value was appropriate for nudging a model that already knew the task. LoRA starts its extra weights from scratch, so it needs a larger one.\n\nThat is the single biggest change in the project. **6.4% to 44.6%.**\n\nOne thing I did not expect. Validation loss picked the wrong checkpoint. Epoch 2 looked slightly worse than epoch 1 on both validation loss and token accuracy, and it was **3.9 points better** on real execution accuracy. Loss is a proxy for a proxy. Run the metric you actually care about, on held out data, at every checkpoint.\n\nIn hindsight the old setup is a bit embarrassing, but every one of these fixes came from the failures, not from knowing better. Something that's true for most things that we do, not just machine learning.\n\nNow that the harness existed, the reward could use it. Instead of a black and white, all or nothing test, the reward is a ladder with partial credit.\n\n| Outcome | Reward | What it buys |\n|---|---|---|\n| No SQL, or it will not parse | 0.0 | nothing |\n| It parses as SQL | 0.2 | syntax is a solved sub-problem |\n| The database accepted and ran it | 0.5 | the tables and columns actually exist |\nThe rows match the reference |\n2.0 |\ncorrect |\n\nThe point of the middle rungs is the zero gradient problem from before. A group of four answers where none is fully correct now scores something like `[0.2, 0.5, 0.5, 0.2]`\n\ninstead of `[1.0, 1.0, 1.0, 1.0]`\n\n. There is real disagreement inside the group, so there is a real update.\n\nThe 0.5 rung is aimed at one specific failure. Going into this, 88% of the model's remaining mistakes were invented column and table names. That rung pays for nothing except the columns existing.\n\n**44.6% to 49.7%.** The mechanism moved in exactly the way the ladder was designed to move it.\n\n```\nunknown_column   647 → 458   (down 29%)\nunknown_table     66 →   7   (down 89%)\n```\n\nTwo models answering the same 2,147 questions share most of their answers, and the shared ones tell you nothing. Only the questions where they disagree carry information. **McNemar's test** looks at exactly those. If the two models were equally good, the disagreements should split roughly evenly between them.\n\nThey did not. RL fixed 251 questions and broke 142. The test says a split that lopsided happens by chance with probability below 0.001.\n\nThe fixed and broken counts matter as much as the test. The headline is plus 5.1 points, but that is a **net**. It is not a clean sweep, and reporting it as one would hide 142 regressions.\n\nNone of these throw an error. They just quietly change what the reward means, and nothing tells you it happened.\n\n**Reward scaling flattens the ladder**: the library divides each group's scores by how spread out that group was. The order of the rungs survives, the spacing does not, so a group whose best answer was genuinely correct ends up pushing exactly as hard as one whose best answer merely parsed. The ladder becomes a ranking. Switching that off keeps the 7.5 times gap between correct and parsed intact.\n\n**The reference model is not the one you think**: RL usually penalizes the model for drifting too far from a reference version of itself, measured as KL divergence. Under LoRA that reference is your adapter switched off, which is the raw base model, not your finetuned checkpoint. So the penalty was measuring how far supervised finetuning had already moved, not what RL was doing. I set it to zero rather than pull the model back toward something I had deliberately trained it away from.\n\n**Row caps must not be able to change a verdict**: the reward stops reading rows once it has one more than the correct answer has. Anything longer could not have matched anyway, so the cap can never turn a right answer into a wrong one. It only stops a hallucinated cross join from materializing a million rows inside the training loop.\n\nReward hacking is when the model finds a way to score well without doing the task. It is usually discovered from a reward curve that climbs while the benchmark stays flat, which is to say after the GPU hours are gone.\n\nSo before spending the GPU hours, I attacked my own reward. I wrote eleven policies that never look at the question at all, and measured how much reward each one could collect. `SELECT 1`\n\nalways parses and never matches. `SELECT * FROM <first table>`\n\nalways runs. `WHERE 1 = 0`\n\ngoes after the empty result loophole. Degenerate cross joins test the row cap and the parser guards.\n\nThe best of them collected **26.8%** of what a real answer gets, and got **1.8%** of questions outright, which is the empty answer slice and exactly where I expected it to score. So the reward is not easy to cheat, and that is a number now rather than an assumption.\n\nThis probe is also what caught the reward scaling default above.\n\nThe original training data told the model to think before answering. The problem was that the \"thinking\" was **one hardcoded sentence repeated across all 5,378 examples**. As scratch paper it is blank. It carries no information about the specific question, so all the model learns is to recite a preamble before answering exactly as it would have anyway.\n\nThe proper version of this is **rejection sampling**, sometimes called STaR(Self-Taught Reasoner). Take a larger model that can actually reason, have it solve each of your training questions eight times while showing its work, then **run every attempt against the database and keep only the ones whose SQL was correct**.\n\nThat filter is the entire technique. A large model is wrong confidently and fluently, and training a small model on fluent wrong reasoning is worse than not training it at all. The harness from earlier is what makes the filter possible, which is the third job that one component ended up doing.\n\nIt worked as a data pipeline. 4,823 verified traces, covering 90.8% of the training questions, with a median of 114 words of reasoning that named real tables and real columns.\n\n**And it made the model worse. 44.6% down to 41.8%.**\n\n| Training data | Tokens the model writes | Of which are SQL | Share of the signal landing on the SQL |\n|---|---|---|---|\n| Canned sentence | 62.6 | 33.6 | 53.7% |\n| Real traces | 195.1 | 31.1 | 15.9% |\n\nTraining only teaches the model on the tokens it writes. With the canned sentence, it wrote about 63 tokens per answer and 34 of them were the SQL, so more than half its practice was on the query itself. With real traces it wrote about 195 tokens and the SQL was still only 31 of them. The query had not got any longer. It was just buried under 114 words of reasoning.\n\nSo the model went from spending half its practice on the query to spending a sixth of it. Same number of passes over the data, same learning rate, 3.4 times less practice on the only part that gets marked. Running it for twice as many passes gave that practice back, and it came out at **47.1%**, above where it started.\n\nRunning reinforcement learning on top of the trace trained model was a wash. Plus 0.1 points, 179 questions fixed and 177 broken. Statistically that is nothing.\n\nThe reason is that both interventions attack the same failure. Traces help the model name real tables and columns because the reasoning it copied named real tables and columns. The execution reward helps with exactly the same thing, and it is better at it. So the traces had already done a partial version of RL's job and left it nothing to work on.\n\nThe best 0.5B model in the project stayed the straightforward one, supervised finetuning plus RL, at **49.7%**.\n\nIt cost real time and it did not produce the headline number, so the temptation is to leave it out. Two reasons not to.\n\nFirst, a negative result with a measured cause is more useful than a positive one without. \"Reasoning traces did not help\" is folklore. \"Reasoning traces diluted the training signal on the answer by 3.4 times, and doubling the passes over the data fixed it\" is something you can act on.\n\nSecond, it is a clean example of the thing this whole project is about. The pipeline ran, the data was verified, the traces were genuinely good, and the model got worse. Nothing about that was visible from the training curves. The only reason I know any of it is that there was a held out number to check.\n\nAt 49.7% the 0.5B model had run out of road with this training setup. The failure breakdown said the remaining errors were not the kind a better reward fixes, so I changed the base model to `Qwen2.5-Coder-1.5B`\n\n. Three times the parameters, and pretrained on code rather than general text, which should help with a task whose output is code.\n\nSame pipeline, nothing else changed. **67.9%**, and 68.1% after RL.\n\nThat is a large jump and it invites the obvious objection. Maybe none of the previous work mattered and I just needed a bigger model. That is an empirical question, so I answered it with an experiment instead of an opinion. Run the **old** pipeline and the **new** one on **both** base models. Four runs, every combination.\n\n| Pipeline | 0.5B | 1.5B | What scale bought |\n|---|---|---|---|\n| Old | 4.6% | 46.9% | +42.4 |\nNew |\n44.6% |\n67.9% |\n+23.2 |\nWhat technique bought |\n+40.1 |\n+20.9 |\n\nBoth effects are large and both hold up statistically. Technique at 1.5B fixed 559 questions and broke 110. Scale under the new pipeline fixed 577 and broke 78.\n\n**Both matter, and they eat into each other**: Technique is worth 20.9 points even on a bigger, code pretrained base, so the work is not just compensating for a weak model. But that is half what it was worth at 0.5B, because the bigger model had already solved some of what the technique was fixing.\n\nThe corners of the table show the same thing. Measured on its own, technique is worth 40.1 points and the bigger model is worth 42.4, so you would expect 82 from doing both. Going from the worst combination to the best actually gets you 63.3, from 4.6% to 67.9%. The missing 19 points are questions both changes would have fixed on their own, and you only get paid for those once.\n\nThe old pipeline at 0.5B also carried a tokenizer bug that stopped generation from terminating properly, while the 1.5B version of it used a working stop token. So the +42.4 in the top row is partly \"a bug was also fixed\". The two numbers I actually rely on are the ones underneath it, where everything else is held fixed.\n\nAnd the scale column is not purely scale. The new base is both larger **and** code pretrained, so those two things are tangled together in the +23.2 and I cannot separate them without a fourth base model I did not train. The technique column is clean, since that is the same base with two pipelines. The scale column should be read as \"bigger and code pretrained\", not \"bigger\".\n\nWorth saying that the old pipeline was set up to lose as gracefully as possible. Its destructive pretraining stage was left out, the library that corrupted the vocabulary was left out, the working stop token was used. It is the strongest fair version of the thing it loses to.\n\nSame reward, same configuration, run on the new base.\n\n```\nRL on the 0.5B base:  +5.1 points   251 fixed, 142 broken   p < 0.001\nRL on the 1.5B base:  +0.2 points   122 fixed, 117 broken   not significant\n```\n\n\"Not significant\" here means the fixed and broken counts are close enough that the difference is indistinguishable from chance. 122 against 117 is a coin flip.\n\nThe strange part is that the mechanism still worked perfectly. Hallucinated columns fell 23%. The share of queries the database would actually run rose from 88.2% to 90.6%. Syntax errors dropped from 5 to 2. **Everything improved except the score.**\n\nThe reason is that the reward pays for executability, and executability had stopped being the bottleneck. At 0.5B, invented columns dominated the failures, so fixing them moved the number. At 1.5B the model already runs 90% of its queries, and what remains are queries that execute perfectly and answer the wrong question. This reward cannot see that. A query that runs and returns wrong rows scores 0.5, exactly the same as one that runs and returns wrong rows for a completely different reason.\n\nTwo diagnostics from that run are worth carrying to any RL project:\n\n**56% of training groups produced no gradient at all**: That is the original v0 failure arriving for the opposite reason. Back then the rollouts were uniformly wrong, so nothing separated them. Now they are uniformly right. **Reinforcement learning gets harder as your policy gets better**, because it runs on disagreement inside a group, and a good model disagrees with itself less.\n\n**Training reward rose 24% while held out accuracy moved 0.2 points**: With entropy falling steadily the whole time. Entropy here is how spread out the model's choices are, and it falling means the model is getting more confident and less varied. That is precisely the shape of the dashboards that started this whole project. The only difference is that this time there was a held out number sitting next to it saying the gain was not real.\n\nEverything so far changes the model's weights. At 68.1% it is worth asking what the remaining 32% actually looks like, because the answer decides what you can do about it without training anything.\n\n```\n1,462  correct                                        68.1%\n  483  runs perfectly, answers the wrong question     22.5%   no error exists\n  202  the database rejects it outright                9.4%   an error message exists\n```\n\nTwo completely different populations. The second one has a signal attached, because when SQLite refuses a query it tells you why, in words, like `no such column: customer_name`\n\n. The first one has nothing. The query runs, rows come back, and everything looks fine.\n\nThe obvious move for the 202 is to show the model its own error and let it try again, up to three rounds.\n\nTwo things make that a measurement rather than a demo. The loop **never sees the correct answer**. Its only stopping signal is whether the database accepted the query, which is all that exists at inference time in the real world. And the rounds are **batched rather than per question**. Round one generates all 2,147, and only the roughly 200 rejections go into round two. Looping question by question is about 50 times slower for an identical answer.\n\n**It bought 1.1 points.** I had predicted 4 and written that down beforehand.\n\nThe reason is the useful part.\n\n```\n202  rejected on the first attempt\n 63  became something the database would run    31% of rejections\n 23  were actually correct                      37% of those repairs\n 40  moved from \"database refused it\" to \"runs fine, wrong rows\"\n```\n\nThere are two multiplications there, not one. **An error message tells you that you are wrong. It never tells you what is right.** So the model fixes the complaint rather than the answer, and two thirds of its repairs migrate from the visible failure bucket into the invisible one.\n\nAt 0.5B it does essentially nothing. 500 rejections, 12 repaired. A weak model told \"no such column: X\" still has no idea which column does exist.\n\nI do not report a significance test for retry, on purpose. The test works by weighing how many questions got fixed against how many got broken, and retry cannot break any. A query the database rejected was already wrong, so replacing it either helps or changes nothing. The broken count was zero in all four runs, which I checked rather than assumed. With nothing on the losing side, the test would call any gain significant at all, even a gain of one question. What actually means something here is the size of the gain against what it costs.\n\nThe 483 queries that run cleanly and return wrong rows produce no error, so retry is structurally blind to them. But they do **disagree with each other**, and disagreement is an indicator/signal.\n\nSo instead of taking the model's single most likely answer, sample eight different ones, run all eight, group them **by the rows they return rather than by the text of the query**, and answer with the largest group. Two queries written completely differently that return identical rows are probably both right. A hallucinated one usually returns something nobody else got.\n\nThe grouping has to go through the same comparison function the benchmark uses, not a hash of the rows. That comparison ignores column order, so `SELECT age, name`\n\nand `SELECT name, age`\n\ncount as one answer. Hashing would be faster and would split exactly the groups voting exists to merge.\n\nTwo things are built in as checks rather than assumed. The first of the eight candidates is always the model's ordinary single answer, the one it would have given without voting, so setting **k to 1** has to reproduce the plain score exactly. If it ever doesn't, something is wired wrong and I find out straight away. And voting at any k only looks at the first k candidates, so generating 16 once lets me score every budget from 1 to 16 without generating anything again.\n\nEvery query that returns nothing looks identical to every other query that returns nothing. A broken filter, a made up condition matching no rows, and a genuinely empty answer all land in the same group and vote together. In every other group, agreement means several differently written queries arrived at the same rows, which is real evidence. In the empty group it only means several queries failed to return anything, which is not. Three people shrugging is not a consensus.\n\nSo an empty group loses to any group that came back with actual rows. Both halves of that trade are measured. It is worth **1.3 points**, and it costs 6 questions where the correct answer really was empty.\n\n| k | vote@k | pass@k, the ceiling | Gap | Gain from doubling |\n|---|---|---|---|---|\n| 1 | 68.1% | 68.1% | 0.0 | |\n| 2 | 69.2% | 70.1% | 0.9 | +1.2 |\n| 4 | 70.6% | 72.8% | 2.2 | +1.4 |\n| 8 | 71.7% | 75.4% | 3.7 | +1.1 |\n16 |\n72.1% |\n76.9% |\n4.8 |\n+0.4 |\n\n`vote@k`\n\nis what the system actually answers with. `pass@k`\n\nis whether **any** of the k candidates was correct, which you can only know by checking the answer key afterwards. It is the ceiling a perfect chooser could reach.\n\nVoting saturates and the ceiling does not. Each doubling buys 1.2, then 1.4, then 1.1, then 0.4. The last doubling returns a third of the one before it. But `pass@k`\n\nkeeps climbing, so **the gap between them widens from 0.0 to 4.8 points**.\n\nThat gap is right answers the model generated and threw away. At 16 samples the bottleneck is no longer producing a correct query. It is recognising the one already in hand. Measured at k = 8 alone this looks like a technique with room left in it, and it is not.\n\n```\ngreedy result  →  vote result\n    correct    →  correct      1452\n  ran, wrong   →  ran, wrong    453\n   rejected    →  rejected      106\n   rejected    →  correct        45\n   rejected    →  ran, wrong     45\n  ran, wrong   →  correct        38   ← invisible to retry\n    correct    →  ran, wrong      8\n\nfixed 83, broke 8    p < 0.001\n```\n\nSignificance testing is legitimate here, because voting can and did break things. Eight regressions makes the comparison two directional again.\n\nThere are two separate results in that table. Voting reaches the bucket retry cannot see at all, 38 questions. And it **beat retry on retry's own ground**, repairing 45 rejected queries against retry's 23. Seven more samples turns out to be a better repair mechanism than one error message, which is worth knowing before building anything cleverer than sampling.\n\n```\n              retry    voting\n1.5B          +1.1      +3.5\n0.5B          +0.2      +3.0\n```\n\n**Retry needs a model good enough to act on feedback. Voting only needs one that is right sometimes**, and then fishes that answer out. The second is a far weaker requirement, which is why voting survives at a model size where retry collapses entirely.\n\nThat asymmetry is only visible because both techniques were measured at both sizes. Told as a 1.5B story, it would have been thrown away.\n\nVoting requires running eight or sixteen candidates anyway. Once you have done that, you already have something else for free, and I think it is the most useful output of the whole project. **How much did the candidates agree with each other?**\n\n| Agreement across 16 samples | Questions | Share | How often correct |\n|---|---|---|---|\nAll 16 agree |\n1,405 | 65.4% | 85.9% |\n| 12 to 15 agree | 274 | 12.8% | 60.6% |\n| 1 to 11 agree | 377 | 17.6% | 46.2% |\n| Nothing ran at all | 91 | 4.2% | 0.0% |\n\nWhen the model agrees with itself it is right 86% of the time, and that covers two thirds of all questions. When it does not, it is close to a coin flip(46.2%).\n\nIn a product that distinction is worth more than the 3.5 points voting adds to the score. It is the difference between a system that silently returns a wrong number and one that can say \"I am not confident about this one, check it\". A wrong SQL answer does not look wrong. It looks like a number in a cell.\n\n91 questions where not one of sixteen attempts produced a query the database would run. Not a wrong answer among them, because there was never an answer at all. Accuracy there is 0.0%, and it is 0.0% by definition rather than by bad luck.\n\nAt eight samples these hide inside the low agreement band, indistinguishable from questions the model merely found hard. At sixteen they separate cleanly, and that separation is the whole value of them. The service can recognize these before it answers rather than after. Saying nothing on 4.2% of questions costs a great deal less than being confidently wrong on them.\n\n**Each band is scored on its own members**: An earlier version of this table lumped 7-of-8 agreement in with 6-of-8 and reported the pair as 62.8%. Scored separately, 7-of-8 was 62.8% and 6-of-8 was 47.9%. Lumping them made the weaker half look 15 points better than it was, and anyone trusting that number would have trusted answers they should not have.\n\n**The bands are deliberately wide**: The middle of the chart is a mess. 5-of-16 scores 33% while 1-of-16 scores 55%, but those are only 27 and 33 questions each, so the gap between them is chance rather than anything real. The order is wrong too, with less agreement sometimes scoring better than more. Splitting that middle into finer bands would be reporting a precision that is not there.\n\n**One answer gets no confidence at all**: If the model only answered once, there is nothing to compare it against, so the service returns \"unmeasured\" instead of a number. A confidence figure nobody has checked is worse than no figure.\n\nThe same pattern shows up on the 0.5B model, just lower, with unanimous answers correct 80.7% of the time instead of 85.9%. That is what makes this a property of the approach rather than a fluke of one particular model.\n\nThe vote vs ceiling gap from the last section has a specific shape. **Voting picks a wrong answer while holding a correct one 3.6% of the time.** That looked like the cheapest remaining win in the project. No training, no GPU, the candidates are already sitting on disk.\n\nSo I looked at the failures before building anything. **87% of them are cases where the wrong answer won six votes to two.** The model is confidently and consistently wrong, and no rule based on counting votes can override a 6-2 majority.\n\nI built two smarter selectors anyway and measured them at +0.1 and −0.1 points. Dead end, and worth the hour it took to prove rather than the week it would have taken to build. Closing that gap needs a model trained to score candidates against the question, not a tiebreak rule.\n\nIf I had to throw away everything in this project except one idea, it would be this one. Accuracy tells you how often a system is right on average. It tells a user nothing about the answer currently on their screen.\n\nSelf agreement tells you about *this* answer, not the average one, it costs nothing once you are sampling, it needs no extra model, and it works well enough to act on. The more valuable output of a system like this is not being right more often. It is knowing when to shut up.\n\nEvery score so far has handed the model the complete, correct schema for the exact database the question is about. Ask about students, get the student database. Nothing else on screen.\n\nNo real deployment works like that. A company's warehouse has hundreds or thousands of tables, nobody knows in advance which database a question belongs to, and the whole thing does not remotely fit in a model's context window. Something has to search the schema first, pick out the handful of tables that look relevant, and pass only those to the model. If that search picks wrong, the model never had a chance.\n\nThis section takes the gift away and measures what it was worth.\n\nMy first design was wrong, and the reason is worth a paragraph.\n\nSpider's databases are tiny. The median one has **4 tables and 19 columns**, and writing it into a prompt as `CREATE TABLE`\n\nstatements costs about 102 tokens. Searching for the right table among four is not a search problem. Any method scores near perfect and you learn nothing.\n\nThe obvious fix is to throw all 206 databases into one big pile and search that. This fails badly.\n\n```\n125 table names appear in more than one database\n436 of 1,053 tables affected  (41%)\n\ncustomers  → 22 databases      student → 12\naddresses  → 15                staff   → 11\n```\n\nThere are 22 different tables called `customers`\n\nin that pile, belonging to 22 unrelated databases. A question asking about customers genuinely does not say which one it means. **No search method can resolve that**, however good, because the information needed to resolve it is not in the question. A test built this way would measure an impossible task and then blame the retriever for failing it.\n\nSo I built a **collision free** pool instead. Take databases one at a time, in order of how many benchmark questions they carry, and skip any database whose table names clash with one already taken. That leaves:\n\nOf those 300 tables, only 107 belong to databases that carry any questions. The other **193 are pure noise**, sitting there to be wrong answers.\n\nAnd the pool genuinely does not fit. Written out as `CREATE TABLE`\n\nstatements it comes to **8,262 tokens, against the 3,072 token prompt limit** these models were evaluated under. Nearly three times over. Search is not an optimization here, it is the only way the prompt fits at all.\n\nOne thing to carry through the rest of this section: every number below is over **1,457 questions, not 2,147**. Putting a retrieval number next to a number from earlier in this post is comparing two different tests.\n\nEach table becomes one short searchable document, just its name and its column names:\n\n```\nstudent: id, name, age, dept_id\n```\n\nThe search query is **the question text and nothing else**. Never the database name. That is deliberate and important: if the retriever knew which database a question came from, it could just look up that database's tables directly, and every number here would be measuring an answer key rather than a search.\n\n**BM25** is keyword matching, the classic search engine approach. A table scores higher when more of the question's words appear in it, and rare words count for more than common ones. Matching the word \"hangar\" tells you a lot, matching \"the\" tells you nothing.\n\nOne detail carries a surprising amount of weight. Schema names are written like `city_code`\n\nor `singerID`\n\n, and people ask questions using words like \"city\" and \"singer\". A normal tokenizer treats `city_code`\n\nas one indivisible thing, so a question about cities can never match it. Mine splits on both `snake_case`\n\nand `camelCase`\n\nand keeps the joined form as well, so `city_code`\n\nbecomes `city_code`\n\n, `city`\n\nand `code`\n\n. Bridging that gap between English and identifiers is most of what this job actually is.\n\n**Dense retrieval** matches on meaning instead of words. A small embedding model turns each table description into a list of numbers, positioned so that texts meaning similar things sit near each other. Do the same to the question, and the nearest tables win. It can match \"how many people are enrolled\" to a `student`\n\ntable without either phrase sharing a single word with the other.\n\nBoth methods return the top **k** tables, where k is just how many you decide to hand the model. Here is how often each finds what the question needs.\n\n| k | BM25 recall | BM25 coverage | Dense recall | Dense coverage |\n|---|---|---|---|---|\n| 1 | 33.9% | 25.1% | 58.3% | 44.0% |\n| 5 | 63.9% | 53.8% | 88.1% | 79.2% |\n10 |\n73.0% |\n64.1% |\n92.6% |\n86.3% |\n| 20 | 78.6% | 70.7% | 95.1% | 90.3% |\n\nThe column that matters is **coverage**, not recall, and the difference between them is not a technicality.\n\nRecall is the share of needed tables that were found. Coverage is the share of questions where **every** needed table was found. Say a question needs to join two tables and the search finds one of them. Recall calls that 50%, a decent partial score. But the question is exactly as unanswerable as if the search had found nothing at all, because you cannot write half a join. Coverage scores it zero, which is the truth.\n\nDense wins clearly, which is not surprising given that questions are English and schemas are identifiers. BM25 is in here as the honest baseline, since it is free and needs no GPU, and if it had come close that would have been worth knowing.\n\nNow run the actual model on what each search method hands it. Four conditions, identical in every respect except which schema text goes into the prompt, all over the same 1,457 questions.\n\n| Condition | Schema shown | Accuracy | Queries that run |\n|---|---|---|---|\nOracle |\nexactly the tables the answer uses | 68.4% |\n92.1% |\n| Gold | the whole correct database | 63.5% | 89.7% |\n| Dense, top 10 | 10 tables out of 300 | 45.2% | 66.8% |\n| BM25, top 10 | 10 tables out of 300 | 37.1% | 57.5% |\n\nTwo findings, and I predicted neither.\n\n**Showing less schema helps, if it is the right schema**: Oracle beats gold by 4.9 points, and gold is only showing tables from the correct database. Every irrelevant table costs something, even a related one.\n\nThat is the exact opposite of what was true in training, where trimmed schemas were actively harmful because the model never learned to pick a table out of a crowd. Training on clutter is necessary. Having no clutter at question time is pure upside.\n\n**Most of the loss is distraction, not absence**: This is the one that surprised me. Split the dense results by whether the search actually found everything the question needed.\n\n```\nall needed tables present   1,257 questions (86.3%)   accuracy 51.9%\nat least one missing          200 questions (13.7%)   accuracy  3.0%\n```\n\nThe missing group behaves exactly as you would expect. 3.0% is effectively zero, and those questions were unanswerable the moment the search missed. Fine.\n\nBut look at the other group. **Every table the question needed was sitting right there in the prompt, and accuracy was still only 51.9% against oracle's 68.4%.** Sixteen and a half points destroyed purely by the nine irrelevant tables sitting next to the right ones. The share of queries the database will even run drops from 92.1% to 66.8%, because the model keeps reaching for a plausible looking table that belongs to some completely unrelated database.\n\nStacking those two costs together:\n\n```\noracle                             68.4%\n  minus distraction  16.5 points →  51.9%   right tables present, plus nine more\n  minus absence       6.7 points →  45.2%   13.7% of questions miss a table\n```\n\n**Distraction costs two and a half times what absence does.**\n\nI had predicted 58 to 61% before running this, by reasoning that accuracy would be roughly coverage multiplied by the baseline. In other words, I assumed that finding the tables was the whole problem, and that once found, the model would perform as it always had. The gap between that prediction and the real 45.2% is exactly the distraction cost, which my mental model of the system had no room for at all.\n\nIf distraction is the dominant cost, then handing the model fewer tables should reduce it. I wrote that prediction down before testing it. Six more runs, both search methods at three different values of k, say it is wrong.\n\n| Method | k | Coverage | Accuracy when covered | Accuracy when missing | Accuracy overall |\n|---|---|---|---|---|---|\n| Dense | 5 | 79.2% | 55.6% | 1.0% | 44.3% |\nDense |\n10 |\n86.3% |\n51.9% |\n3.0% |\n45.2% |\n| Dense | 20 | 90.3% | 49.4% | 2.1% | 44.8% |\n| BM25 | 5 | 53.8% | 63.6% | 2.2% | 35.3% |\n| BM25 | 10 | 64.1% | 57.1% | 1.3% | 37.1% |\n| BM25 | 20 | 70.7% | 54.1% | 2.6% | 39.0% |\n\nLook at the dense rows first. Overall accuracy reads 44.3, 45.2, 44.8 across a fourfold change in how many tables get retrieved. Flat enough that you would reasonably conclude k does not matter much.\n\nThat conclusion would be wrong, and the last two columns show why. **Two opposing forces are cancelling out.** Coverage climbs from 79% to 90%, because searching wider finds the right tables more often. Accuracy on questions that were already covered falls from 56% to 49%, because every extra table is another distraction. One goes up, the other goes down, and the total sits still.\n\nBM25 starts so starved of recall, only 54% coverage at k=5, that the coverage gain outweighs everything else and its overall number climbs from 35.3 to 39.0. Its covered accuracy falls on exactly the same schedule though, 64% to 57% to 54%.\n\n**Whether raising k appears to help depends entirely on whether your search method still had recall left to gain.** It never depends on the distraction going away, which got worse in every single condition tested.\n\nSo the honest reading is narrower than \"retrieve less\". The number of tables you retrieve is not a dial worth tuning. It trades coverage against distraction at roughly one for one, and the 16.5 point distraction penalty survives every setting I tried. What would actually move this is making the model itself robust to irrelevant tables, by training it on cluttered schemas instead of clean ones. That is a training change, not a search change.\n\nA project can report 68.1% accuracy and say nothing at all about whether an answer takes 200 milliseconds or 30 seconds. So the last piece is an HTTP service, and a benchmark of it.\n\nThe service **reuses the voting and retry code directly** rather than reimplementing it. That is the point. A second copy of the clustering or selection logic could drift away from the numbers those techniques were measured at, and nothing would tell you. A request returns the SQL, the rows, a confidence band from the agreement table, and a timing breakdown.\n\nMeasured on one A10G handling **one request at a time**, which is what a service does, unlike the evaluation harness that batches sixteen together. Over the full test split, so the accuracy column is real evidence rather than a spot check.\n\n| Mode | Accuracy | p50 | p95 | p99 | Mean | Queries/sec |\n|---|---|---|---|---|---|---|\n| Greedy | 67.9% | 2,652 ms | 5,100 ms | 7,162 ms | 3,088 ms | 0.32 |\nVote, 8 samples |\n71.3% |\n5,650 ms | 10,789 ms | 15,272 ms | 6,522 ms | 0.15 |\n| Retry, 3 rounds | 68.9% | 2,714 ms | 9,739 ms | 16,042 ms | 3,777 ms | 0.26 |\n\np50 is the median request. p95 is the slowest one in twenty. p99 is the slowest one in a hundred.\n\nThat accuracy column is doing quiet work. It reproduces the harness from a completely different code path with different batching, giving +3.4 for voting and +1.0 for retry against the harness's +3.5 and +1.1.\n\n**Retry's median is indistinguishable from plain greedy**, 2,714 ms against 2,652, because most requests never enter the loop at all. Every bit of its cost sits in the tail.\n\nIts mean of 3,777 ms hides that from both directions simultaneously. It is 40% above a median that describes most requests, and 76% below a p99 that describes the requests a user actually complains about. It describes neither.\n\nVoting is the opposite shape. It costs 2.1 times more at the median, but its p99 is only 2.7 times its own median, so what you measure in staging is what you get in production.\n\n**The mean says retry is cheaper. The tail says voting is more predictable.** Users experience the tail.\n\nThat last one is the whole system running. Three questions against a test database, answered by the service. The first two are correct and unanimous. The third asks for distinct types of planes owned by pilots, and returns four plausible plane names from the wrong table. Nothing in the SQL or the output reveals that. The only sign is one sample out of eight disagreeing, which drops the confidence badge from high to medium.\n\nAll of this is one `g5.xlarge`\n\nin us-east-1. A single NVIDIA A10G with 24GB, four vCPUs, $1.006 an hour on demand. The three serving benchmarks above took ~1.8, ~4 and ~2.5 hours respectively, which is most of a working day of GPU time spent purely on measuring something I had already built.\n\nThat ratio is the honest summary of the project. Far more compute went into finding out whether things worked than into making them work.\n\nFour pieces of infrastructure earned their place:\n\n**Automatic shutdown when idle**: A cron job checks GPU utilization every minute and shuts the box down after 30 minutes below 5%. A forgotten `g5.xlarge`\n\nis about $170 a week, and this is the single highest value cost control in the project.\n\nBut the first version measured the wrong thing. **GPU utilization alone is not \"is this machine in use\".** An evaluation run spends real minutes loading weights, building prompts, executing SQL against SQLite and scoring results, all at 0% GPU. That script shut down a live run twice before I fixed it. Idleness is now graded. If a process is still holding GPU memory, or a human is still logged in, the limit becomes 180 minutes instead of 30. Not immunity, because \"forgot to log out\" must not cost $170 a week either.\n\n**Rescuing work from spot instances**: Spot instances are 50 to 60% cheaper, and AWS can reclaim them with a two minute warning delivered through the instance metadata service. Reclaiming means terminate, not stop, so the disk goes with it. A service polls for that warning and syncs checkpoints to S3 when it arrives. Two minutes is enough to do one thing quickly and not enough to do anything clever.\n\nIt is not the primary mechanism though. A missed notice or a kernel panic gives no warning at all, so checkpoints also go to S3 on every save. Resuming is not restarting the same machine, it is a fresh machine pulling the last checkpoint from S3 and carrying on from that step, which I tested by killing a run and deleting its disk.\n\n*One thing to note is that I never actually ran on spot. AWS took forever to process my quota request and by the time it came through, most of the project was already done. So this is built and tested but not battle worn. It is a genuinely good feature and half the price is half the price, so if you use it, let me know how it holds up.*\n\n**Never mirror a checkpoint directory to S3.** The tidy way to stop S3 filling up with old checkpoints is `aws s3 sync --delete`\n\n, which mirrors the local directory. It is also one fresh instance away from deleting the only copy of a run, because a new machine's checkpoint directory is empty and mirroring an empty directory deletes everything. Old checkpoints are about 110MB each. Letting them pile up is far cheaper than the failure they prevent.\n\nAlso, I originally set everything up in us-east-2, because GPU instances ran about 30% cheaper there. Then I found out I could not launch an instance at all.\n\nAWS caps how many of each instance type you can run, per region, and those caps start at zero for GPU instances until you ask for an increase. Mine in us-east-2 was zero and stayed zero, so the cheaper price was for something I was not allowed to buy. Everything moved to us-east-1, where I already had approval for a few. Check your limit in a region before you plan around its pricing.\n\nAlso worth knowing: a stopped instance still charges you for its disk. Mine was about $16 a month for a 200GB volume doing nothing at all. If you are finished with a machine, terminate it rather than leaving it stopped.\n\n**Something trained to pick the answer**: At sixteen tries the model writes a correct query for 77% of questions and picks it only 72% of the time. Those 4.8 points are right answers it already produced and threw away. I tried two hand written rules for choosing better and they scored +0.1 and −0.1, so this needs a model trained to judge candidates against the question, not another rule.\n\n**Training on messy schemas**: The model learned on tidy, correct database descriptions, then gets handed cluttered ones full of irrelevant tables and loses 16.5 points to the clutter. That is fixable in training, by showing it messy schemas while it learns, rather than in the search step.\n\n**Faster serving**: 2.7 seconds for a typical question is slow, and the service currently handles one request at a time. Processing several at once on the GPU is the standard fix and the single biggest weakness in these numbers.\n\n**Not multi turn reinforcement learning**, which was my original plan. The idea was to train the model on the whole back and forth of the retry loop rather than on single answers. Two things killed it. Only 37% of the queries it repairs end up correct, so there is not much there to learn from. And at 1.5B, over half of training groups already teach the model nothing, because all its attempts agree with each other and it needs disagreement to learn. Training on longer conversations makes that worse.\n\nThe headline is that a 1.5B model matched a 7B one. That is true, and it is also the least interesting thing here.\n\nThe model did not get good because it got bigger. It went from 6.4% to 49.7% at half a billion parameters, before I touched the model size at all, and every one of those gains came from something outside the weights. A way to check answers by running them. A reward that had a gradient in it. Training data that did not hide the hard part. Then, once the model was as good as that size allowed, the same measuring tools were pointed at a bigger base and the gains transferred intact.\n\nAnd the last 3.4 points, the ones that actually drew level with the 7B, came from no training whatsoever. Sample eight answers, run all eight, keep whichever result most of them agreed on. That is a system decision, not a model decision.\n\nLook at what the pieces actually are, stripped of SQL.\n\nOnly the first is SQL specific, and only barely. Anywhere you can mechanically decide whether an output is correct, all six apply. Code that has to compile and pass tests. Structured extraction validated against a schema. Maths with a checker. An API call that either succeeds or does not. In each case you can build the same loop, and in each case a small model wrapped in it will go further than its parameter count suggests.\n\nThis is why the small model + good system approach works, and why so many people are converging on it right now. You are not trying to beat a frontier model at everything. You are picking one task, building the infrastructure that knows what correct looks like, and letting a cheap model take as many attempts as it needs.\n\nThe expensive part was never the model. It was building the thing that could tell me when I was wrong.\n\nThank you for reading my blog, this was fun to build, and I learnt a lot of things too. Happy to receive your thoughts in the comments! Connect with me on [LinkedIn](https://www.linkedin.com/in/pradhyumna-n-holla/) and [X](https://x.com/PradHolla).\n\n**The benchmark**\n\n**The techniques**\n\n**Tools**\n\n`GRPOConfig`\n\ndefaults carefully, since two of them silently changed the meaning of my reward.**AWS**\n\n[EC2 G5 Instances](https://aws.amazon.com/ec2/instance-types/g5/) — High performance GPU based instances for graphics intensive applications and ML inference\n\n[Spot instance interruption notices](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-interruptions.html) — the two minute warning, and how to poll for it from the instance metadata service. This is what the checkpoint rescue in this post is built on, and it is worth reading before you put a training run on spot rather than after.", "url": "https://wpnews.pro/news/how-to-build-a-tiny-1-5b-text-to-sql-model-that-beats-a-7b", "canonical_source": "https://dev.to/aws-builders/how-to-build-a-tiny-15b-text-to-sql-model-that-beats-a-7b-298", "published_at": "2026-09-04 07:10:20+00:00", "updated_at": "2026-09-04 07:24:20.820601+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-research", "ai-tools", "developer-tools"], "entities": ["Qwen2.5-0.5B", "Qwen2.5-Coder-1.5B", "Qwen2.5-Coder-7B-Instruct", "Spider", "LoRA", "GRPO", "NVIDIA A10G"], "alternates": {"html": "https://wpnews.pro/news/how-to-build-a-tiny-1-5b-text-to-sql-model-that-beats-a-7b", "markdown": "https://wpnews.pro/news/how-to-build-a-tiny-1-5b-text-to-sql-model-that-beats-a-7b.md", "text": "https://wpnews.pro/news/how-to-build-a-tiny-1-5b-text-to-sql-model-that-beats-a-7b.txt", "jsonld": "https://wpnews.pro/news/how-to-build-a-tiny-1-5b-text-to-sql-model-that-beats-a-7b.jsonld"}}