I built an AI tutor for SQL that won't hand you the answer straight away. Then I tested it against a normal version that does. Six people practised with one version or the other. A week later, they took a 12-question SQL test with no tutor at all. The "try first" group got all of their answers right
I built an AI tutor for SQL that won't hand you the answer straight away. Then I tested it against a normal version that does. Six people practised with one version or the other. A week later, they took a 12-question SQL test with no tutor at all. The "try first" group got all of their answers right: 36 of 36 (3 people × 12 questions). The "give me the answer" group got 25 of 36. That looks like a clear win. But when I presented it at my course's Demo Day, I told the room I don't credit the win to my tutor. This post explains why. It also covers the one question from a judge that I wish I had answered on my slides. This was my final project for the 100x Engineers AI course. I wanted to answer one design question every AI tutor faces: When a learner is stuck and presses Help, what should the tutor give them? Most tools give the answer. ChatGPT does. Copilot does. It feels helpful, and the learner moves on. But a large study (Bastani and colleagues, PNAS 2025) should worry anyone building these tools. Students who practised maths with an AI that gave answers did worse on a later exam than students who had no AI at all. Once the AI was gone, they were worse off. So I asked a narrower question: If the tutor makes you try first, and gives a hint before the answer, can you do more a week later, once the tutor is gone? An SQL tutor on one web page. You read a question in plain English, write a SQL query, and press Run. The page tells you if you're right. When you're stuck, you press Help. What Help does is the whole experiment. I built two versions of the page. In experiment-speak, each version is called an arm, so you'll see "Arm A" and "Arm B" below. when you press Help Arm A: try first Arm B: give the answer before you've tried anything refuses: "give it one run first" shows the answer after one attempt a hint, written by an LLM shows the answer after two more attempts shows the answer shows the answer Arm B is the baseline: it behaves like most AI tools do today, so it's what I compare Arm A against. Everything else was identical: the same page, the same 16 practice questions, the same checking. One setting was the only difference. A judge at Demo Day called the experiment "arm wrestling". The name stuck, so it's the name of this series. Six people took part, mostly developer colleagues. Three used Arm A and three used Arm B. I split them so both groups had similar SQL skill, based on two earlier rounds of testing. Practice. 16 SQL questions on GROUP BY and HAVING, with the tutor. Wait 5 to 7 days. Final test. 12 new questions on the same skills. No Help button for anyone. Step 3 is the point. I didn't care how well people did with the tutor. I cared what they could still do once it was gone. Before I saw any results, I wrote down how I would read them. That included ten checks that, if they failed, would mean the result couldn't be trusted. More on why that mattered below. Each group had 3 people answering 12 questions, so 36 answers per group. out of 36 answers Arm A: try first Arm B: give the answer correct, after any number of tries 36 25 correct on the first try 17 17 questions they at least opened 36 29 wrong first tries they went on to fix 19 of 19 8 of 12 Look at the second row. On the first try, the two groups were identical: 17 each. So the tutor didn't make anyone better at writing a correct query from memory. The whole gap came from what people did after a wrong answer: Arm A opened every question. Arm B left 7 blank. Arm A fixed all 19 of its wrong first tries. Arm B fixed 8 of 12. That's not what I predicted. I expected people to remember more. What I saw was people who kept going and fixed their mistakes. This is the part I'm proudest of, even though it looks like a loss. Two of my ten checks failed. The groups didn't behave as differently as planned. Arm A's whole idea is "try before you get help". That only matters if Arm B people don't try first. But they did. On average, Arm B people made almost one attempt (0.83) before pressing Help anyway. So Arm A didn't create a habit that Arm B lacked. The test was too easy for Arm A. Arm A scored 100%. When a group hits the top score, you can't tell how far ahead it really is. Maybe it was just ahead by a little. Then two things skewed the result that I hadn't planned for. Question order. The 12 questions ran from easiest skill to hardest. Three people stopped early, so they all missed the hardest questions. My biggest gap between the groups, 6 answers on the hardest skill, was really just 6 blank answers. Not everyone used Help. One Arm A person never pressed Help during practice. So he never experienced the "try first" tutor at all, and he still scored 12 of 12. If I count only the four people who actually used Help, the gap shrinks from 11 answers to 3. Put together: the result leans toward "try first". But I can't say "try first" caused it. So the claim I can defend is narrower: Making learners try first seems to change what they do after a wrong answer. It did not change what they could get right on the first try. At Demo Day, a judge asked: "Were the practice and the test done live, with you watching?" They weren't. I sent each person a link, and they did it in their own time. He said that wasn't on the slides, and he had assumed it was live. He was right to ask. It changes how you read the result. When someone works unsupervised, you can't see what else they used. A ChatGPT tab is one Cmd+Tab away. And it doesn't affect both groups equally. Think about who has a reason to leave the page: Arm B people have no reason. The answer is one click away. Arm A people are blocked until they make an attempt. That's exactly when you'd paste the question into ChatGPT. If Arm A people did that, they skipped the struggle the tutor was built to create. That makes Arm A's tutor less effective than it should be, and shrinks the gap between the groups. The final test was unsupervised too. If people looked things up there, both groups score higher, and the gap shrinks again. So working unsupervised probably made my result look weaker than it really was, not stronger. That's a good story. I just didn't tell it, because it wasn't on the slide. The lesson: if a detail about how you ran the test would change how someone reads the result, put it on the slide. If you leave it out, people assume the standard version, and that may not be what you did. Shuffle the question order for each person. Then stopping early doesn't always hit the hardest questions. Use harder test questions, so neither group can score 100%. Make "correct on the first try" the main score. It's the number my new prediction says won't change. That makes it the number that could prove me wrong. Track who actually used the feature. Someone who never presses Help never experienced the tutor. Run the test live, or at least log when people switch tabs or paste. A big gap can come from somewhere you didn't look. 36 vs 25 looked like better memory. The first-try row showed it wasn't. Write your checks before you see the results. That's the only reason I could say "two checks failed" instead of explaining them away afterwards. Changing your prediction isn't failing. Mine moved from "people remember more" to "people recover better". That change is the most useful thing the project produced. Say how you ran the test. Live or unsupervised, question order, who actually used the feature. Readers fill in any gap with their own assumption. Next in this series: how I built the tutor so that the LLM never decides whether to help. Plain code makes that decision, and the LLM only writes the hint. That one rule is what made the experiment possible to check. The app is live, and you can try both versions: Arm A, try first: https://tranquil-starlight-f0129e.netlify.app/index.html?arm=A Arm B, give the answer: https://tranquil-starlight-f0129e.netlify.app/index.html?arm=B / withholding-pilot Learning OS — 100x Cohort 7 final capstone Vikrant Negi. Solo. Demo Day track. Submission 7 Oct 2026. An SQL tutor that makes you try before it helps, and a 6-person pilot testing whether that changes what you can do a week later, with the tutor gone. For where things stand today, read STATUS.md. That is the only file that claims to be current. Try it Live app: https://tranquil-starlight-f0129e.netlify.app/ Withholding tutor: https://tranquil-starlight-f0129e.netlify.app/index.html?arm=A Answer-giving baseline: https://tranquil-starlight-f0129e.netlify.app/index.html?arm=B What each one does: ?arm=A opens the withholding tutor. Press Help before trying, and it refuses. Try once, and Help gives a hint. Two more tries, and it gives the answer. ?arm=B opens the baseline. Help gives the full answer at once. Same page, same 16 practice questions, same grader. One config flag is the only difference. The result Arm A scored 36 of 36 person-questions (3 learners x 12 held-out questions). Arm… View on GitHub If you've built anything that decides how much an AI should help, I'd love to hear how you measured it. Please feel free to leave any feedback.
Key Takeaways #
- •I built an AI tutor for SQL that won't hand you the answer straight away
- •This story was reported by Dev.to , covering developments in thedev space.
- •AI advancements continue to reshape industries — read the full article on Dev.to for complete coverage.
📖 Continue reading the full article: