{"slug": "does-ai-tutoring-help-students-learn-without-ai", "title": "Does AI Tutoring Help Students Learn Without AI?", "summary": "A November 14 preprint by Jun Li and colleagues reports that instructor-designed AI activities raised exam scores by an estimated 0.32 standard deviations (95% confidence interval 0.06 to 0.59) in supervised, in-person exams taken without AI, based on 4,768 students across 44 class-terms at 19 universities. The randomized trial assigned classes to start AI activities immediately or wait a term, and found no clear change in engagement or satisfaction measures. The study excluded 10 class-terms where researchers could not verify assessments were taken without AI, a rule added after preregistration.", "body_md": "In a Python lab, one of the tools in a recent study had a narrow job. It was there to explain errors and help students follow the logic of their code. The graded code was theirs to write. That’s a small design choice with a big effect on what the student gets to practice. If the bot writes the fix, the student may get a working program without ever finding out where their reasoning went wrong. [\\[1\\] Classroom examples from the university trial](https://arxiv.org/pdf/2611.19099v1#page=23)\n\nHow would you tell if that kind of help worked? You could look at how many students finished the lab. You could ask how useful they found the tool. Jun Li and colleagues looked at something else: how students did on exams where they couldn’t use AI. Their November 14 preprint reports higher scores in classes that brought in AI activities designed by their own instructors. The study covered 7,200 students at 19 universities, though the exam analysis used a smaller sample. [\\[1\\] Li and colleagues’ preprint](https://arxiv.org/html/2611.19099v1)\n\nThe instructors submitted a plan before they knew when they’d get to use it. One tool made practice exams that put familiar concepts in new settings. Another played a client who asked students to explain their modeling choices. The research team helped instructors put their ideas into practice. [\\[1\\] Study methods and examples](https://arxiv.org/html/2611.19099v1)\n\nThink about what the practice exam changes. A worksheet headed “Linear Regression” has already made one decision for the student. Put the same concept inside an unfamiliar problem, and the student has to spot when it applies. That’s part of the skill. The AI can help supply more examples, while the instructor decides what students should have to figure out for themselves.\n\nThe researchers randomly assigned classes to start their AI activities right away or wait a term. They looked at how exam scores in each group changed from earlier terms. That matters because a class that usually gets high marks might do well again for reasons that have little to do with the new tool. The comparison asks how much each group moved from where it started. [\\[1\\] Trial design](https://arxiv.org/html/2611.19099v1)\n\nThe exam analysis covered 4,768 students enrolled across 44 class-terms during the trial. The exams were supervised and held in person. Most were on paper. The researchers left out 10 class-terms because they couldn’t verify that students had taken the assessments without AI. They added that rule after preregistration, so it’s worth knowing when you read the result. [\\[1\\] Exam criteria and analysis sample](https://arxiv.org/html/2611.19099v1)\n\nThe estimated gain was **0.32 standard deviations**, with a 95% confidence interval of 0.06 to 0.59. In plain terms, the researchers expressed the gain relative to how much scores usually varied within each class. That gave them a common scale across courses with different exams. The study found no clear change in the measures it used for engagement or satisfaction. [\\[1\\] Exam results and uncertainty](https://arxiv.org/pdf/2611.19099v1#page=25)\n\nThe average covers several kinds of AI help. A department looking at that number still has to decide where to spend its time. Would students benefit more from extra practice problems or from a simulated client who pushes back on their analysis? The trial was too small to settle that question for individual tools. It also leaves open how long the gains last. [\\[1\\] Study limitations](https://arxiv.org/html/2611.19099v1)\n\nIn a 2025 study at a high school in Turkey, students used two versions of GPT-4 during math practice. One worked much like a general chatbot. The other was set up to give hints drawn from teachers’ guidance. Both groups did better on practice problems than students working without AI. Then came the exam. The general chatbot group scored lower than the group that had practiced without AI. For the group that got hints, the researchers found no clear difference from the group without AI. [\\[2\\] Bastani and colleagues’ high-school math trial](https://hamsabastani.github.io/education_llm.pdf)\n\nA University of Maryland trial ran into a more basic issue. The study assistant could draw on course materials and was available right inside the course site. About 15% of the students offered it used it even once. In the comparison between sections of the same courses, the sections offered the tool had lower final grades and less activity on the course site. Students in both groups could use other AI tools. That makes the result easier to read as evidence about offering this assistant than about the effect of using it for tutoring. [\\[3\\] Liu and colleagues’ university working paper](https://docs.iza.org/dp18989.pdf)\n\nThose two studies point to different things to check. If hardly anyone uses a tutor, find out whether it fits into the way students already study. If they use it and finish more exercises, give them a fresh problem on their own. The extra check matters even when the practice session goes well.\n\nGo back to the Python lab. Suppose the skill you want to teach is tracing a bug through a loop. During practice, the tutor could ask what value the student expects on the next pass through the loop. If the prediction is wrong, it could suggest a line to inspect. The student still has to work out what the program does.\n\nNow write the question they’ll answer without the tutor. Give them a short piece of code they haven’t seen, with the same kind of bug. Ask where the program first goes wrong and how they know. A student who can show you that has learned something you’d never see in a count of completed labs.\n\n[Does AI Tutoring Help Students Learn Without AI?](https://pub.towardsai.net/does-ai-tutoring-help-students-learn-without-ai-7ea9c9b8523d) was originally published in [Towards AI](https://pub.towardsai.net) on Medium, where people are continuing the conversation by highlighting and responding to this story.", "url": "https://wpnews.pro/news/does-ai-tutoring-help-students-learn-without-ai", "canonical_source": "https://pub.towardsai.net/does-ai-tutoring-help-students-learn-without-ai-7ea9c9b8523d?source=rss----98111c9905da---4", "published_at": "2026-11-19 12:01:02+00:00", "updated_at": "2026-11-19 12:42:06.827187+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-research"], "entities": ["Jun Li", "GPT-4", "arXiv"], "also_reported_by": [], "takeaway": "Instructor-designed AI activities that keep graded work in students' hands produced a 0.32 standard-deviation gain on supervised, AI-free exams, with no measurable change in engagement or satisfaction.", "recommendation": {"advice": "Measure AI tutoring by supervised, AI-free exam scores against a randomized wait-term control rather than completion or satisfaction, and exclude class-terms where AI-free assessment can't be verified.", "playbook": "evals-testing", "playbook_url": "https://wpnews.pro/playbooks/evals-testing"}, "alternates": {"html": "https://wpnews.pro/news/does-ai-tutoring-help-students-learn-without-ai", "markdown": "https://wpnews.pro/news/does-ai-tutoring-help-students-learn-without-ai.md", "text": "https://wpnews.pro/news/does-ai-tutoring-help-students-learn-without-ai.txt", "jsonld": "https://wpnews.pro/news/does-ai-tutoring-help-students-learn-without-ai.jsonld"}}