# Devs Learning With AI Scored 17 Points Lower in Anthropic's Trial. Juniors Code Solo First

> Source: <https://dev.to/gabrielanhaia/devs-learning-with-ai-scored-17-points-lower-in-anthropics-trial-juniors-code-solo-first-o66>
> Published: 2026-10-25 13:01:27+00:00

Give a developer an AI assistant while they learn a new library, and they finish about as fast as someone without one. Then quiz them on that library. In [Anthropic's randomized trial](https://www.anthropic.com/research/AI-assistance-coding-skills), the AI group averaged 50%. The group that coded by hand averaged 67%.

That is a 17-percentage-point gap, and the biggest piece of it was on debugging questions. The speed difference was about two minutes and not statistically significant.

My take: if your team hands new hires a coding agent on day one, you are skipping the weeks where they learn to debug. Juniors should write the first version of unfamiliar code themselves, use the agent to explain things, and earn delegation later.

The paper is [How AI Impacts Skill Formation](https://arxiv.org/abs/2601.20245) by Judy Hanwen Shen and Alex Tamkin, posted to arXiv on January 28, 2026. [Anthropic's write-up](https://www.anthropic.com/research/AI-assistance-coding-skills) describes the 52 participants as "mostly junior" software engineers. Everyone had used Python weekly for over a year, had tried AI coding assistants, and had never used [Trio](https://trio.readthedocs.io/), a Python library for async concurrency.

Each person got a short warm-up, then 35 minutes to build two Trio features: a timer that prints every second while other work runs, and a record fetcher that handles missing records. Half could use an AI assistant during the Trio task. Half could not. Then everyone took a quiz: 14 questions, 27 points, covering seven Trio concepts through debugging, code reading and conceptual questions ([paper, v2](https://arxiv.org/html/2601.20245v2)).

The results, from the paper and Anthropic's post:

|  | AI group | Hand-coding group | 
|---|---|---|
| Quiz average | 50% | 67% | 
| Task time | about 23 min | about 25 min | 
| Time difference | not significant (p = 0.391) |  | 
| Quiz difference | 4.15 of 27 quiz points (about 17% of the quiz), Cohen's d = 0.738, p = 0.010 |  | 

The [paper](https://arxiv.org/html/2601.20245v2) says the largest score gap was on debugging questions and the smallest on code reading. It also counts errors. The median hand coder hit 3 errors during the task, the median AI user hit 1. Of the 12 people who finished both tasks with zero errors, only two coded by hand.

That is the part I keep coming back to. The AI group had a smoother hour and learned less from it. The authors suspect the errors were doing the teaching.

The hand coders ran into Trio-specific mistakes the paper names: a `RuntimeWarning` for a coroutine that was never awaited, and a `TypeError` from passing a coroutine where Trio expected a function. Here is the first kind, in the shape of the study's timer task:

``` python
import trio

async def ticker():
    for i in range(3):
        print("tick", i)
        trio.sleep(1)  # bug: missing await

async def main():
    async with trio.open_nursery() as nursery:
        nursery.start_soon(ticker)

trio.run(main)
```

Without the `await`, `trio.sleep(1)` only creates a coroutine object and never runs it. The ticks print back to back, and Python warns that the coroutine was never awaited. Figuring out why teaches you what a coroutine is. Pasting the warning into a chat window teaches you how to paste.

The paper backs that up with its six interaction patterns. [Anthropic's post](https://www.anthropic.com/research/AI-assistance-coding-skills) gives approximate averages for each:

The lowest score in the table belongs to the people who used the AI to debug. The conceptual-inquiry group was the second fastest pattern overall, after AI delegation, and kept most of the learning. Asking "what does a nursery do" cost them almost nothing in time.

*Let the agent answer questions. Let the human find the bug.*

Read this before you quote the 17 points in a planning meeting.

The caveats narrow the take: the risk is highest when someone is new to the thing they are building, and a junior is new to almost everything.

Three stages, per area of the codebase, not per calendar month:

If your team uses Claude Code, its built-in [Learning output style](https://code.claude.com/docs/en/output-styles) is a decent fit for stage 2. Claude handles routine code, explains its choices in `Insight` blocks, and leaves the design-decision pieces for you to write at a `TODO(human)` marker, then waits. Switch with `/output-style learning`, or set it in `.claude/settings.local.json`:

```
{
  "outputStyle": "Learning"
}
```

The docs say the value is case-sensitive, so `learning` in the file falls back to the default style. The docs also say Claude treats an output style as an instruction it may not always follow, so it won't enforce stage 1 for you.

I build Hermes IDE, so weigh this section with that in mind. Hermes runs each agent in its own real terminal, and plain shells open the same way. A junior can keep a shell for the solo version and an agent session for questions in the same window.

*Hermes 2.3.4: the bar above a Claude Code session, with the Commands list open.*

Since [2.3.4](https://github.com/hermes-hq/hermes-ide/releases), the bar above each session puts Model, Plan and Review in the same place for every agent, and the Commands list includes your own commands and skills. A Claude Code command like `/output-style` still belongs to Claude Code. Hermes just runs the agent's own interface.

For stage 3, review matters more than the tool. Hermes saves every agent turn, and the Review Desk (⌘G, Ctrl+Shift+G on Windows and Linux) shows the changes by turn. A mentor can point at one turn and ask the junior to explain it, the same follow-up step the high scorers did on their own.

*Hermes 2.3.4: the Review Desk by turn, with turn 1's diff and a commit message drafted from the turns.*

Hermes won't stop anyone from delegating on day one. Your team has to make that rule. It is source-available under BSL 1.1 and runs on macOS, Windows and Linux. [Download it here](https://hermes-ide.com/download) or read the code on [GitHub](https://github.com/hermes-hq/hermes-ide). Hermes IDE is not affiliated with Nous Research or its Hermes Agent.

[Anthropic's write-up](https://www.anthropic.com/research/AI-assistance-coding-skills) ends on the uncomfortable line: "Cognitive effort—and even getting painfully stuck—is likely important for fostering mastery." Teams are buying agents to remove exactly that.

Do you let new hires use agents from day one, or do they write the first version alone? And if you have watched someone learn to debug with an agent beside them, did it stick? Tell me in the comments.

Interviews still grade the skill this study says AI can erode: reasoning through a problem yourself, out loud, without a tool to fall back on. My *System Design Pocket Guide: Interviews* walks through 15 real designs step by step. Use it the way the study's best group used the AI: sketch your own design first, then read mine and argue with it.