cd /news/large-language-models/i-showed-my-coding-agent-its-own-bug… · home › topics › large-language-models › article
[ARTICLE · art-171447] src=dev.to ↗ pub= topic=large-language-models verified=true sentiment=· neutral

I Showed My Coding Agent Its Own Bug. It Copied the Bug Twice.

A developer testing llama3.1:8b locally through Ollama found that feeding a coding agent its own broken code and failing test cases produced worse results than a blind retry. In one case the error-aware retry returned the identical broken next-permutation function twice, character for character, while a blind retry with no diagnostic information fixed the bug on the first attempt. The experiment ran 12 algorithmic tasks with a shared baseline and up to two retries across no-retry, blind-retry and error-aware-retry conditions at $0 cost.

recommendation

Test error-aware retry prompts against blind retries on your own model before assuming failure feedback helps; here blind retry beat it.

more in playbook: Evals and testing for LLM apps
by read7 min views6 publishedNov 18, 2026

My local model wrote a next-permutation function that failed 3 of 5 test cases. I showed it the broken code, the exact inputs it got wrong, and what it returned versus what was expected, twice. Both times it sent back the identical broken function, character for character. A separate run where I gave the same model zero information beyond "that didn't work, try again" fixed the bug completely on the first attempt.

I wrote this with AI assistance, and I fact-checked and edited every claim myself.

That result is the opposite of what I expected going in. The obvious assumption behind every auto-retry loop I have built is that showing the model its own failure is strictly better than a blind retry, since more information should never hurt. On this run, with this model, it did not just fail to help. It produced a worse outcome than giving the model nothing at all.

The setup, in one line: llama3.1:8b run locally through Ollama, 12 algorithmic coding tasks, one shared baseline attempt per task, then up to 2 extra attempts under each of 3 conditions (no retry, blind retry, error-aware retry). $0 total cost, no paid API calls. Full raw results, grading harness, and reference implementations: linked at the end.

The question this raises, stated as something you could go test yourself: if a cheap retry loop has a fixed attempt budget and no independent verification step, does showing the agent its own error ever reliably beat a blind retry, or is the real value of "self-correction" prompts concentrated in cases the first retry would have fixed anyway?

Each of the 12 tasks got exactly one baseline attempt, generated with llama3.1:8b at Ollama's default temperature of 0.7, zero system prompt, one natural-language prompt describing the function to write. If the baseline passed every test, all three conditions were recorded as that single result and no retries ran, which is what happened on 9 of the 12 tasks. If the baseline failed at least one test, the same baseline branched into three independently tracked conditions:

Sharing one baseline across all three conditions, instead of generating three independent first attempts, holds the starting point constant. A difference in final score is then attributable to the retry condition, not to ordinary attempt-to-attempt variance in a fresh first try.

FOLLOWUP_TEMPLATE = """Your previous solution had a bug. Here is your code:

{previous_code}

It failed these cases:
{failure_detail}

Write a corrected version of the function."""

That template, plus the specific failing inputs and actual-versus-expected outputs, is the entire difference between the blind and error-aware conditions. Everything else, including the model, temperature, and task prompt, is identical.

The 12 tasks are classic algorithmic problems: next permutation, Roman numeral conversion, a decode-ways dynamic-programming count, valid parenthesis strings with wildcards, spiral matrix traversal, and others, each with 4 to 6 test inputs chosen to include at least one edge case. An earlier, easier task set of ordinary list and string utilities produced a ceiling effect: 11 of 12 solved perfectly on the first attempt, leaving nothing to analyze. That set was discarded, not reported as a null result, the same lesson I had already documented from an earlier LLM-judge experiment in this series.

Grading pulled the first fenced code block out of each response, ran it in an isolated python3 -I subprocess with a 15-second timeout, and compared the output against a hand-verified reference implementation for each test input. A pass required an exact match or the correct exception type.

3 of 12 tasks failed the baseline and produced real retry data. The other 9 passed baseline perfectly and are not analyzed further; the three conditions are identical by construction when there is nothing to retry.

Task Baseline Blind retry path Blind final Error-aware path Error-aware final
next_greater_permutation 2/5 2 → 5 5/5 2 → 2 → 2 2/5
valid_parenthesis_with_star 4/6 4 → 3 → 4 4/6 4 → 4 → 5 5/6
decode_ways 0/6 0 → 3 → 2 2/6 0 → 2 → 2 2/6

On next_greater_permutation, the baseline bug was a one-line slicing mistake that reversed the wrong part of the list. Blind retry, given nothing but "try again," produced a different implementation on its first extra attempt and fixed the bug completely. Error-aware retry, shown the exact failing cases and the specific wrong output for each, returned the same buggy function on attempt two, then the same buggy function again on attempt three. I diffed the three responses directly: the error-aware attempts are character-for-character identical to the baseline.

On decode_ways, the baseline crashed every single test with an IndexError, an off-by-one after prepending a sentinel character to the input string. Blind retry's second attempt rewrote the whole approach as a memoized recursion and reached 3 of 6, its best result in the sequence, then its third attempt abandoned that approach for a third, more broken one and dropped to 2 of 6. More attempts did not mean more correct, even within a single condition. Error-aware retry patched the crash on its second attempt but shifted the same off-by-one into a silent wrong-answer bug instead of a crash, then made a cosmetic, functionally identical edit on its third attempt. Final score: also 2 of 6, tied with blind retry, for a different reason.

valid_parenthesis_with_star is the one case where error-aware retry won outright: final score 5 of 6 versus blind retry's 4 of 6, the only task in this run where seeing the actual failing test genuinely moved the needle. Blind retry's own path shows a regression below baseline on its second attempt (4 → 3) before recovering to a tie with baseline on its third, never beating it.

Taken together, across the 3 tasks that actually failed, error-aware retry's final scores were worse than or equal to blind retry's final scores on 2 of 3, and better on 1 of 3. Neither strategy dominates the other.

The failure that stands out is not that error-aware retry sometimes loses. It is that on next_greater_permutation, a loop with a fixed retry budget and no independent check would have shipped the identical broken function twice in a row and called it done, because nothing in the setup checks whether a "corrected" attempt actually changed anything before accepting it. Telling an agent exactly what it got wrong is not the same as the agent using that information. In this run, the model treated the error detail as something to acknowledge rather than something to act on, and produced the same answer back.

The decode_ways result is the other half of the lesson: retry is not a monotonic climb toward correctness even when you do give the model room to try again. A third attempt that abandons a partially-working second attempt for something worse is a realistic failure mode of any "keep trying until the budget runs out" loop, not an edge case.

My own practical change from this: a retry loop I would actually trust needs a cheap verification step after each attempt, comparing the new output against the previous one, not just against a pass/fail threshold, so a loop can tell the difference between "tried something different and it helped," "tried something different and it didn't," and "returned the exact same thing." None of the three conditions here had that check, and it is the first thing I would add before using error-aware retry in anything real.

This is one model, llama3.1:8b, on one set of 12 tasks, with one specific error-aware-retry prompt template, run once each with no repeated trials per condition. It is not a benchmark of this model's general coding ability and it does not claim to generalize to other models, other task difficulties, or other ways of phrasing a retry prompt. All 12 tasks are reported, including the 9 that produced no signal, and all 3 failing tasks' full attempt sequences are reported, including the 2 of 3 where error-aware retry did not win.

Reproducibility: full tasks, reference solutions, grading harness, raw results, failure writeups, and the chart above, at github.com/priyansh18/medium-articles/tree/d74f0b7/retry-made-it-worse.

If you run a retry loop with a fixed attempt budget and no verification step between attempts, has it ever actually caught a case where the "corrected" output was identical to the broken one, or is that failure mode just invisible until someone goes and diffs the responses?

── more in #large-language-models 4 stories · sorted by recency
── more on @ollama 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/i-showed-my-coding-a…] indexed:0 read:7min 2026-11-18 · —