The Error Message Didn't Matter | barkup-bench Study AJ

A study from barkup-bench found that returning structured error messages to AI models for patch correction is unnecessary for model performance. Testing showed models achieve identical recovery rates whether given full issues, bare codes, or a simple failure notice, indicating the feedback is for developer experience rather than model improvement.

For thirty-five studies, one mechanism rode along in every arm of our benchmark without ever being tested: when a model's patch fails validation, we return the structured issues verbatim for a correction round. It felt obviously load-bearing. Study AJ finally isolated it, by seeding known failures and varying only the feedback text: the full structured issues, bare issue codes, or nothing but 'the anchored patch was invalid.' The result is parity on all three models. Opus recovered 45 of 45 in every arm, including from the bare sentence. Gemini scored an identical 42 of 45 in every arm, missing the same three cells each time no matter what we told it. Told a patch failed, models simply re-derive the correct edit from the task and the tree. The verbatim-issues commitment survives with its rationale corrected: it is developer UX, not model recovery. The error message was for us.