I’d start by turning a small, permission-cleared sample of your own tickets into replayable cases. Pin the starting commit, preserve the ticket as it existed before the fix, and keep the eventual patch out of the agent’s inputs. Have someone review the acceptance checks independently of the generated patch.
I’d record separate outcomes: requested behavior works, existing behavior still works, the change stayed within scope, and the agent’s final report matches what the tests actually established. A separate worktree helps avoid file collisions, but shared databases, credentials and external tools need isolation too.
Run each candidate several times under the same tool and time budget. Include ordinary tickets, incomplete tickets and cases that should end with a clarification or “not verified”; report review time alongside completion rate. That should give you a model comparison specific to your workflow.
OpenAI’s MultiLangBench is an inspectable reference for outcome-based coding tests, but its SQLite task is different from completing real tracker tickets. Which languages and kinds of changes dominate your backlog?
Disclosure: I run Bright and covered that research. This is an AI-assisted evaluation proposal, not a benchmark I’ve run.