cd /news/ai-agents/how-do-you-reproduce-llm-agent-failu… · home › topics › ai-agents › article
[ARTICLE · art-150382] src=discuss.huggingface.co ↗ pub= topic=ai-agents verified=true sentiment=↑ positive

How do you reproduce LLM agent failures? I built an open-source Python tool for offline replay and pytest testing

Stepfork, an open-source Python tool for offline replay and pytest testing of LLM agent failures, is inviting contributions after a user shared fixtures and results showing how a test can catch implementation bugs while missing the original requirement. The tool currently relies on developer-defined expectations, and the maintainer said generated tests and authoritative test results should have separate trust boundaries. Interested contributors were asked to open a GitHub issue with fixtures and proposed improvements or submit a pull request.

read1 min views3 publishedOct 15, 2026

Thanks for sharing both fixtures and the actual results! The examples make the issues much clearer, especially how a test can catch implementation bugs while missing the original requirement.

The runner-owned verdict example is interesting too. I agree that generated tests and authoritative test results should have separate trust boundaries.

Stepfork currently relies on developer-defined expectations, but these examples highlight areas worth exploring further.

Would you be interested in contributing this to Stepfork? Feel free to open a GitHub issue with the fixtures and proposed improvements, or submit a PR if you’d like to work on the implementation. I’d be happy to review and collaborate.

Really appreciate the time you’ve put into investigating this!

── more in #ai-agents 4 stories · sorted by recency
── more on @stepfork 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/how-do-you-reproduce…] indexed:0 read:1min 2026-10-15 · —