{"slug": "how-do-you-reproduce-llm-agent-failures-i-built-an-open-source-python-tool-for", "title": "How do you reproduce LLM agent failures? I built an open-source Python tool for offline replay and pytest testing", "summary": "Stepfork, an open-source Python tool for offline replay and pytest testing of LLM agent failures, is inviting contributions after a user shared fixtures and results showing how a test can catch implementation bugs while missing the original requirement. The tool currently relies on developer-defined expectations, and the maintainer said generated tests and authoritative test results should have separate trust boundaries. Interested contributors were asked to open a GitHub issue with fixtures and proposed improvements or submit a pull request.", "body_md": "Thanks for sharing both fixtures and the actual results! The examples make the issues much clearer, especially how a test can catch implementation bugs while missing the original requirement.\n\nThe runner-owned verdict example is interesting too. I agree that generated tests and authoritative test results should have separate trust boundaries.\n\nStepfork currently relies on developer-defined expectations, but these examples highlight areas worth exploring further.\n\nWould you be interested in contributing this to Stepfork? Feel free to open a GitHub issue with the fixtures and proposed improvements, or submit a PR if you’d like to work on the implementation. I’d be happy to review and collaborate.\n\nReally appreciate the time you’ve put into investigating this!", "url": "https://wpnews.pro/news/how-do-you-reproduce-llm-agent-failures-i-built-an-open-source-python-tool-for", "canonical_source": "https://discuss.huggingface.co/t/how-do-you-reproduce-llm-agent-failures-i-built-an-open-source-python-tool-for-offline-replay-and-pytest-testing/190525#post_7", "published_at": "2026-10-15 01:08:55+00:00", "updated_at": "2026-10-15 05:52:09.156555+00:00", "lang": "en", "topics": ["ai-agents", "developer-tools", "ai-tools", "mlops"], "entities": ["Stepfork", "GitHub", "pytest", "Python"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/how-do-you-reproduce-llm-agent-failures-i-built-an-open-source-python-tool-for", "markdown": "https://wpnews.pro/news/how-do-you-reproduce-llm-agent-failures-i-built-an-open-source-python-tool-for.md", "text": "https://wpnews.pro/news/how-do-you-reproduce-llm-agent-failures-i-built-an-open-source-python-tool-for.txt", "jsonld": "https://wpnews.pro/news/how-do-you-reproduce-llm-agent-failures-i-built-an-open-source-python-tool-for.jsonld"}}