How do you reproduce LLM agent failures? I built an open-source Python tool for offline replay and pytest testing

Stepfork, an open-source Python tool for offline replay and pytest testing of LLM agent failures, is inviting contributions after a user shared fixtures and results showing how a test can catch implementation bugs while missing the original requirement. The tool currently relies on developer-defined expectations, and the maintainer said generated tests and authoritative test results should have separate trust boundaries. Interested contributors were asked to open a GitHub issue with fixtures and proposed improvements or submit a pull request.

Thanks for sharing both fixtures and the actual results The examples make the issues much clearer, especially how a test can catch implementation bugs while missing the original requirement. The runner-owned verdict example is interesting too. I agree that generated tests and authoritative test results should have separate trust boundaries. Stepfork currently relies on developer-defined expectations, but these examples highlight areas worth exploring further. Would you be interested in contributing this to Stepfork? Feel free to open a GitHub issue with the fixtures and proposed improvements, or submit a PR if you’d like to work on the implementation. I’d be happy to review and collaborate. Really appreciate the time you’ve put into investigating this