We have a weekly CI task that finds DocSpring tests that failed and then passed on retry. I give the report to a coding agent, and it investigates the failures while I work on other things.
I posted about this on X recently because I used to struggle to make time for this work. You can spend a long time investigating a failure that disappears as soon as you try it locally. Now I can ask an agent to work through the CI logs and follow the failure into the code.
Some of those investigations have found real product bugs: generated cURL commands that broke on an apostrophe, sample data outside its own validation rules, and temporary files that could disappear before our PDF renderer read them. A passing retry had made each failure look less urgent.
The reporting runs automatically. I start each fix round and approve the push after review. The changes then go through our usual CI, QA, and release process.
Keep the failures that a retry hides #
Our Rails suite uses rspec-retry. Ordinary examples get one retry in CI, and JavaScript browser examples get two. That can keep an intermittent failure from blocking unrelated work, but the final green result leaves out some useful history.
We already printed a summary of retried examples. Our CI configuration also deleted application logs and diagnostic artifacts after successful jobs. To investigate recurring failures, we needed a small record that survived a successful run and could be collected later.
At the end of the suite, we now write a JSON file for each test node that had retries. Here is a synthetic entry:
{
"id": "./spec/features/report_spec.rb[1:2]",
"location": "./spec/features/report_spec.rb:20",
"description": "renders a synthetic report",
"failed_attempts": 1,
"final_result": "passed",
"first_error": "expected the default window size"
}
The example ID matters. Several tests can point to the same line in an RSpec shared example; grouping only by that location can merge unrelated failures. We keep both the ID and the location.
The JSON goes in a separate directory, outside the successful-job cleanup, and CircleCI stores it as an artifact:
- store_artifacts:
path: spec/results/retries
CircleCI artifacts let us retrieve those records after the job finishes. The equivalent on another CI service would work too. The requirement is to retain the retry history even when the test eventually passes.
Turn those records into a weekly queue #
A scheduled job reads the previous week's retry artifacts through the CircleCI API. It groups them by example and ranks them by the number of distinct pipelines in which the example failed and then passed on retry.
If an example fails twice in one pipeline, that is still one pipeline in this ranking. We retain the failed-attempt count too, along with the branches and the first error message.
An example that failed every attempt goes in a separate count. That distinction describes the observed run; it does not tell us whether the underlying cause is deterministic. An intermittent failure can exhaust its retries too.
When an example exceeds our configured weekly budget, the digest job fails and our existing CI failure notification brings it to our attention. I can then give the report to an agent with access to the relevant code and CI artifacts.
I want the agent to explain why a retry changed the outcome. More retries would make the report quieter without answering that question.
A browser restart hid state left by another test #
One set of failures reported that screenshot dimensions had changed. The test usually passed when retried.
The cause was in an earlier test. Billing tests resized the browser window and left it that way. A later screenshot test inherited the different viewport. Our retry setup restarted the browser, which restored its default dimensions, so the retry passed.
The fix establishes the default window size before each browser example. Tests that need another size can still resize it themselves.
Once the agent read our retry callback, the passing retry helped explain the failure: it had started with a fresh browser. The test hadn't recovered on its own.
A random apostrophe found a broken cURL command #
Our dashboard generates sample cURL commands from a template's schema. The tests exercise those examples using generated data, which sometimes includes an apostrophe in a fake last name.
The inline command wrapped its JSON body in single quotes. An apostrophe inside that JSON ended the shell string early. A later attempt could generate a different name, and the command would work.
There was a related mistake in the downloadable Bash script. It put the JSON in a quoted heredoc, but still applied the escaping intended for a single-quoted shell argument. That escaping changed the body of the request.
The two places needed different handling. The inline command needed to escape an apostrophe for its shell context; the quoted heredoc needed the JSON left alone.
To check this properly, the new regression tests run the generated scripts in Bash with curl replaced by a function. The function captures the body that would have been sent, and the test parses it and checks the original value. No network request is needed.
We also replayed both versions with the synthetic value Editor's proof. Before the fix, the inline command failed in Bash. The downloadable script ran, but its request body failed JSON parsing. Afterward, both preserved the original text.
Changing the test data to avoid apostrophes would have made the test more predictable while leaving the generated commands broken for valid input.
A temporary file disappeared before it was read #
Another failure came from the inline HTML-to-PDF renderer used by self-hosted installations. It writes header and footer HTML to temporary files, then passes their paths to the rendering process.
The code kept each Tempfile in a local variable while building the command. After that method returned, the command retained the path, but nothing retained the Tempfile object.
Ruby's Tempfile.new attaches cleanup to garbage collection. If GC ran before the renderer opened the file, the file could already be gone. Whether the test passed depended on when garbage collection happened.
The fix keeps the header and footer objects on the service instance until rendering finishes, then closes and removes them in an ensure block. The regression test forces GC between command construction and the simulated renderer reading the files. It checks that their contents remain readable, and that the files are removed afterward.
We replayed that test condition against both versions. The old implementation lost the files before the read; the fixed implementation retained them and cleaned them up at the end.
Forcing GC lets the regression test check the failure directly. It no longer depends on getting unlucky during a PDF test.
Freezing time still left a second clock running #
An earlier investigation concerned rate-limit tests. Rack::Attack counts requests in fixed time windows. If a test crossed a boundary, requests started counting in a new window and the expected limit was never reached.
Freezing Ruby's clock sounds sufficient, but the counters were stored in Redis with an expiry. Redis continued measuring real elapsed time. Freezing near the end of a window could leave the counter with very little time before expiration.
The fix freezes at the beginning of the hour, aligning with the windows these tests use and giving their counters the full expiry period. It still matters that the example finishes within that period: freezing Ruby does not freeze Redis.
The review caught the Redis expiry problem after the first fix froze Ruby's clock. That extra check mattered: the first explanation had accounted for only one of the clocks involved.
What I ask the agent to deliver #
I give the agent the digest and ask it to investigate the causes. This is an adapted prompt, rather than a transcript of a particular run:
Find the relevant CI runs and read the failure artifacts. Trace the cause through the test and application code. Reproduce it locally where possible, and add a regression test that fails before the fix and passes afterward. Keep unrelated causes in separate signed commits. Run the affected checks and review the diff. Report anything you could not reproduce, and wait for approval before pushing.
The artifact and the code often need to be read together. A screenshot mismatch did not explain the viewport leak. A missing file did not explain why a Ruby object had become eligible for garbage collection.
Other failures came from background workers still running when database cleanup began, and overlapping changes to Timecop's shared clock state in our test helpers. A separate product bug was in our sample-data generator: a truthiness check skipped the exclusive-bound adjustment when the lower bound was zero. Zero is falsy in JavaScript, but it was a valid bound in that schema.
Where an investigation reveals a rule future work should follow, we put it in the nearest AGENTS.md. For example, a browser test can change its own viewport, but the next test must start from a known size. The setup code enforces the rule; the note explains why it exists.
We've also written a custom RuboCop rule for flaky tests. When a mistake can be detected automatically, I'd rather have CI catch it than rely on someone remembering the earlier investigation.
Work I can now get through #
I haven't measured the hours this saves. These fix rounds are also too recent to tell us how much they reduce recurring failures over time. The next weekly reports will help answer that.
I am much more willing to take on this work now. I can start an investigation and keep doing other things while the agent follows it through. I still need to assess its explanation and review the changes, but I don't have to personally work through every possible cause first.
That is also why I care about code quality when agents write the code. A reproducible failure and a test that checks the intended behavior give the agent something concrete to work against. Preserving retry history makes more of those problems available to investigate.
If your suite already retries failing tests, start by saving the attempts that disappeared behind the final passing result. Pick one recurring failure and ask an agent to explain exactly why it passed the second time. You may end up fixing something your users can run into as well.