We recently sped up our integration tests by a factor of ten by running many of them in parallel. There were several flaky tests in the new test suite. Claude was really useful in getting rid of almost all flakyness, but I made some mistakes along the way. Here is what I learnt.
Our system is very well covered by integration tests. We have a complete local system running, and all integration tests interact with the running system via FIX connections (traffic) and HTTP connections (administration). Previously, our 1360 integration tests ran sequentially, and took around 40 minutes to complete. A recent rewrite got the running time down to around 4 minutes instead. The gain comes from being able to run many tests in parallel.
To be able to run tests fast in parallel, the test suite relies on instruments that are defined once, then reused for many tests. This however makes the tests vulnerable to interference between tests. After the rewrite, there were typically a few tests each run that failed. If the test is run in isolation, it passes. So this is the definition of a flaky test: it passes in isolation, but may fail (due to timing issues) when run as part of a larger test suite.
Even though it is fairly easy to try the test in isolation after a failing run, it is much better to get rid of the flakiness once and for all.
1. False Start
I was not very familiar with the testing framework, but I thought Claude would be good at finding and fixing the flaky tests. Each failing test produced a stack trace from where it failed. Often, the stack trace would just show where a wait for a desired state timed out, so the underlying cause was not obvious. Claude excelled at finding the causes of flakiness. So I got carried away, and accepted and committed the proposed fixes without really understanding the rationales. Since the problems appeared randomly, it is not always easy to tell if the underlying problem had actually been fixed.
The flaky tests became noticeably fewer, but did not disappear completely. When I asked my colleague, who had ported many of the test cases, to review the changes, he questioned many of the commits. Several of them sounded quite plausible, but had no effect. I realized that I did not understand most of the fixes, but had just blindly accepted whatever Claude suggested.
2. Understanding
So I started over, and this time took the time to really understand each fix. I would ask Claude about all aspects of the fix I did not understand. Many times, Claude was correct, and I learnt more about how the tests and the framework work. But occasionally, I would get these types of responses when I questioned the solutions:
“You’re right to push on this, and I need to walk back what I said”
“Fair challenge — let me be honest about what I can and can’t demonstrate.”
“So I was overstating it earlier”
Eventually, through our joint effort, we would arrive at a solution. By not blindly accepting Claude’s suggestions, I learnt a lot more about the tests, and I avoided adding pointless code to the tests.
3. Claude is Very Capable
In many of the cases, Claude did a fantastic job of finding the cause of the flakiness. Getting stack traces analyzed is a breeze, as is crosschecking multiple log files, checking the time stamps for overlapping actions. Claude’s encyclopedic knowledge of Java and associated frameworks really helps. However, because Claude is so good in so many cases, it is easy to assume that all of its output is correct.
4. Faster Understanding
I have really come to appreciate how Claude can explain not only a proposed solution, but also more broadly, the existing code in general. It is now much faster for me to understand existing code. I first ask Claude to give me an overview how some part of the system works. Then I can drill down into specific areas by asking follow-up questions.
5. Unlocking New Uses
Before LLMs, it is doubtful if it would have been worth the effort to chase down all the causes of flakiness. It would simply have taken too much time, even if the end result is good to have. This probably applies to other uses as well. Something that was not worth doing before is now possible to do cheaply enough. This is great.
Conclusion
Claude was a great help in tracking down the various causes of flaky tests. However, not everything it suggests is correct. Because it does many things so well, it is very easy to just accept every change it suggests. Therefore, being able to understand if the proposed fix is valid or not is very valuable if you care about your system.
Currently, less than one run in ten has a flaky test failure. Much better than before, but still some work for Claude and me to do.