I see more and more converts to mutation testing, as folks learn that extending the horizon of agentic coding requires much, much stronger quality gates.
Mutation testing deliberately injects errors into code to create “mutant” versions – like turning a + into a -, or replacing a string with “” – so we can see if any of our automated tests catch the error. If no tests fail and the mutant survives, that reveals a potential gap in the test suite.
Mutation testing is a specialised version of what I’ve been calling “Brown M&M tests”. Perhaps you know the story – Van Halen had a clause buried in their venue rider that there should be a big bowl of M&Ms in the dressing room with all the brown ones taken out.
Not an example of diva-like rock start excess, but a very practical test of whether the venue had paid attention to every detail. Van Halen’s live show had many complex technical elements. If they entered their dressing room and saw brown M&Ms in the bowl, they double-checked everything.
The rise of mutation testing’s encouraging and frankly long overdue. But don’t stop there!
How are you testing your linter rules? I might deliberately inject unused imports into random source files to see if my automated code review finds them all. I might deliberately inject security vulnerabilities, or race conditions, or nonsense identifier names – all example of “brown M&Ms” – so I can test for gaps in those quality gates.
(Now, go on, admit it, “dark factory” folks, this probably never occurred to you, did it?)
I realised – through experiment and research going back 3+ years – that people’s confidence in LLM-generated code has more to do with whether or not they see the “brown M&Ms in the bowl” than the actual code’s quality.
When I see these claims of high-quality generated code that I’ve not been able to reproduce, I’ve stopped asking “What are they doing that I’m not doing?” – at this point I’m way ahead of the curve – and now ask “What am I seeing that they’re not seeing?”
Join me for a hands-on, evidence-based workshop – free of hype and hopium – on Code Craft & AI. Tues Oct 6th 18:45 BST. Just £99 + VAT for self-funding learners.