{"slug": "exhaustive-testing-is-still-impossible-even-with-llms", "title": "Exhaustive Testing Is Still Impossible, Even with LLMs", "summary": "LLMs are discovering bugs in mature, stable software, confirming that exhaustive testing remains impossible and impractical, according to an industry observer. The rapid adoption of LLM-generated code in software development is happening without a mental model or risk assessment, and LLMs cannot yet verify whether built software meets requirements or solves customer problems, as that context lies outside the code. The author argues that risk-based testing remains the right approach and that while LLMs can accelerate test case generation, they cannot yet do it autonomously without losing the learning process.", "body_md": "I think we are about to find out (yet again) that exhaustive testing is impossible. Or at least impractical. LLMs are already discovering bugs in mature, stable software. That is confirmation that exhaustive testing was never possible and still is not possible.\n\nBut the real price is probably coming. We (as industry that moves ahead with adopting full LLMs in SDLC) are creating, reviewing, and merging LLM-generated code at full speed without understanding how it works. No mental model. No risk assessment. Just speed.\n\nI am not saying here don’t use AI/LLMs. I want to talk more about how can we create an environment that will guide an LLM toward a direction that is verifiable and we can trust the output.\n\nCoding is solved is true only if coding means producing a lot of code that runs. In case coding means engineering and quality and verification we still have a long way to go. What works is that we can throw a lot to LLMs and as they are so great with working with text hey can help a lot.\n\nBut they are not yet there to verify on their own:\n\n(1) If what we built is what was required to be built (verify it works as expected)\n\n(2) If what we built is what is needed (validate solution fit for customer problem)\n\nThose require context thatis outside the code.\n\nThat’s why I go back to some fundamentals of software and we have to revisit and see which principles are still standing and which not. Exhaustive testing is impossible is holding. LLMs are not solving it. Quantum computers maybe it will when they reach market scale.\n\nIf exhaustive testing is impossible then a couple of consequences follow directly and testing remains a problem of selecting a finite set of observations from a much larger space of possible behaviours/inputs/outputs.\n\nTesting based on risks is still the right approach.\n\nWhen I write testing here you are maybe thinking about tester. But you are a bit misguided if that is the case. Developers are doing testing, product managers are doing testing, technical founder are doing testing and a wide range of people are doing testing.\n\nAs it was the case before with automated testing it is now with generative test cases and automatic running: you should make sure your environment is generating the proper test cases that fit your business/product/market/risks and the solution is the correct one for you problem.\n\nLLMs can accelerate a lot of these findings but they cannot (yet) do it on their own. A field observation is that the more you delegate out of these decisions to LLMs the learning process that fuels these kind of decisions is lost and it becomes harder to steer.", "url": "https://wpnews.pro/news/exhaustive-testing-is-still-impossible-even-with-llms", "canonical_source": "https://allaboutcoding.ghinda.com/exhaustive-testing-is-still-impossible-even-with-llms/", "published_at": "2026-08-24 07:20:00+00:00", "updated_at": "2026-08-24 11:45:51.788202+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-safety", "ai-ethics", "developer-tools"], "entities": [], "alternates": {"html": "https://wpnews.pro/news/exhaustive-testing-is-still-impossible-even-with-llms", "markdown": "https://wpnews.pro/news/exhaustive-testing-is-still-impossible-even-with-llms.md", "text": "https://wpnews.pro/news/exhaustive-testing-is-still-impossible-even-with-llms.txt", "jsonld": "https://wpnews.pro/news/exhaustive-testing-is-still-impossible-even-with-llms.jsonld"}}