Yes you can run exploratory testing with ai

AI can effectively run exploratory testing when prompts are well-framed, according to testing expert Cakehurst Ryan. The author argues that with proper heuristics and guidance, AI tools like Claude can surface real bugs and quality insights, though human expertise remains essential for defining exploration scope and success criteria. The post cites GitClear data showing code churn rose from 3.1% to 5.7% as AI adoption increased, and METR research finding experienced developers were 19% slower with early 2025 AI tooling due to rework.

Yes you can run exploratory testing with ai AI can actually run your exploratory testing… it just needs some help deciding what’s worth exploring, what good enough looks like and when to stop. There’s a lot of posts and discourse on social media that claims that AI can’t do testing, especially nothing as complex as exploratory testing. In fact this is something I said back in my post can generative AI write good exploratory tests https://cakehurstryan.com/2023/07/03/can-generative-ai-write-good-exploratory-tests/ … that the answer was only “sort of”. What’s changed since then isn’t the AI model but how I think about framing my AI prompts. It turns out that AI really can explore through a browser or the code , apply heuristics and surface real insights, bugs and quality narratives, so long as our prompt to do something is good Like with any AI driven development, naive basic asks won’t give a good response / outcome, the real skill is knowing how to frame exploratory testing that finds real and useful outcomes. In this post I’ll share how I run exploratory testing using AI with a Claude skill and why my specific testing expertise is still needed. Why exploratory testing matters more than ever When your team is shipping code and features faster than ever what’s going to stop them delivering the wrong thing or AI slop ? Exploratory testing As I’ve talked about to developers and engineers https://cakehurstryan.com/2020/05/15/why-im-talking-to-developers-about-exploratory-testing/ before uncertainty and the unknown in what we’ve built or are going to build means more risk for building the wrong thing. Moving faster and taking understanding and context away from humans by using AI means there’s more uncertainty about what we’re building which is riskier. We need to find ways to reduce that risk by testing and exploring early https://cakehurstryan.com/2022/10/24/dont-wait-to-test-go-explore/ throughout the software development lifecycle to create certainty. Reducing costs Another factor that we need to take into account with AI based development is the cost of things. Rework through getting things wrong is waste, we spend time, money and tokens on fixing something that’s broken. GitClear https://www.gitclear.com/ai assistant code quality 2025 research found code churn lines reverted or rewritten within two weeks rose from 3.1% to 5.7% as AI adoption climbed and METR’s https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/ controlled trial measured experienced developers as 19% slower with early 2025 AI tooling with most of it lost to reviewing and reworking generated output. One way to reduce rework, token usage and costs is to try to build the right thing first time. This means trying to identify negative and edge cases early to bake them in to what’s being built. Using exploration for issue and AI slop prevention rather than late stage detection which results in costly rework. As I’ve seen in many product and engineering teams that I’ve worked with people generally don’t know what good looks like for testing https://www.youtube.com/watch?v=lFbFKEwlK9w&list=PLKBhokJ0qd3 Qms3DloAbdq0zTGLQ0pFE&t=1s and this extends to exploration. Testing gets limited to verifying what was asked for through the happy path, something that limits us to knowing that something was correct rather than good. Often I’ve seen siloed thinking where risks related to how code or products work in the real world get pushed later and later… into a testing phase or from customer feedback; this won’t work for AI development. Teams are moving faster than ever with AI accelerating code and feature development, the existing bottlenecks of human led review and validation will just get worse. Without a way to look at what we build for how good it is at pace, the slop can ship faster than anyone can catch it. Using AI for exploratory testing I’m currently using AI to both generate exploratory test ideas and to drive exploratory testing sessions. I’ve found these to both be really successful in identifying useful risks and identifying issues that I care about. Hot take: Most people aren’t very good at exploratory testing. As I’ve written about before https://cakehurstryan.com/2021/11/02/why-adhoc-testing-is-not-exploratory-testing/ most people tend to assume exploratory testing is unstructured, usually a bug hunt that comes from “clicking around and finding out”. We see a lot of discussions talking about intuition or having a testers mindset for this type of testing… that there’s some dark art to being able to test a system. Good exploratory testing follows a structure: it has a goal for what it’s looking for, a scope of what to look at and identifies tools and heuristics to follow to help guide testing. When I think about structuring my exploration I tend to follow the guidance set out in Explore It by Elisabeth Hendrickson https://pragprog.com/titles/ehxta/explore-it/ . Structured tests are the thing that’s moved me from “AI kinda works…” to seeing that this actually can work. When creating skills for AI to do exploratory testing I’ve focused on providing structure for the exploration to guide and set expectations for the testing it does. Creating exploratory tests 1. Identifying risks For structured exploration I want to explore to find out information related to a specific risk, so the first part of creating tests is determining the risks related to what’s going to be built. To do this I create AI agents that take each feature request / AC and look for risks based on tried and tested quality heuristics; these are easily identifiable across the internet https://www.ministryoftesting.com/insights/software-testing-heuristics-mind-the-gap with some big and well trusted names Elisabeth Hendrickson, Ministry of Testing attached to them. Since quality and testing can be infinite I create agents to categorise each risk for Likelihood x Impact and create a score between 1 – 10 which can be used to pragmatically select what we test for: High risk appetite : select risks with a score of 7 Medium risk appetite : select risks with a score of 5 Low risk appetite : select all risks This is something that can be tweaked in accordance to an organisations views about risk and what good enough looks like, or can be bypassed for types of risk such as – surface all security risks regardless of risk score . This also means we don’t waste tokens by creating tests for risks that we wouldn’t care about / would never fix… in fact we can point this skill to previously discovered defects that were fixed by the team to give a view on risks we’d care about. 2. Writing tests From the risks that have been identified I create agents that take them and frame them into something an AI can use for exploration. To do this I set the agent to write tests per risk that follow the “explore target, with tools and heuristics, to discover information about a specific risk” format. Each of my tests then has a clear focus and a goal that I am able to control in my up front test design. When specifying the tools and heuristics for each test the “with” part this will differ by context and should draw upon what’s available. I try to provide examples of what I like when critiquing the AI’s outputs so that it can learn what’s important to me. It’s important, especially early on, not to just fire and forget an AI prompt but to look at it and start steering it to better outcomes; that’s human in the loop / botsitting in action and provides a better result. You can find my Claude skill for creating exploratory testing charters from a spec in my QE skills Github repo https://github.com/cakehurstryan/QE-Skills . Running exploratory tests Once I’ve got the test charters I pass them into Claude to actually run the exploration. This works because the structure of the test charter framing, scope and risk based goal gives the AI a targeted success criteria that keeps testing tight and meaningful. 1. Setting up the test runner To explore via the UI I run Claude in a Chrome browser using the Claude chrome plugin so that it can navigate pages, read the page DOM and interact like a user would. For deeper work I use Claude Code within a downloaded repo with my AI enabled Playwright framework https://cakehurstryan.com/2026/04/27/creating-a-playwright-framework-with-ai/ included so that it can explore across the UI, API and code layer. The set up for both are really easy and you can get AI to help you to do this. 2. Running the exploratory testing session I give the AI prompt the charter and let it work, usually saying the type of report that I like to see as an output. Claude then navigates to the target I’ve given it and applies the tools and heuristics to start finding information, then reports on it. This is where botsitting matters because Claude will be verbose in what its trying to do, so you can interrupt and say “I don’t want this” or steer it to something that matters. Here’s a charter I actually ran, for accessibility and usability across a web page. You can see that I’ve provided structure for the exploration by using the explore / with / to discover to set a target, goal and scope and that I’ve also provided clear heuristics to use. Explore the homepage of <car dealership website and the content within it with trusted UX and usability heuristics, user personas for e-commerce sites and WCAG 2.2 accessibility standards to discover risks that the homepage would not be understandable, usable and accessible for a user trying to buy a car 3. Reviewing the report Reports will differ because of non-determism, so if you have a report type you like make sure to feed the prompt with an example. I ask for a clear reference of which heuristic found the issue and a narrative as to why it matters, along with a scoring of how bad this issue is. here is an abridged set of the results from my previous prompt to show an example of what the exploration comes back with. - Critical - No entry point to the actual task. The homepage has no search, no browse-by-make/model, and no filter. The only featured cars are jokes a DeLorean and the Mystery Machine, both labelled “not a real deal” . A buyer who knows what they want has nowhere to go. Fails Nielsen Flexibility and efficiency and Recognition rather than recall. - High - Joke price shown before the disclaimer. The “£88/mth” spec block reads as real until you hit the “not a real deal” line underneath it. Fails Nielsen Error prevention. - Ignores e-commerce conventions. No product grid, no configurator, no clear quote CTA. Navigation pushes “Karaoke” and “Movies” alongside buying. Fails Consistency and standards. - Autoplaying audio with no control. An autoplaying looping video embed risks failing WCAG 2.2 SC 1.4.2 Audio Control Level A . Worst hit: screen-reader users, whose navigation audio gets drowned out. 4. Deciding when to stop The charter’s risk appetite does most of this for me; it didn’t try to test everything and instead it gave me a pragmatic set of findings mapped to the risks I said I cared about then it stopped. That’s the up front design paying off for me. By keeping an eye on what the AI is doing I can also tell it to stop directly, or I can even give it a timebox for exploration in terms of time or token cost. I then refine these based on the outcomes I see, using my experience and judgement. Tips for using AI for exploratory testing It’s worth noting that if you don’t immediately get good results that you might need to train the AI on what good looks like and build its context. Reply to the tests or test results it gives you to say what you like and don’t like and tell it to commit that to memory or update the skill . This should be seen as a living process that will need to change to give you the results that you want and need. Because we can both create and execute exploratory tests using AI agents it means this is automatable We can set these skills and agents up in a pipeline to run quickly, helping to provide signals of quality whilst preserving speed of delivery. I’ve found it successful to run exploratory test ideation following a Triforce / Three Amigos session which I also have created a Claude Skill for and running executing exploratory testing following a deployment to a test environment. Getting Claude to execute running tests against your browser to drive exploratory testing from the UI is simple, you can download a browser plugin that allows this to happen. I’ve also found that Claude code will work well across UI, API and the code layer if you ask it to explore from a downloaded repo that has an AI enabled Playwright framework https://cakehurstryan.com/2026/04/27/creating-a-playwright-framework-with-ai/ included, this will provide more detailed exploration and allow for codifying tests you want to rerun as Playwright acceptance tests. A point on non-determinism Let’s address the elephant in the room… yes testing with an AI is going to give different results each time you run it because models are non-deterministic. But here’s the thing, thats actually good and is something we want Exploratory testing should have an element of randomness to it in order to provide us information about the risk; it’s not automated checks where we want to run the same script over and over again. The thing we want to stay the same is the test structure, what are we testing and what’s our goal? How we get to finding that information is allowed to change because that’s the whole point of exploration. So don’t spend a lot of time trying to correct for this, embrace it as a way of reducing uncertainty within a scoped structure. Oh and don’t build your CI/CD pipeline to pass or fail around the results of exploratory testing because these non-deterministic results will cause false failures and a flaky pipeline. Instead take the report and use human in the loop judgement to review and fix for them as you would for any other bug, then codify anything worth rerunning as a deterministic Playwright test. Wrapping up There will always be people who don’t think you can automate testing. They’ll say that it cannot think, reason or understand a customer it can if you tell it to , that it spits out slop it will if your prompt is lazy , doesn’t have a tester’s mindset it can if you structure your tests right or that it just hallucinates correct it does, that’s why we need more testing and human review . But are these objections well founded or are they coming from a fear of not being needed anymore and losing their jobs a very valid concern . The things I’d say are real barriers for automated AI based exploration are actually the cost of tokens so make sure you scope risk appetite correctly , people might lose their testing skills so keep them sharp by reviewing AI based testing and that it’s hard to provide meaningful context for your product / organisation a foundation that needs defining https://cakehurstryan.com/2026/06/12/ai-readiness-radar/ . Additionally, for many people a big barrier will be reframing what exploratory testing really is… away from ad-hoc random bug hunting and towards something structured that actually is automatable. We can automate exploratory testing, but this only works as well as the structure: charter, risk appetite and heuristics you give the AI. That’s where a skilled quality professional comes in; you need to decide what’s worth exploring, what “good enough” looks like through context and heuristics make for good testing and when to stop. The needs of testing have moved away from using your skills and intuition whilst doing the testing and more towards using them when framing the exploration that needs to happen. If you want to try this out then please use the Claude Skills in my QE skills Github repo https://github.com/cakehurstryan/QE-Skills and let me know how they work for you. Thanks for reading . If this resonated or if you think that I’ve got it wrong there’s more to read on the blog… or get in touch via LinkedIn. I’ll talk quality to anyone who wants to listen