Superpowers vs Plain Old Debugger in Explyt

A developer's controlled comparison of two AI debugging workflows found that an agent using the Superpowers systematic-debugging skill missed a real bug despite following a structured method, while Explyt's built-in Debug skill solved it. The Superpowers run proposed a fix that masked the failure and added a test seam, whereas Explyt used runtime facts to identify the root cause more efficiently.

An AI agent can follow a debugging checklist and still miss the bug. The question I kept coming back to was simple: who ran the experiment? I interviewed our users, mostly senior developers. In those interviews, debugging a hard problem came up most often. In practice, the investigation is still manual: a developer supplies logs, sets breakpoints, tests a theory, and pastes the result back into the chat. A systematic debugging method gives the work a clear order: gather facts, form a hypothesis, run an experiment, then change the code. Superpowers https://github.com/obra/superpowers was created by Jesse Vincent https://blog.fsck.com and the Prime Radiant https://primeradiant.com team. Matt Pocock helped popularize it; the collection includes a systematic-debugging skill https://www.skills.sh/obra/superpowers/systematic-debugging . I ran Opus 5 on the same real bug and the same prompt twice. The only thing I changed was the debugging workflow: Superpowers in one run, Explyt's built-in Debug skill in the other. I ran one controlled case; it is not a benchmark of either product. One number makes the contrast plain: Explyt used 67k tokens to solve this case, while Superpowers used 132k. The transcripts explain the gap: the debugger supplied a runtime fact early, while the other run spent more context on speculative code, tests, and refactoring. The test case came from a Kotlin project on Windows. The IDE log contained: java.nio.file.AccessDeniedException: C:\Config.Msi at java.nio.file.Files.list Files.java:3422 at FacadeMentionCandidates.listDir FacadeMentionCandidates.kt:83 at FacadeMentionCandidates.listChildren FacadeMentionCandidates.kt:69 at MentionSuggestionMapperKt$suggestionsFor$2.invokeSuspend MentionSuggestionMapper.kt:36 The relevant code first checked whether a path was a directory and then enumerated it with Files.list . The exception came from a protected Windows Installer directory. FacadeMentionCandidates makes filesystem roots available to the @ mention flow, so a user navigating through a path such as @/C:/ can eventually reach a directory that the IDE process cannot enumerate. The error was not handled at the flow boundary. It escaped the suggestion flow and was reported as an unhandled coroutine exception in WorkspaceBootstrap , with the IDE blaming the plugin. The same listing path is also used while searching around the focused file, so this was a small function with more than one way to fail. The Superpowers run started well. It searched the log instead of reading the whole file, followed the stack into FacadeMentionCandidates , found the unguarded Files.list , and stated a plausible root-cause hypothesis. Then the investigation stopped before the experiment. The run never set a debugger breakpoint or inspected a value from the failing execution. The proposed fix was to wrap the directory listing in a try/catch , log the exception, and return emptyList . That choice made the failure disappear from the call stack, but it changed the meaning of the result. An empty directory and a directory denied by ACL became the same value; other I/O failures still had no user-facing outcome. The mention UI could not tell the developer what happened, and the option to mention the directory remained indistinguishable from a normal empty-folder case. The regression test followed the same assumption. It asserted that an unreadable directory produced no children. On Windows, the first version could not create the required POSIX permission state and was skipped. To force the exception, the agent then added a readDirectory function to the production constructor so the test could inject a throwing fake. That seam existed for the test, not for the product behavior. The run finished with seven passing tests, but those tests established that the workaround returned an empty list. They did not establish that the user-facing behavior was correct, or that the error had been verified against a real paused execution. I discarded the patch: it hid the distinction we needed and left a test-only abstraction in the production API. That is how a checklist turns into a ritual. Every step appears in the transcript; the experiment never does. A plausible local edit stands in for evidence. With Explyt's built-in debugging skill, the model started from the same log and source files, but it could use the JetBrains debugger as part of the investigation. It created a minimal reproducer, set a line breakpoint on the Files.list call, and ran the test under the debugger. When execution stopped, the value was concrete: | Debugger evaluation | Observed value | |---|---| dir | C:\Config.Msi | Files.isDirectory dir | true | Files.isReadable dir | false | Files.list dir | throws java.nio.file.AccessDeniedException | The call stack matched the production trace: listDir → listChildren → MentionSuggestionMapper.suggestionsFor , inside the coroutine that builds mention suggestions. The debugger turned "a protected path probably causes this" into a verified statement about the running program: the directory check passes, the process cannot read the directory, and the unguarded enumeration kills the suggestion flow. The evidence kept the fix at the filesystem boundary, with focused regression coverage and the normal listing path left intact. The mention protocol and runtime model stayed unchanged, and debug prints were unnecessary. The narrow fix passed review and is now in production. A try/catch was enough here. What mattered was choosing it after observing the failing execution, instead of using it as the first explanation for the failure. | Superpowers run | Explyt run | | |---|---|---| | Starting point | Same log and source | Same log and source | | Investigation channel | Text, stack trace, and tests | JetBrains debugger plus source and tests | | Key fact | Inferred from AccessDeniedException | Observed at a breakpoint: directory, permissions, and throw | | Token use in this run | 132k | 67k | | First implementation | Swallow the error as emptyList | Keep the fix at the proven failing boundary | | Test signal | Passing tests for the fallback | Runtime evidence followed by a focused regression test | | Result | Patch discarded as technical debt | Reviewed minimal fix deployed to production | I could see the cleanup cost in the diff. The discarded patch carried a workaround-driven test seam. The shipped patch stayed at the failing boundary and left the normal listing path intact. For a runtime bug, a coherent explanation from a log is still only a theory. I want one fact captured from the running program: a variable value, a call stack, an execution path, or the exact branch that contradicts the model's theory. Explyt is an AI agent for JetBrains IDEs. In this run, it used a supported JetBrains IDE and run configuration. Its Debug workflow started the code under the JetBrains debugger and inspected breakpoints, variable values, call stacks, and execution paths before editing. The IDE supplies the evidence; the developer still decides whether the hypothesis and the resulting diff are acceptable. What I can say from these transcripts is limited. Superpowers can be combined with a debugger, and Explyt's debugger depth depends on the IDE, language, and run configuration. Check the feature matrix https://explyt.ai/docs/explyt-test/feature-matrix for the supported combinations. The token count points in the same direction. Explyt solved this case in 67k tokens; Superpowers consumed 132k. A debugger did not merely add another tool to the menu. It gave the model the fact it needed before more speculative turns accumulated. JetBrains has been running paired A/B tests on "token-saving" skills, with a useful warning against reading too much into their README claims. In the Caveman test https://blog.jetbrains.com/ai/2026/07/speak-to-ai-agents-like-cavemen-tosave-tokens/ , the advertised 65% reduction became 8.5% fewer output tokens on real agentic tasks, with activation forced. In the newer rtk test https://blog.jetbrains.com/ai/2026/07/rtk-claude-code-token-savings/ , the with-rtk arm cost a median 7.6% more per task at low reasoning effort, and showed no saving at high effort. The latest Ponytail test https://blog.jetbrains.com/ai/2026/07/ponytail-skill-claude-tested/ found a 10.3% cost reduction on its own benchmark. The results are mixed; the consistent lesson is to measure the whole agent run, not a skill's self-reported counter. If you already work in a JetBrains IDE, install Explyt from the Marketplace https://explyt.ai/t/l/edit , open a reproducible failing test, and let the agent inspect the first runtime value that contradicts its theory. Then review the diff before accepting the fix.