Computer Use Agents Are Excellent QA Engineers

Computer Use Agents are Excellent QA Engineers #

  The recent proliferation of software such as Grok Bot, Instinct, Codex, etc. indicates that computer use is one of
  the most rapidly expanding frontiers in LLM agent capabilities. In my work as a founding engineer at

      [Fulcrum](https://www.withfulcrum.com/), I have found a particularly compelling use case for computer use agents as quality assurance (QA)
      engineers. In the following I will briefly explain the setup which has so far been effective.

My day to day work involves designing, building, and maintaining a large feature set of a SaaS platform which aims to revolutionize insurance brokerage operations. We have a number of different workflows with varying degrees of human oversight from long-running agent workloads in which our users may not examine the output for an hour at a time to more classical UX in which our users rapidly click through an interface and take a number of manual actions to accomplish their goals and shape the output to their liking. In this work, the user only has a couple of interfaces: their email client and our application.

In setting up a computer use agent QA engineer for my needs, I endeavor to shape the harness as a hybrid system in which the agent can straddle both the user persona, in which the application is more or less a black box, and the developer persona in which the inner workings of the system are transparent and observable. So what do the access patterns of the agent need to look like?

To give the QA agent access for all relevant workflows, it requires:

• Its own email address, scoped as an admin with the ability to impersonate arbitrary users on the application. The agent can then access both user interfaces, the email client and the application, in the same way a human user would.
• Read-access to the code repository, in order for it to have context related to the most recent feature developments, and their intended product use cases.
• Read and write access to a central project management and issue tracking system, such that it can track any bugs in the workflows it might find with very rich context for the engineer who will eventually be assigned to the fix. We use Linear at Fulcrum.
• Read and write access to the central messaging board which the organization uses for collaboration (Slack, etc).
• The ability to maintain its own knowledge repository, typically a skill , which is seeded by the developer, but modified and shaped to its ideal purpose as the agent encounters roadblocks in its work.

With these basic building blocks, I have found that computer use agents can, for the most part, significantly reduce the amount of QA I manually conduct.

The workflow roughly looks like this:

1. Seed the QA skill by instructing the agent to:

2. Parse through all of the commits from the time since its last exercise of the QA workflow and compile a list of user-facing, atomic features that were shipped in that time period.

3. For each feature, build up context for the intended flows a user might take in the application by examining the implementation in code.

4. Create a plan to exercise happy paths and creative edge cases which might represent the full spectrum of user flows.

5. Richly document its work with screenshots, clear reproduction steps, and code context.

6. If it encounters a bug or something which might block a user's ideal experience, it should formulate an issue in such a way as to make the eventual fix work of the engineer as seamless as possible.

7. Continue until each atomic feature has been exercised.

8. Create a scheduled action, perhaps after each deployment or nightly, in which the agent performs each step in #1, and as it encounters any roadblocks it either creatively overcomes or escalates the conundrum.

9. At all relevant steps, it updates its own knowledge repository such that its future QA work is more reliable and efficient.

It was a lovely feeling the first time I checked Slack in the morning and found that the autonomous QA engineer had discovered a couple of issues stemming from my prior day's work and richly explained their likely solution, with clear reproduction steps, all while I was sleeping. I encourage anyone developing user-facing software to try this workflow out in their day to day work.