Building an AI QA Agent That Tests Your Jira Tickets in a Real Browser

A developer built an open-source AI QA agent that takes a Jira ticket ID and a test environment URL, reads the ticket, writes test steps, and executes them in a real headful Chrome window. The agent combines Agent Factory, a studio for designing and chatting with agents, with BaaS (Browser as a Service), which drives the browser by opening pages, clicking, waiting, and taking screenshots. Setup instructions cover running both components locally via Docker, configuring an OpenAI-backed LLM client, and wiring Jira API token authentication into the Test Orchestrator agent.

Give it a Jira ticket ID and a test environment URL, and this agent reads the ticket, writes test steps, and runs them in a real Chrome window while you watch. It's built from two open-source pieces. Agent Factory is the studio where agents are designed, edited, and chatted with. BaaS Browser as a Service is the part that actually drives the browser: it opens pages, clicks, waits, and takes screenshots on the agent's behalf. This post walks through setting both up locally and running the bug check agent against your own app. Required: Recommended: Clone the repo: Ursa-Minor-Beta/agent-factory-docker-api-ui https://github.com/Ursa-Minor-Beta/agent-factory-docker-api-ui . Then create your env file and bring everything up with the bundled database: cp .env.example .env edit .env – at minimum set the admin email and password docker-compose --profile with-db up --build The admin email and password in .env are what you'll use to log in to the factory later, so pick something you'll remember. I recommend running BaaS locally while you debug and in Docker while you test, because locally you can watch exactly what the agent does in the browser. If the local setup fights you OS permissions and the like , fall back to Docker. It's stable. Start MongoDB and create the env file: cd baas docker compose up -d mongodb cp .env.example .env vim .env Set these values in .env : API KEY=<pick-a-key BROWSER EXECUTABLE='/Applications/Google Chrome.app/Contents/MacOS/Google Chrome' BROWSER HEADFUL=true LLM CLIENT=openai OPENAI TOKEN="<your token here " OPENAI ORGANIZATION="<your organization here " BROWSER HEADFUL=true is what makes the Chrome window visible. The path above is for macOS; point it at your own Chrome binary on other systems. Remember your API KEY , because the studio needs it in Step 3. Next, map host.docker.internal to localhost so the Dockerized factory and the local BaaS can find each other: sudo vim /etc/hosts add this line: 127.0.0.1 host.docker.internal Finally, load the env and start BaaS: set -a && . ./.env && set +a go run ./cmd/baas You'll see warnings about missing Pandoc, PDF-to-image, and wmctrl. Ignore them: you don't need those tools, and this is what a successful start looks like. Open the studio at http://localhost:8080 http://localhost:8080 and log in with the admin credentials from Step 1. Then: BAAS API KEY , set to the same value as API KEY in the BaaS .env . That's the whole setup. To check the browser actually works, go to Agents → Browser Screenshot and enter https://google.com . A new Chrome window will open. It may ask for permissions, so allow them. If everything is wired up, the window loads Google and the chat returns a browser session ID plus a screenshot of the page. Log in to Atlassian API tokens https://id.atlassian.com/manage-profile/security/api-tokens and choose Create API token with scopes . Then encode your email and token together: echo -n "you@example.com:your-api-token" | base64 Add the output to the studio's Secrets under the name Jira auth . In the factory, go to Agents , find Test Orchestrator , and click Edit . Two things to change: input-1 node, replace the default URL https://google.com with your test environment URL, and set your jiraSubdomain . http-1 node. For login, I recommend hardcoding it in http-1 . Login flows rarely change, so scripting them once saves tokens on every run. If you'd rather skip a separate login step, set the body of http-1 to the following. It starts a browser session, opens the URL, and takes a screenshot: "body": "{\"program\":\"navigate '{{node:input-1.url}}' ; sleep '3s' ; waitReady body,'timeout:20s' ; takeScreenshot 'screenshot', 'timeout:10s' ;\", \"sessionID\": \"{{node:http-0.response.result.sessionID}}\", \"stopSession\": false}", Open Test Orchestrator and click Chat . Send it your Jira ticket ID something like SCRUM-165 and your test environment URL. The agent takes it from there and runs the first test. The prompt in llm-generate-steps is where test steps get written. Replace the example details there with what you know about your application. Pasting in your user documentation, if you have any, works well. The orchestrator uses Test Step Executor as a skill, and its prompts are worth adapting to your app and its quirks. Each LLM node has one job: llm-0 llm-1 llm-2 I hope this agent saves you some manual testing. If you run into problems or have ideas, get in touch. One honest ask: this is a side project, and I'd like to know whether to keep building it. If it's useful to you, star the repo on GitHub https://github.com/Ursa-Minor-Beta/agent-factory-docker-api-ui , send me a DM, or leave a comment below. Any of those tells me the time is well spent. Keep humans on the loop and don’t ship on Fridays