Veta: AI agent that QA-tests Android apps

Veta, an AI agent that QA-tests Android apps using a swarm of autonomous sub-agents, runs 100% of its AI inference on AMD GPUs via Fireworks AI or self-hosted vLLM on ROCm. The system replaces scripted UI tests with AI-driven visual, functional, and accessibility testing, scaling from 10 to 100 concurrent sessions on Kubernetes.

Autonomous AI agent swarm for visual, functional, and accessibility testing of Android apps and mobile web — 100% of AI inference runs on AMD GPUs Fireworks AI on AMD Instinct / self-hosted vLLM on ROCm , with containerized Android instances running locally. Overview overview • Features features • Architecture architecture • How it works how-it-works • API api • Getting started getting-started • Project structure project-structure • Tech stack tech-stack Built for theNo benchmarks, no constraints, just build: give an AI model eyes and hands, and let it QA your Android apps. AMD Developer Hackathon ACT II — Unicorn Track. Veta replaces brittle, scripted UI tests with an AI-driven QA agent that watches an Android screen , decides what to do next based on a plain-English task description, and executes actions via ADB. Each run produces a pass/fail verdict, a complete action trace, and video/log artifacts — the same output a human QA tester would produce, at machine speed. Sessions run in parallel across a fleet of containerized Android instances redroid , managed by a scheduler with a warm pool and autoscaling. A web dashboard lets humans launch sessions, watch live screen mirrors, and review results. | Input type | Description | |---|---| APK | Upload an Android APK, install it on a clean Android instance, and run a task against the native app | Web domain | Provide a URL to test inside a sandboxed mobile browser, with optional login credentials | Context file | Attach .txt , .md , .docx , or .pdf checklist alongside the task — contents are folded into the agent's working prompt at runtime | The system starts at ~10 concurrent sessions on a single host and scales to ~100 concurrent sessions on Kubernetes. Every single model call — planning, execution, verification, reporting — runs on AMD GPUs. Zero inference happens on non-AMD hardware by default. | Path | What runs on AMD | Hardware | |---|---|---| Fireworks AI default | Planner, executor, verifier, reporter | AMD Instinct MI250/MI300X | Self-hosted vLLM + ROCm | Same pipeline, your hardware | Any AMD GPU via ROCm | The agent has no fallback to non-AMD inference providers. Gemini, OpenRouter, and Anthropic are listed as optional alternatives in the settings UI — none are enabled or used by default. Out of the box, 100% of AI compute is AMD. Sub-agent architecture — Planner creates a checklist before execution. Executor drives per-step decisions. Verifier reviews the verdict before finalizing. Reporter writes a structured post-run analysis. Each is independently promptable and swappable. Fixed action vocabulary — tap , long press , swipe , scroll , drag , type , type multi , key , back , dismiss keyboard , wait , assert , done , request manual control , note . The model never emits raw shell commands. Ground-truth screen tracking — Per-step screen changed flag from hierarchy hash comparison, plus Android activity name monitoring via dumpsys . The model knows "which screen am I on" independently of visual appearance. Toast interception — Post-action polling of the accessibility hierarchy catches system Toasts ~2-3.5s window with code-level certainty, not visual guesswork. Loop detection — Tracks repeated identical actions with no screen changed effect; auto-inserts back after 3 repeats to break stuck states. Verification gate — Before accepting a done verdict, a second model pass verifier checks every planned checkpoint against the action history. Can send the session back for more exploration if evidence is thin. Post-run analysis — Reporter sub-agent generates a structured write-up: executive summary, checkpoint breakdown, technical explanation, severity rating, remediation recommendations. Available via API and PDF/CSV export. Containerized Android fleet — Redroid containers with per-container binder device slots, health checks, warm pool management, and clean-state teardown between sessions. Real-time streaming — WebSocket for log entries, step records, status changes, and live device screen mirror during a running session. Degrades to polling fallback if WebSocket fails. Step replay player — Focus view and filmstrip view with touch marker animation, screenshot browsing, and auto-play. Export — PDF report jsPDF with professional layout, verdict, metadata, checkpoint breakdown, steps table, logs table and CSV log export. Session controls — End stop + cancel and restart running sessions from the UI. Device identity profiles — Pixel 7 Pro, Pixel 6, Samsung S24, and more — spoofed via ro.build properties for realistic testing. 100% AMD-powered AI — Fireworks AI on AMD Instinct default or self-hosted vLLM on AMD ROCm. Every call model json in the agent pipeline hits AMD GPUs. php +------------------+ People / CI -- | New Session | APK / URL + task description | Input Form |--------------------------------+ +------------------+ | v +------------------+ +-------------------+ | Fleet Dashboard | <--------------- | Session | | live overview | | Scheduler | +------------------+ +--------+----------+ | +-----------------------------------------------------+------------------------------------------+ v v v v redroid container redroid container redroid container ... up to ~100 + agent sidecar + agent sidecar + agent sidecar observe- decide- act observe- decide- act observe- decide- act | | | +--------- Object storage screenshots, step JPEGs, logs +--------- Results DB Postgres — sessions, steps, log entries, analyses, devices, settings, users | Component | Responsibility | |---|---| New Session form | Accept APK upload or domain URL, task description, device/browser profile, optional context file | Fleet Dashboard | Live overview of every session with status bar, search, filter, sort, pagination | Session Detail | Live screen mirror + structured log stream + replay player + AI analysis panel | Session Scheduler | Dispatch loop 2s tick + fleet loop 10s tick : queued sessions - reserve/boot device - dispatch agent | Android container redroid | One isolated Android 11 instance per session via Docker, with per-container binder device slots | Agent sidecar | Observe - decide - act loop. Sub-agents: planner, executor, verifier, reporter | Device Manager | Container lifecycle: provision, deploy, health-check, teardown, warm pool maintenance | Postgres DB | Source of truth for session metadata, steps, log entries, device state, session analyses, settings, users | File storage | Per-step replay screenshots downscaled JPEG , uploaded APKs, stored on local filesystem | Each session runs a single sidecar process that cycles through five steps: Observe — Capture a screenshot adb screencap via adbutils and the UI element hierarchy uiautomator dump parsed into a flat element list with bounds, text, content-desc, and class . Decide — Send image + hierarchy + task description + action history to the AI model. The model returns one structured action from the fixed vocabulary. Act — Translate the action into ADB calls: input tap , input swipe , input text , input keyevent , etc. Settle — Poll for system Toasts in the ~1s post-action window, then capture a fresh screenshot for the next step. Ground truth check — Per-step hierarchy hash detects whether the screen actually changed; activity name tracking detects navigation independently of visual appearance. The model must explicitly emit assert expected vs. actual comparison and done pass/fail verdict with reasoning — it cannot wander indefinitely. Task + screenshot | v +------------------+ | Planner | Creates 3-6 concrete checkpoints from task + screenshot + APK manifest activities | once | Degrades gracefully to empty list - executor/verifier fall back to free-form task +--------+---------+ | plan checklist v +------------------+ | Executor | Per-step: screenshot + hierarchy + history - one action | per step | Retries on bad generation 3 attempts . Loop detection. Activity tracking. +--------+---------+ | "done" with proposed verdict v +------------------+ | Verifier | Checkpoint-by-checkpoint review against action history + final screenshot | once | Can set should continue=true - executor keeps going. Falls back to executor's verdict. +--------+---------+ | final verdict v +------------------+ | Reporter | Post-verdict write-up: summary, breakdown, technical explanation, severity, recommendations | once | Persisted to DB. Serves Analysis tab + PDF export. Degrades to checkpoint-derived fallback. +------------------+ - User submits an APK or URL , task description, and optional context file through the New Session form. - API creates a queued session row with sequential ID e.g. S9403 . - Session Scheduler's dispatch loop every 2s picks up queued sessions up to the parallelism limit. - Scheduler reserves an idle device or cold-provisions one binds a port + binder slot, boots redroid container, waits for adb + sys.boot completed . - Session transitions booting - initializing - running . The agent sidecar installs the APK if applicable , runs the planner, then begins the observe-decide-act loop. - When the executor emits done , the verifier reviews the evidence. If it passes, the session transitions to analyzing device is released immediately , the reporter generates the write-up, then the session settles into passed or failed . - Artifacts step screenshots, action log, analysis are persisted. The container is torn down. The fleet loop every 10s re-deploys the device for the warm pool. - The Fleet Dashboard and Session Detail reflect the final verdict and analysis in real time. | Status | Meaning | |---|---| queued | Waiting for an available device slot | booting | Android container is starting up | initializing | Installing APK / preparing agent | running | Agent loop is actively executing the task | awaiting questions | Review mode: waiting for user answers to clarifying questions | awaiting approval | Review mode: waiting for user to approve the generated test plan | analyzing | Verdict reached; reporter generating write-up | passed | Task completed successfully | failed | Task completed but assertions failed | stopped | Manually stopped by a user | error | Infrastructure or system error | timed out | Agent exceeded max steps 50 | - Each redroid container runs Android 11 1080x2400, 480dpi with nav bar + soft keyboard disabled via policy control and pm disable-user . - Devices are identified by D-{hex} ID, assigned a host port 5555-5655 range and a binder slot binder/hwbinder/vndbinder triplet for the kernel's binder driver — a hard cap set at modprobe time on the host. - Warm pool: idle, healthy devices maintained up to a user-configurable target. Devices in error state are reaped after 30s backoff. - Health checks run every 10s: container running + sys.boot completed . - Device identity profiles Pixel 7 Pro / Pixel 6 / Samsung S24 are applied via ro.build property overrides before any app touches the device. The backend is a Python FastAPI server at /api . Interactive docs at http://localhost:8000/docs . | Method | Path | Purpose | |---|---|---| POST | /api/sessions | Create a new session multipart form: APK file or URL + task + device profile | GET | /api/sessions | List sessions cursor-paginated, filterable by status/device/build/text search | GET | /api/sessions/stats | Dashboard stats running/queued/passed/failed counts + success rate | GET | /api/sessions/{id} | Get single session details | PATCH | /api/sessions/{id}/control | Control session action: end or restart | POST | /api/sessions/{id}/answers | Submit answers to review clarifying questions | POST | /api/sessions/{id}/approve | Approve review-mode test plan and queue session | POST | /api/sessions/{id}/regenerate-plan | Regenerate test plan from existing answers | POST | /api/sessions/{id}/back-to-questions | Go back to clarifying questions from plan review | GET | /api/sessions/{id}/logs | Get log entries with cursor since | POST | /api/sessions/{id}/logs | Create a log entry | GET | /api/sessions/{id}/steps | Get replay steps with cursor since step | GET | /api/sessions/{id}/steps/{n}/screenshot | Get step screenshot JPEG | GET | /api/sessions/{id}/analysis | Get session analysis write-up | POST | /api/sessions/{id}/analysis/regenerate | Regenerate analysis for a finished session | WS | /api/sessions/{id}/events | WebSocket — real-time log/step/status deltas | WS | /api/sessions/{id}/stream | WebSocket — live screen frame stream PNG | | Method | Path | Purpose | |---|---|---| GET | /api/devices | List all devices | POST | /api/devices | Provision a new device boots redroid container | GET | /api/devices/capacity | Binder slot usage total/used/available | POST | /api/devices/{id}/start | Start an offline device | POST | /api/devices/{id}/stop | Stop a running device | POST | /api/devices/{id}/restart | Restart a device re-deploy container | POST | /api/devices/{id}/maintenance | Toggle maintenance mode | POST | /api/devices/{id}/warm | Adjust warm instance count delta -8 to +8 | DELETE | /api/devices/{id} | Remove device tear down container, free port/slot | | Method | Path | Purpose | |---|---|---| POST | /api/auth/login | Login email + password | POST | /api/auth/signup | Register new user | POST | /api/auth/logout | Logout | | Method | Path | Purpose | |---|---|---| GET | /api/settings | Get app settings workspace, region, notifications, parallelism, API key, warm pool size, max devices | PUT | /api/settings | Update app settings | - Docker Engine and Docker Compose for Postgres and redroid containers - Python 3.11+ - Node.js 20+ - An AI provider API key Fireworks AI on AMD Instinct recommended, or any OpenAI-compatible provider Development hot reload : ./dev.sh Production built frontend, no reload : ./start.sh Both scripts start Postgres if not running , run migrations, and launch the full stack. Ctrl-C stops everything. cd backend cp .env.example .env Edit .env to set your AI API KEY and any other settings. docker compose up -d Or manually: docker run -d --name veta-pg \ -e POSTGRES USER=postgres \ -e POSTGRES PASSWORD=postgres \ -e POSTGRES DB=veta \ -p 5432:5432 \ postgres:16-alpine cd backend pip install -e . alembic upgrade head uvicorn app.main:app --reload --host 0.0.0.0 --port 8000 The API starts at http://localhost:8000 . Docs at http://localhost:8000/docs . On first boot, the server auto-seeds admin user admin / admin and default settings. cd frontend npm install npm run dev The dashboard starts at http://localhost:8080 . Login with any email/password. Set VITE API URL to change the backend address default http://localhost:8000 . Load binder kernel module adjust device count for your host sudo modprobe binder linux devices="binder,hwbinder,vndbinder" Example for 3 concurrent devices: devices="binder,hwbinder,vndbinder,binder1,hwbinder1,vndbinder1,binder2,hwbinder2,vndbinder2" Ensure adb is installed and accessible adb devices Set required env vars export REDROID IMAGE=redroid/redroid:11.0.0-latest | Variable | Default | Description | |---|---|---| DATABASE URL | postgresql+asyncpg://postgres:postgres@localhost:5432/veta | Postgres connection string | REDROID IMAGE | redroid/redroid:11.0.0-latest | Android container image | ADB HOST | localhost | ADB host address | REDROID PORT START | 5555 | Start of port range for device ADB | REDROID PORT END | 5655 | End of port range up to ~100 devices | BINDER SLOTS | auto-detected from /dev | Override for binder slot count | AI API KEY | -- | AI provider API key also configurable via Settings UI | AI PROVIDER | fireworks | AI provider: fireworks , veta-ai , google-gemini , openrouter , deepseek , anthropic | AI MODEL | provider default | Model name override | MODEL ENDPOINT | provider default | API base URL override | veta/ Docs/ Design documents, wireframes, logo server/ vLLM inference server AMD ROCm veta inference server.ipynb Notebook: serve vision models on AMD GPUs via vLLM bin/ cloudflared binary for tunnel backend/ FastAPI backend API app/ main.py FastAPI entrypoint + lifespan auto-seed, stale cleanup, dispatch + fleet loops config.py pydantic-settings .env / env vars database.py Async SQLAlchemy engine + session factory dispatcher.py Session dispatch loop + fleet health loop device manager.py Redroid container lifecycle, port/binder allocation, warm pool, health checks device events.py SSE device state change notifications apk utils.py APK metadata extraction aapt + apkutils screenshot store.py Per-step screenshot downscale + JPEG storage models/ SQLAlchemy ORM models session, step, log entry, device, settings, session analysis, user schemas/ Pydantic request/response schemas camelCase JSON routers/ REST + WebSocket endpoint handlers sessions, steps, logs, devices, auth, settings, stream services/ session analysis.py Build + store session analysis bridges reporter sub-agent to DB adapters/ Port implementations ApkInspectorImpl, etc. alembic/ Database migrations scripts/ seed.py Seed admin user + default settings uploads/ Uploaded APKs + step screenshot storage agent/ AI agent sidecar agents/ base.py Shared retry logic generate with retries planner.py Pre-execution checkpoint builder executor.py Per-step action decision verifier.py Post-decision verification reporter.py Post-verdict write-up core/ adb.py adbutils device handle device.py Screenshots, action execution, marker extraction, activity tracking hierarchy.py XML hierarchy parsing, element signature hashing model client.py Multi-provider AI client Gemini SDK + OpenAI-compat with retry + JSON extraction apk.py APK metadata extraction, install/launch on device cancellation.py Thread-safe session cancellation thinking hooks.py Real-time thinking indicator callbacks modes.py Mode configs review/run + allowed action groups skill text.py Loads agent playbook into system prompt ports/ Protocol interfaces for dependency injection apk inspector.py ApkInspector protocol get all activities, get apk details skills/ Agent playbook loaded into system context SKILL.md Master playbook — decision loop, confidence/targeting rules, verdict discipline actions.md Full action vocabulary with semantics for each action type elements.md UI element taxonomy — how to recognize and interact with different UI components scenarios.md Step-by-step playbooks for authentication, permissions, onboarding, forms, search debugging.md Recovery strategies for stuck states, loop detection, unexpected screens memory.py SessionMemory — action history, loop detection, activity tracking, screen stack orchestrator.py Session state machine wiring all sub-agents together booting - ... - passed/failed phases.py Phase-based lifecycle SetupPhase, ReviewPhase, RunPhase runner.py Compatibility entrypoint delegates to orchestrator frontend/ TanStack Start web dashboard src/ routes/ File-based routing root.tsx Root layout + sidebar + auth index.tsx Fleet Dashboard stat bar, search, filter, sort, pagination, session cards session.$id.tsx Session Detail live mirror, log stream, replay player, analysis panel, export, filmstrip new-session.tsx New Session form APK/URL toggle, file drag-drop, context attachment, device profile devices.tsx Device Farm card grid, provision dialog, warm pool controls, binder tracking history.tsx Run History table view, sort, filter settings.tsx Settings workspace, execution, API key, notifications, integrations login.tsx / signup.tsx Auth pages components/ session-card.tsx Fleet dashboard session card session-analysis-panel.tsx Analysis tab UI ui/ shadcn/ui primitives 50+ components lib/ api.ts API client fetch + WebSocket wrappers, all endpoints auth.ts Session storage auth session-report.ts PDF + CSV export jsPDF with professional layout public/ Static assets favicon, logo Located in agent/agents/ . Each is independently promptable and can be retried or swapped without affecting the others. | Agent | File | Purpose | |---|---|---| planner | planner.py | Builds 3-6 concrete checkpoints from task + first screenshot + APK manifest . Runs once before execution. | executor | executor.py | Per-step decision: screenshot + hierarchy + history - one action. Retries on bad generation 3 attempts . | verifier | verifier.py | Checkpoint-by-checkpoint review against action history. Can send session back for more exploration. | reporter | reporter.py | Post-verdict structured write-up. Falls back to checkpoint-derived summary on model failure. | Located in agent/core/ . | Module | Purpose | |---|---| device | Screenshots, action execution dispatch, activity tracking, marker extraction | hierarchy | UiAutomator hierarchy dump + parsing + element signature hashing | model client | Multi-provider HTTP client Gemini SDK + OpenAI-compat, Fireworks on AMD Instinct , JSON extraction, retry logic | apk | APK install, package discovery, app launch on device | adb | Shared adbutils device handle | cancellation | Thread-safe session cancellation registry | modes | Mode configs review/run + action group definitions | skill text | Agent playbook loaded into the executor's system prompt | | Provider | SDK | Backend hardware | Models | Status | |---|---|---|---|---| Fireworks AI | openai | AMD Instinct MI250/MI300X | qwen3.7-plus , firefunction-v2 , Gemma 4 | ✅ Default 100% AMD | Veta AI self-hosted | openai | AMD ROCm vLLM | Any HF vision model Qwen2.5-VL, Gemma, Llama | ✅ 100% AMD | Google Gemini | google-genai | Google TPU | gemma-4-26b-a4b-it | ⬜ Optional | OpenRouter | openai | Varies | openai/gpt-4o-mini , any routed model | ⬜ Optional | Deepseek | openai | Varies | deepseek-v4-flash | ⬜ Optional | Anthropic | openai | Varies | Anthropic models via OpenAI-compat endpoint | ⬜ Optional | 100% of inference runs on AMD out of the box.The two AMD-backed providers Fireworks + self-hosted ROCm handle every agent call. Gemini, OpenRouter, Deepseek, and Anthropic are optional alternatives — none are enabled by default. The agent playbook agent/skills/ is loaded into the executor's system context: | File | Purpose | |---|---| SKILL.md | Master playbook — decision loop, confidence/targeting rules, verdict discipline | actions.md | Full action vocabulary with semantics for each action type | elements.md | UI element taxonomy — recognizing and interacting with buttons, sliders, toggles, dropdowns, dialogs | scenarios.md | Step-by-step playbooks for authentication, permissions, onboarding, forms, search | debugging.md | Recovery strategies for stuck states, loop detection, unexpected screens | | Layer | Technology | |---|---| Backend | Python 3.11+, FastAPI, SQLAlchemy async , Alembic, PostgreSQL | Agent loop | Python, httpx, Pillow, google-genai SDK, openai SDK | Frontend | React 19, TanStack Start, TanStack Router, Tailwind CSS v4, shadcn/ui | AI inference cloud, default | Fireworks AI — AMD Instinct MI250/MI300X 100% AMD | AI inference self-hosted | vLLM + AMD ROCm 100% AMD | Android runtime | Docker, redroid Android 11 , ADB, adbutils, uiautomator | Infrastructure | Docker, Docker Compose dev , Kubernetes target | Real-time | WebSocket live frame stream + events bus , SSE fallback | Export | jsPDF + jspdf-autotable PDF , CSV | | Approach | Cost per session ~50 steps | When to use | |---|---|---| Fireworks API AMD Instinct | ~$0.02-0.05 | Development, low-volume QA | Self-hosted vLLM on ROCm | GPU electricity only | Production, high-volume CI/CD | Gemini CachedContent | ~$0.01 95% cached | Optional fallback / multi-provider |