# Veta: AI agent that QA-tests Android apps

> Source: <https://github.com/Vip3r-MC/Veta>
> Published: 2026-07-16 14:03:48+00:00

*Autonomous AI agent swarm for visual, functional, and accessibility testing of Android apps and mobile web — 100% of AI inference runs on AMD GPUs (Fireworks AI on AMD Instinct / self-hosted vLLM on ROCm), with containerized Android instances running locally.*

[Overview](#overview) • [Features](#features) • [Architecture](#architecture) • [How it works](#how-it-works) • [API](#api) • [Getting started](#getting-started) • [Project structure](#project-structure) • [Tech stack](#tech-stack)

Built for theNo benchmarks, no constraints, just build: give an AI model eyes and hands, and let it QA your Android apps.[AMD Developer Hackathon ACT II]— Unicorn Track.

Veta replaces brittle, scripted UI tests with an **AI-driven QA agent that watches an Android screen**, decides what to do next based on a plain-English task description, and executes actions via ADB. Each run produces a pass/fail verdict, a complete action trace, and video/log artifacts — the same output a human QA tester would produce, at machine speed.

Sessions run in parallel across a fleet of containerized Android instances (redroid), managed by a scheduler with a warm pool and autoscaling. A web dashboard lets humans launch sessions, watch live screen mirrors, and review results.

| Input type | Description |
|---|---|
APK |
Upload an Android APK, install it on a clean Android instance, and run a task against the native app |
Web domain |
Provide a URL to test inside a sandboxed mobile browser, with optional login credentials |
Context file |
Attach `.txt` , `.md` , `.docx` , or `.pdf` checklist alongside the task — contents are folded into the agent's working prompt at runtime |

The system starts at **~10 concurrent sessions** on a single host and scales to **~100 concurrent sessions** on Kubernetes.

**Every single model call — planning, execution, verification, reporting — runs on AMD GPUs.** Zero inference happens on non-AMD hardware by default.

| Path | What runs on AMD | Hardware |
|---|---|---|
Fireworks AI (default) |
Planner, executor, verifier, reporter | AMD Instinct MI250/MI300X |
Self-hosted vLLM + ROCm |
Same pipeline, your hardware | Any AMD GPU via ROCm |

The agent has no fallback to non-AMD inference providers. Gemini, OpenRouter, and Anthropic are listed as optional alternatives in the settings UI — none are enabled or used by default. Out of the box, **100% of AI compute is AMD.**

**Sub-agent architecture**— Planner creates a checklist before execution. Executor drives per-step decisions. Verifier reviews the verdict before finalizing. Reporter writes a structured post-run analysis. Each is independently promptable and swappable.**Fixed action vocabulary**—`tap`

,`long_press`

,`swipe`

,`scroll`

,`drag`

,`type`

,`type_multi`

,`key`

,`back`

,`dismiss_keyboard`

,`wait`

,`assert`

,`done`

,`request_manual_control`

,`note`

. The model never emits raw shell commands.**Ground-truth screen tracking**— Per-step`screen_changed`

flag from hierarchy hash comparison, plus Android activity name monitoring via`dumpsys`

. The model knows "which screen am I on" independently of visual appearance.**Toast interception**— Post-action polling of the accessibility hierarchy catches system Toasts (~2-3.5s window) with code-level certainty, not visual guesswork.**Loop detection**— Tracks repeated identical actions with no`screen_changed`

effect; auto-inserts`back`

after 3 repeats to break stuck states.**Verification gate**— Before accepting a`done`

verdict, a second model pass (verifier) checks every planned checkpoint against the action history. Can send the session back for more exploration if evidence is thin.**Post-run analysis**— Reporter sub-agent generates a structured write-up: executive summary, checkpoint breakdown, technical explanation, severity rating, remediation recommendations. Available via API and PDF/CSV export.**Containerized Android fleet**— Redroid containers with per-container binder device slots, health checks, warm pool management, and clean-state teardown between sessions.**Real-time streaming**— WebSocket for log entries, step records, status changes, and live device screen mirror during a running session. Degrades to polling fallback if WebSocket fails.**Step replay player**— Focus view and filmstrip view with touch marker animation, screenshot browsing, and auto-play.** Export**— PDF report (jsPDF with professional layout, verdict, metadata, checkpoint breakdown, steps table, logs table) and CSV log export.**Session controls**— End (stop + cancel) and restart running sessions from the UI.** Device identity profiles**— Pixel 7 Pro, Pixel 6, Samsung S24, and more — spoofed via ro.build properties for realistic testing.** 100% AMD-powered AI**— Fireworks AI on AMD Instinct (default) or self-hosted vLLM on AMD ROCm. Every`call_model_json()`

in the agent pipeline hits AMD GPUs.

``` php
                      +------------------+
      People / CI --> |  New Session     |  APK / URL + task description
                      |  Input Form      |--------------------------------+
                      +------------------+                                |
                                                                          v
                      +------------------+                      +-------------------+
                      |  Fleet Dashboard |  <--------------->   |  Session          |
                      |  (live overview) |                      |  Scheduler        |
                      +------------------+                      +--------+----------+
                                                                          |
                      +-----------------------------------------------------+------------------------------------------+
                      v                          v                          v                                          v
               redroid container           redroid container           redroid container                    ... up to ~100
               + agent sidecar             + agent sidecar             + agent sidecar
               (observe->decide->act)      (observe->decide->act)      (observe->decide->act)
                      |                          |                          |
                      +---------> Object storage (screenshots, step JPEGs, logs)
                      +---------> Results DB (Postgres — sessions, steps, log entries, analyses, devices, settings, users)
```

| Component | Responsibility |
|---|---|
New Session form |
Accept APK upload or domain URL, task description, device/browser profile, optional context file |
Fleet Dashboard |
Live overview of every session with status bar, search, filter, sort, pagination |
Session Detail |
Live screen mirror + structured log stream + replay player + AI analysis panel |
Session Scheduler |
Dispatch loop (2s tick) + fleet loop (10s tick): queued sessions -> reserve/boot device -> dispatch agent |
Android container (redroid) |
One isolated Android 11 instance per session via Docker, with per-container binder device slots |
Agent sidecar |
Observe -> decide -> act loop. Sub-agents: planner, executor, verifier, reporter |
Device Manager |
Container lifecycle: provision, deploy, health-check, teardown, warm pool maintenance |
Postgres DB |
Source of truth for session metadata, steps, log entries, device state, session analyses, settings, users |
File storage |
Per-step replay screenshots (downscaled JPEG), uploaded APKs, stored on local filesystem |

Each session runs a single sidecar process that cycles through five steps:

**Observe**— Capture a screenshot (`adb screencap`

via adbutils) and the UI element hierarchy (`uiautomator dump`

parsed into a flat element list with bounds, text, content-desc, and class).**Decide**— Send image + hierarchy + task description + action history to the AI model. The model returns one structured action from the fixed vocabulary.**Act**— Translate the action into ADB calls:`input tap`

,`input swipe`

,`input text`

,`input keyevent`

, etc.**Settle**— Poll for system Toasts in the ~1s post-action window, then capture a fresh screenshot for the next step.** Ground truth check**— Per-step hierarchy hash detects whether the screen actually changed; activity name tracking detects navigation independently of visual appearance.

The model must explicitly emit `assert`

(expected vs. actual comparison) and `done`

(pass/fail verdict with reasoning) — it cannot wander indefinitely.

```
Task + screenshot
    |
    v
+------------------+
|  Planner         |  Creates 3-6 concrete checkpoints from task + screenshot (+ APK manifest activities)
|  (once)          |  Degrades gracefully to empty list -> executor/verifier fall back to free-form task
+--------+---------+
         | plan checklist
         v
+------------------+
|  Executor        |  Per-step: screenshot + hierarchy + history -> one action
|  (per step)      |  Retries on bad generation (3 attempts). Loop detection. Activity tracking.
+--------+---------+
         | "done" with proposed verdict
         v
+------------------+
|  Verifier        |  Checkpoint-by-checkpoint review against action history + final screenshot
|  (once)          |  Can set should_continue=true -> executor keeps going. Falls back to executor's verdict.
+--------+---------+
         | final verdict
         v
+------------------+
|  Reporter        |  Post-verdict write-up: summary, breakdown, technical explanation, severity, recommendations
|  (once)          |  Persisted to DB. Serves Analysis tab + PDF export. Degrades to checkpoint-derived fallback.
+------------------+
```

- User submits an APK (or URL), task description, and optional context file through the New Session form.
- API creates a
`queued`

session row with sequential ID (e.g.`S9403`

). - Session Scheduler's dispatch loop (every 2s) picks up queued sessions up to the parallelism limit.
- Scheduler reserves an idle device or cold-provisions one (binds a port + binder slot, boots redroid container, waits for adb +
`sys.boot_completed`

). - Session transitions
`booting`

->`initializing`

->`running`

. The agent sidecar installs the APK (if applicable), runs the planner, then begins the observe-decide-act loop. - When the executor emits
`done`

, the verifier reviews the evidence. If it passes, the session transitions to`analyzing`

(device is released immediately), the reporter generates the write-up, then the session settles into`passed`

or`failed`

. - Artifacts (step screenshots, action log, analysis) are persisted. The container is torn down. The fleet loop (every 10s) re-deploys the device for the warm pool.
- The Fleet Dashboard and Session Detail reflect the final verdict and analysis in real time.

| Status | Meaning |
|---|---|
`queued` |
Waiting for an available device slot |
`booting` |
Android container is starting up |
`initializing` |
Installing APK / preparing agent |
`running` |
Agent loop is actively executing the task |
`awaiting_questions` |
Review mode: waiting for user answers to clarifying questions |
`awaiting_approval` |
Review mode: waiting for user to approve the generated test plan |
`analyzing` |
Verdict reached; reporter generating write-up |
`passed` |
Task completed successfully |
`failed` |
Task completed but assertions failed |
`stopped` |
Manually stopped by a user |
`error` |
Infrastructure or system error |
`timed_out` |
Agent exceeded max steps (50) |

- Each redroid container runs Android 11 (1080x2400, 480dpi) with nav bar + soft keyboard disabled via
`policy_control`

and`pm disable-user`

. - Devices are identified by
`D-{hex}`

ID, assigned a host port (5555-5655 range) and a**binder slot**(binder/hwbinder/vndbinder triplet) for the kernel's binder driver — a hard cap set at`modprobe`

time on the host. - Warm pool: idle, healthy devices maintained up to a user-configurable target. Devices in
`error`

state are reaped after 30s backoff. - Health checks run every 10s: container running +
`sys.boot_completed`

. - Device identity profiles (Pixel 7 Pro / Pixel 6 / Samsung S24) are applied via
`ro.build`

property overrides before any app touches the device.

The backend is a Python FastAPI server at `/api`

. Interactive docs at `http://localhost:8000/docs`

.

| Method | Path | Purpose |
|---|---|---|
`POST` |
`/api/sessions` |
Create a new session (multipart form: APK file or URL + task + device profile) |
`GET` |
`/api/sessions` |
List sessions (cursor-paginated, filterable by status/device/build/text search) |
`GET` |
`/api/sessions/stats` |
Dashboard stats (running/queued/passed/failed counts + success rate) |
`GET` |
`/api/sessions/{id}` |
Get single session details |
`PATCH` |
`/api/sessions/{id}/control` |
Control session (action: `end` or `restart` ) |
`POST` |
`/api/sessions/{id}/answers` |
Submit answers to review clarifying questions |
`POST` |
`/api/sessions/{id}/approve` |
Approve review-mode test plan and queue session |
`POST` |
`/api/sessions/{id}/regenerate-plan` |
Regenerate test plan from existing answers |
`POST` |
`/api/sessions/{id}/back-to-questions` |
Go back to clarifying questions from plan review |
`GET` |
`/api/sessions/{id}/logs` |
Get log entries with cursor (`since` ) |
`POST` |
`/api/sessions/{id}/logs` |
Create a log entry |
`GET` |
`/api/sessions/{id}/steps` |
Get replay steps with cursor (`since_step` ) |
`GET` |
`/api/sessions/{id}/steps/{n}/screenshot` |
Get step screenshot (JPEG) |
`GET` |
`/api/sessions/{id}/analysis` |
Get session analysis write-up |
`POST` |
`/api/sessions/{id}/analysis/regenerate` |
Regenerate analysis for a finished session |
`WS` |
`/api/sessions/{id}/events` |
WebSocket — real-time log/step/status deltas |
`WS` |
`/api/sessions/{id}/stream` |
WebSocket — live screen frame stream (PNG) |

| Method | Path | Purpose |
|---|---|---|
`GET` |
`/api/devices` |
List all devices |
`POST` |
`/api/devices` |
Provision a new device (boots redroid container) |
`GET` |
`/api/devices/capacity` |
Binder slot usage (total/used/available) |
`POST` |
`/api/devices/{id}/start` |
Start an offline device |
`POST` |
`/api/devices/{id}/stop` |
Stop a running device |
`POST` |
`/api/devices/{id}/restart` |
Restart a device (re-deploy container) |
`POST` |
`/api/devices/{id}/maintenance` |
Toggle maintenance mode |
`POST` |
`/api/devices/{id}/warm` |
Adjust warm instance count (delta -8 to +8) |
`DELETE` |
`/api/devices/{id}` |
Remove device (tear down container, free port/slot) |

| Method | Path | Purpose |
|---|---|---|
`POST` |
`/api/auth/login` |
Login (email + password) |
`POST` |
`/api/auth/signup` |
Register new user |
`POST` |
`/api/auth/logout` |
Logout |

| Method | Path | Purpose |
|---|---|---|
`GET` |
`/api/settings` |
Get app settings (workspace, region, notifications, parallelism, API key, warm pool size, max devices) |
`PUT` |
`/api/settings` |
Update app settings |

- Docker Engine and Docker Compose (for Postgres and redroid containers)
- Python 3.11+
- Node.js 20+
- An AI provider API key (Fireworks AI on AMD Instinct recommended, or any OpenAI-compatible provider)

**Development** (hot reload):

```
./dev.sh
```

**Production** (built frontend, no reload):

```
./start.sh
```

Both scripts start Postgres (if not running), run migrations, and launch the full stack. `Ctrl-C`

stops everything.

```
cd backend
cp .env.example .env
```

Edit `.env`

to set your `AI_API_KEY`

and any other settings.

```
docker compose up -d
```

Or manually:

```
docker run -d --name veta-pg \
  -e POSTGRES_USER=postgres \
  -e POSTGRES_PASSWORD=postgres \
  -e POSTGRES_DB=veta \
  -p 5432:5432 \
  postgres:16-alpine
cd backend
pip install -e .
alembic upgrade head
uvicorn app.main:app --reload --host 0.0.0.0 --port 8000
```

The API starts at `http://localhost:8000`

. Docs at `http://localhost:8000/docs`

.

On first boot, the server auto-seeds admin user (

`admin`

/`admin`

) and default settings.

```
cd frontend
npm install
npm run dev
```

The dashboard starts at `http://localhost:8080`

. Login with any email/password.

Set

`VITE_API_URL`

to change the backend address (default`http://localhost:8000`

).

```
# Load binder kernel module (adjust device count for your host)
sudo modprobe binder_linux devices="binder,hwbinder,vndbinder"

# Example for 3 concurrent devices:
# devices="binder,hwbinder,vndbinder,binder1,hwbinder1,vndbinder1,binder2,hwbinder2,vndbinder2"

# Ensure adb is installed and accessible
adb devices

# Set required env vars
export REDROID_IMAGE=redroid/redroid:11.0.0-latest
```

| Variable | Default | Description |
|---|---|---|
`DATABASE_URL` |
`postgresql+asyncpg://postgres:postgres@localhost:5432/veta` |
Postgres connection string |
`REDROID_IMAGE` |
`redroid/redroid:11.0.0-latest` |
Android container image |
`ADB_HOST` |
`localhost` |
ADB host address |
`REDROID_PORT_START` |
`5555` |
Start of port range for device ADB |
`REDROID_PORT_END` |
`5655` |
End of port range (up to ~100 devices) |
`BINDER_SLOTS` |
auto-detected from `/dev` |
Override for binder slot count |
`AI_API_KEY` |
-- | AI provider API key (also configurable via Settings UI) |
`AI_PROVIDER` |
`fireworks` |
AI provider: `fireworks` , `veta-ai` , `google-gemini` , `openrouter` , `deepseek` , `anthropic` |
`AI_MODEL` |
provider default | Model name override |
`MODEL_ENDPOINT` |
provider default | API base URL override |

```
veta/
  Docs/                        Design documents, wireframes, logo
  server/                      vLLM inference server (AMD ROCm)
    veta_inference_server.ipynb  Notebook: serve vision models on AMD GPUs via vLLM
    bin/                         cloudflared binary for tunnel

  backend/                     FastAPI backend API
    app/
      main.py                  FastAPI entrypoint + lifespan (auto-seed, stale cleanup, dispatch + fleet loops)
      config.py                pydantic-settings (.env / env vars)
      database.py              Async SQLAlchemy engine + session factory
      dispatcher.py            Session dispatch loop + fleet health loop
      device_manager.py        Redroid container lifecycle, port/binder allocation, warm pool, health checks
      device_events.py         SSE device state change notifications
      apk_utils.py             APK metadata extraction (aapt + apkutils)
      screenshot_store.py      Per-step screenshot downscale + JPEG storage
      models/                  SQLAlchemy ORM models (session, step, log_entry, device, settings, session_analysis, user)
      schemas/                 Pydantic request/response schemas (camelCase JSON)
      routers/                 REST + WebSocket endpoint handlers (sessions, steps, logs, devices, auth, settings, stream)
      services/
        session_analysis.py    Build + store session analysis (bridges reporter sub-agent to DB)
    adapters/                  Port implementations (ApkInspectorImpl, etc.)
    alembic/                   Database migrations
    scripts/
      seed.py                  Seed admin user + default settings
    uploads/                   Uploaded APKs + step screenshot storage

  agent/                       AI agent sidecar
    agents/
      base.py                  Shared retry logic (generate_with_retries)
      planner.py               Pre-execution checkpoint builder
      executor.py              Per-step action decision
      verifier.py              Post-decision verification
      reporter.py              Post-verdict write-up
    core/
      adb.py                   adbutils device handle
      device.py                Screenshots, action execution, marker extraction, activity tracking
      hierarchy.py             XML hierarchy parsing, element signature hashing
      model_client.py          Multi-provider AI client (Gemini SDK + OpenAI-compat) with retry + JSON extraction
      apk.py                   APK metadata extraction, install/launch on device
      cancellation.py          Thread-safe session cancellation
      thinking_hooks.py        Real-time thinking indicator callbacks
      modes.py                 Mode configs (review/run) + allowed action groups
      skill_text.py            Loads agent playbook into system prompt
    ports/                     Protocol interfaces for dependency injection
      apk_inspector.py         ApkInspector protocol (get_all_activities, get_apk_details)
    skills/                    Agent playbook (loaded into system context)
      SKILL.md                 Master playbook — decision loop, confidence/targeting rules, verdict discipline
      actions.md               Full action vocabulary with semantics for each action type
      elements.md              UI element taxonomy — how to recognize and interact with different UI components
      scenarios.md             Step-by-step playbooks for authentication, permissions, onboarding, forms, search
      debugging.md             Recovery strategies for stuck states, loop detection, unexpected screens
    memory.py                  SessionMemory — action history, loop detection, activity tracking, screen stack
    orchestrator.py            Session state machine wiring all sub-agents together (booting -> ... -> passed/failed)
    phases.py                  Phase-based lifecycle (SetupPhase, ReviewPhase, RunPhase)
    runner.py                  Compatibility entrypoint (delegates to orchestrator)

  frontend/                    TanStack Start web dashboard
    src/
      routes/                  File-based routing
        __root.tsx             Root layout + sidebar + auth
        index.tsx              Fleet Dashboard (stat bar, search, filter, sort, pagination, session cards)
        session.$id.tsx        Session Detail (live mirror, log stream, replay player, analysis panel, export, filmstrip)
        new-session.tsx        New Session form (APK/URL toggle, file drag-drop, context attachment, device profile)
        devices.tsx            Device Farm (card grid, provision dialog, warm pool controls, binder tracking)
        history.tsx            Run History (table view, sort, filter)
        settings.tsx           Settings (workspace, execution, API key, notifications, integrations)
        login.tsx / signup.tsx Auth pages
      components/
        session-card.tsx       Fleet dashboard session card
        session-analysis-panel.tsx  Analysis tab UI
        ui/                    shadcn/ui primitives (50+ components)
      lib/
        api.ts                 API client (fetch + WebSocket wrappers, all endpoints)
        auth.ts                Session storage auth
        session-report.ts      PDF + CSV export (jsPDF with professional layout)
    public/                    Static assets (favicon, logo)
```

Located in `agent/agents/`

. Each is independently promptable and can be retried or swapped without affecting the others.

| Agent | File | Purpose |
|---|---|---|
planner |
`planner.py` |
Builds 3-6 concrete checkpoints from task + first screenshot (+ APK manifest). Runs once before execution. |
executor |
`executor.py` |
Per-step decision: screenshot + hierarchy + history -> one action. Retries on bad generation (3 attempts). |
verifier |
`verifier.py` |
Checkpoint-by-checkpoint review against action history. Can send session back for more exploration. |
reporter |
`reporter.py` |
Post-verdict structured write-up. Falls back to checkpoint-derived summary on model failure. |

Located in `agent/core/`

.

| Module | Purpose |
|---|---|
`device` |
Screenshots, action execution dispatch, activity tracking, marker extraction |
`hierarchy` |
UiAutomator hierarchy dump + parsing + element signature hashing |
`model_client` |
Multi-provider HTTP client (Gemini SDK + OpenAI-compat, Fireworks on AMD Instinct), JSON extraction, retry logic |
`apk` |
APK install, package discovery, app launch on device |
`adb` |
Shared adbutils device handle |
`cancellation` |
Thread-safe session cancellation registry |
`modes` |
Mode configs (review/run) + action group definitions |
`skill_text` |
Agent playbook loaded into the executor's system prompt |

| Provider | SDK | Backend hardware | Models | Status |
|---|---|---|---|---|
Fireworks AI |
`openai` |
AMD Instinct MI250/MI300X |
`qwen3.7-plus` , `firefunction-v2` , Gemma 4 |
✅ Default (100% AMD) |
Veta AI (self-hosted) |
`openai` |
AMD ROCm (vLLM) |
Any HF vision model (Qwen2.5-VL, Gemma, Llama) | ✅ 100% AMD |
Google Gemini |
`google-genai` |
Google TPU | `gemma-4-26b-a4b-it` |
⬜ Optional |
OpenRouter |
`openai` |
Varies | `openai/gpt-4o-mini` , any routed model |
⬜ Optional |
Deepseek |
`openai` |
Varies | `deepseek-v4-flash` |
⬜ Optional |
Anthropic |
`openai` |
Varies | Anthropic models via OpenAI-compat endpoint | ⬜ Optional |

100% of inference runs on AMD out of the box.The two AMD-backed providers (Fireworks + self-hosted ROCm) handle every agent call. Gemini, OpenRouter, Deepseek, and Anthropic are optional alternatives — none are enabled by default.

The agent playbook (`agent/skills/`

) is loaded into the executor's system context:

| File | Purpose |
|---|---|
`SKILL.md` |
Master playbook — decision loop, confidence/targeting rules, verdict discipline |
`actions.md` |
Full action vocabulary with semantics for each action type |
`elements.md` |
UI element taxonomy — recognizing and interacting with buttons, sliders, toggles, dropdowns, dialogs |
`scenarios.md` |
Step-by-step playbooks for authentication, permissions, onboarding, forms, search |
`debugging.md` |
Recovery strategies for stuck states, loop detection, unexpected screens |

| Layer | Technology |
|---|---|
Backend |
Python 3.11+, FastAPI, SQLAlchemy (async), Alembic, PostgreSQL |
Agent loop |
Python, httpx, Pillow, google-genai SDK, openai SDK |
Frontend |
React 19, TanStack Start, TanStack Router, Tailwind CSS v4, shadcn/ui |
AI inference (cloud, default) |
Fireworks AI — AMD Instinct MI250/MI300X (100% AMD) |
AI inference (self-hosted) |
vLLM + AMD ROCm (100% AMD) |
Android runtime |
Docker, redroid (Android 11), ADB, adbutils, uiautomator |
Infrastructure |
Docker, Docker Compose (dev), Kubernetes (target) |
Real-time |
WebSocket (live frame stream + events bus), SSE fallback |
Export |
jsPDF + jspdf-autotable (PDF), CSV |

| Approach | Cost per session (~50 steps) | When to use |
|---|---|---|
Fireworks API (AMD Instinct) |
~$0.02-0.05 | Development, low-volume QA |
Self-hosted vLLM on ROCm |
GPU electricity only | Production, high-volume CI/CD |
Gemini CachedContent |
~$0.01 (95% cached) | Optional fallback / multi-provider |