Autonomous AI agent swarm for visual, functional, and accessibility testing of Android apps and mobile web — 100% of AI inference runs on AMD GPUs (Fireworks AI on AMD Instinct / self-hosted vLLM on ROCm), with containerized Android instances running locally.
Overview • Features • Architecture • How it works • API • Getting started • Project structure • Tech stack
Built for theNo benchmarks, no constraints, just build: give an AI model eyes and hands, and let it QA your Android apps.[AMD Developer Hackathon ACT II]— Unicorn Track.
Veta replaces brittle, scripted UI tests with an AI-driven QA agent that watches an Android screen, decides what to do next based on a plain-English task description, and executes actions via ADB. Each run produces a pass/fail verdict, a complete action trace, and video/log artifacts — the same output a human QA tester would produce, at machine speed.
Sessions run in parallel across a fleet of containerized Android instances (redroid), managed by a scheduler with a warm pool and autoscaling. A web dashboard lets humans launch sessions, watch live screen mirrors, and review results.
| Input type | Description |
|---|---|
| APK | |
| Upload an Android APK, install it on a clean Android instance, and run a task against the native app | |
| Web domain | |
| Provide a URL to test inside a sandboxed mobile browser, with optional login credentials | |
| Context file | |
Attach .txt , .md , .docx , or .pdf checklist alongside the task — contents are folded into the agent's working prompt at runtime |
The system starts at ~10 concurrent sessions on a single host and scales to ~100 concurrent sessions on Kubernetes.
Every single model call — planning, execution, verification, reporting — runs on AMD GPUs. Zero inference happens on non-AMD hardware by default.
| Path | What runs on AMD | Hardware |
|---|---|---|
| Fireworks AI (default) | ||
| Planner, executor, verifier, reporter | AMD Instinct MI250/MI300X | |
| Self-hosted vLLM + ROCm | ||
| Same pipeline, your hardware | Any AMD GPU via ROCm |
The agent has no fallback to non-AMD inference providers. Gemini, OpenRouter, and Anthropic are listed as optional alternatives in the settings UI — none are enabled or used by default. Out of the box, 100% of AI compute is AMD.
Sub-agent architecture— Planner creates a checklist before execution. Executor drives per-step decisions. Verifier reviews the verdict before finalizing. Reporter writes a structured post-run analysis. Each is independently promptable and swappable.Fixed action vocabulary—tap
,long_press
,swipe
,scroll
,drag
,type
,type_multi
,key
,back
,dismiss_keyboard
,wait
,assert
,done
,request_manual_control
,note
. The model never emits raw shell commands.Ground-truth screen tracking— Per-stepscreen_changed
flag from hierarchy hash comparison, plus Android activity name monitoring viadumpsys
. The model knows "which screen am I on" independently of visual appearance.Toast interception— Post-action polling of the accessibility hierarchy catches system Toasts (~2-3.5s window) with code-level certainty, not visual guesswork.Loop detection— Tracks repeated identical actions with noscreen_changed
effect; auto-insertsback
after 3 repeats to break stuck states.Verification gate— Before accepting adone
verdict, a second model pass (verifier) checks every planned checkpoint against the action history. Can send the session back for more exploration if evidence is thin.Post-run analysis— Reporter sub-agent generates a structured write-up: executive summary, checkpoint breakdown, technical explanation, severity rating, remediation recommendations. Available via API and PDF/CSV export.Containerized Android fleet— Redroid containers with per-container binder device slots, health checks, warm pool management, and clean-state teardown between sessions.Real-time streaming— WebSocket for log entries, step records, status changes, and live device screen mirror during a running session. Degrades to polling fallback if WebSocket fails.Step replay player— Focus view and filmstrip view with touch marker animation, screenshot browsing, and auto-play.** Export**— PDF report (jsPDF with professional layout, verdict, metadata, checkpoint breakdown, steps table, logs table) and CSV log export.Session controls— End (stop + cancel) and restart running sessions from the UI.** Device identity profiles**— Pixel 7 Pro, Pixel 6, Samsung S24, and more — spoofed via ro.build properties for realistic testing.** 100% AMD-powered AI**— Fireworks AI on AMD Instinct (default) or self-hosted vLLM on AMD ROCm. Everycall_model_json()
in the agent pipeline hits AMD GPUs.
+------------------+
People / CI --> | New Session | APK / URL + task description
| Input Form |--------------------------------+
+------------------+ |
v
+------------------+ +-------------------+
| Fleet Dashboard | <---------------> | Session |
| (live overview) | | Scheduler |
+------------------+ +--------+----------+
|
+-----------------------------------------------------+------------------------------------------+
v v v v
redroid container redroid container redroid container ... up to ~100
+ agent sidecar + agent sidecar + agent sidecar
(observe->decide->act) (observe->decide->act) (observe->decide->act)
| | |
+---------> Object storage (screenshots, step JPEGs, logs)
+---------> Results DB (Postgres — sessions, steps, log entries, analyses, devices, settings, users)
| Component | Responsibility |
|---|---|
| New Session form | |
| Accept APK upload or domain URL, task description, device/browser profile, optional context file | |
| Fleet Dashboard | |
| Live overview of every session with status bar, search, filter, sort, pagination | |
| Session Detail | |
| Live screen mirror + structured log stream + replay player + AI analysis panel | |
| Session Scheduler | |
| Dispatch loop (2s tick) + fleet loop (10s tick): queued sessions -> reserve/boot device -> dispatch agent | |
| Android container (redroid) | |
| One isolated Android 11 instance per session via Docker, with per-container binder device slots | |
| Agent sidecar | |
| Observe -> decide -> act loop. Sub-agents: planner, executor, verifier, reporter | |
| Device Manager | |
| Container lifecycle: provision, deploy, health-check, teardown, warm pool maintenance | |
| Postgres DB | |
| Source of truth for session metadata, steps, log entries, device state, session analyses, settings, users | |
| File storage | |
| Per-step replay screenshots (downscaled JPEG), uploaded APKs, stored on local filesystem |
Each session runs a single sidecar process that cycles through five steps:
Observe— Capture a screenshot (adb screencap
via adbutils) and the UI element hierarchy (uiautomator dump
parsed into a flat element list with bounds, text, content-desc, and class).Decide— Send image + hierarchy + task description + action history to the AI model. The model returns one structured action from the fixed vocabulary.Act— Translate the action into ADB calls:input tap
,input swipe
,input text
,input keyevent
, etc.Settle— Poll for system Toasts in the ~1s post-action window, then capture a fresh screenshot for the next step.** Ground truth check**— Per-step hierarchy hash detects whether the screen actually changed; activity name tracking detects navigation independently of visual appearance.
The model must explicitly emit assert
(expected vs. actual comparison) and done
(pass/fail verdict with reasoning) — it cannot wander indefinitely.
Task + screenshot
|
v
+------------------+
| Planner | Creates 3-6 concrete checkpoints from task + screenshot (+ APK manifest activities)
| (once) | Degrades gracefully to empty list -> executor/verifier fall back to free-form task
+--------+---------+
| plan checklist
v
+------------------+
| Executor | Per-step: screenshot + hierarchy + history -> one action
| (per step) | Retries on bad generation (3 attempts). Loop detection. Activity tracking.
+--------+---------+
| "done" with proposed verdict
v
+------------------+
| Verifier | Checkpoint-by-checkpoint review against action history + final screenshot
| (once) | Can set should_continue=true -> executor keeps going. Falls back to executor's verdict.
+--------+---------+
| final verdict
v
+------------------+
| Reporter | Post-verdict write-up: summary, breakdown, technical explanation, severity, recommendations
| (once) | Persisted to DB. Serves Analysis tab + PDF export. Degrades to checkpoint-derived fallback.
+------------------+
- User submits an APK (or URL), task description, and optional context file through the New Session form.
- API creates a
queued
session row with sequential ID (e.g.S9403
). - Session Scheduler's dispatch loop (every 2s) picks up queued sessions up to the parallelism limit.
- Scheduler reserves an idle device or cold-provisions one (binds a port + binder slot, boots redroid container, waits for adb +
sys.boot_completed
). - Session transitions
booting
->initializing
->running
. The agent sidecar installs the APK (if applicable), runs the planner, then begins the observe-decide-act loop. - When the executor emits
done
, the verifier reviews the evidence. If it passes, the session transitions toanalyzing
(device is released immediately), the reporter generates the write-up, then the session settles intopassed
orfailed
. - Artifacts (step screenshots, action log, analysis) are persisted. The container is torn down. The fleet loop (every 10s) re-deploys the device for the warm pool.
- The Fleet Dashboard and Session Detail reflect the final verdict and analysis in real time.
| Status | Meaning |
|---|---|
queued |
|
| Waiting for an available device slot | |
booting |
|
| Android container is starting up | |
initializing |
|
| Installing APK / preparing agent | |
running |
|
| Agent loop is actively executing the task | |
awaiting_questions |
|
| Review mode: waiting for user answers to clarifying questions | |
awaiting_approval |
|
| Review mode: waiting for user to approve the generated test plan | |
analyzing |
|
| Verdict reached; reporter generating write-up | |
passed |
|
| Task completed successfully | |
failed |
|
| Task completed but assertions failed | |
stopped |
|
| Manually stopped by a user | |
error |
|
| Infrastructure or system error | |
timed_out |
|
| Agent exceeded max steps (50) |
- Each redroid container runs Android 11 (1080x2400, 480dpi) with nav bar + soft keyboard disabled via
policy_control
andpm disable-user
. - Devices are identified by
D-{hex}
ID, assigned a host port (5555-5655 range) and abinder slot(binder/hwbinder/vndbinder triplet) for the kernel's binder driver — a hard cap set atmodprobe
time on the host. - Warm pool: idle, healthy devices maintained up to a user-configurable target. Devices in
error
state are reaped after 30s backoff. - Health checks run every 10s: container running +
sys.boot_completed
. - Device identity profiles (Pixel 7 Pro / Pixel 6 / Samsung S24) are applied via
ro.build
property overrides before any app touches the device.
The backend is a Python FastAPI server at /api
. Interactive docs at http://localhost:8000/docs
.
| Method | Path | Purpose |
|---|---|---|
POST |
||
/api/sessions |
||
| Create a new session (multipart form: APK file or URL + task + device profile) | ||
GET |
||
/api/sessions |
||
| List sessions (cursor-paginated, filterable by status/device/build/text search) | ||
GET |
||
/api/sessions/stats |
||
| Dashboard stats (running/queued/passed/failed counts + success rate) | ||
GET |
||
/api/sessions/{id} |
||
| Get single session details | ||
PATCH |
||
/api/sessions/{id}/control |
||
Control session (action: end or restart ) |
||
POST |
||
/api/sessions/{id}/answers |
||
| Submit answers to review clarifying questions | ||
POST |
||
/api/sessions/{id}/approve |
||
| Approve review-mode test plan and queue session | ||
POST |
||
/api/sessions/{id}/regenerate-plan |
||
| Regenerate test plan from existing answers | ||
POST |
||
/api/sessions/{id}/back-to-questions |
||
| Go back to clarifying questions from plan review | ||
GET |
||
/api/sessions/{id}/logs |
||
Get log entries with cursor (since ) |
||
POST |
||
/api/sessions/{id}/logs |
||
| Create a log entry | ||
GET |
||
/api/sessions/{id}/steps |
||
Get replay steps with cursor (since_step ) |
||
GET |
||
/api/sessions/{id}/steps/{n}/screenshot |
||
| Get step screenshot (JPEG) | ||
GET |
||
/api/sessions/{id}/analysis |
||
| Get session analysis write-up | ||
POST |
||
/api/sessions/{id}/analysis/regenerate |
||
| Regenerate analysis for a finished session | ||
WS |
||
/api/sessions/{id}/events |
||
| WebSocket — real-time log/step/status deltas | ||
WS |
||
/api/sessions/{id}/stream |
||
| WebSocket — live screen frame stream (PNG) |
| Method | Path | Purpose |
|---|---|---|
GET |
||
/api/devices |
||
| List all devices | ||
POST |
||
/api/devices |
||
| Provision a new device (boots redroid container) | ||
GET |
||
/api/devices/capacity |
||
| Binder slot usage (total/used/available) | ||
POST |
||
/api/devices/{id}/start |
||
| Start an offline device | ||
POST |
||
/api/devices/{id}/stop |
||
| Stop a running device | ||
POST |
||
/api/devices/{id}/restart |
||
| Restart a device (re-deploy container) | ||
POST |
||
/api/devices/{id}/maintenance |
||
| Toggle maintenance mode | ||
POST |
||
/api/devices/{id}/warm |
||
| Adjust warm instance count (delta -8 to +8) | ||
DELETE |
||
/api/devices/{id} |
||
| Remove device (tear down container, free port/slot) |
| Method | Path | Purpose |
|---|---|---|
POST |
||
/api/auth/login |
||
| Login (email + password) | ||
POST |
||
/api/auth/signup |
||
| Register new user | ||
POST |
||
/api/auth/logout |
||
| Logout |
| Method | Path | Purpose |
|---|---|---|
GET |
||
/api/settings |
||
| Get app settings (workspace, region, notifications, parallelism, API key, warm pool size, max devices) | ||
PUT |
||
/api/settings |
||
| Update app settings |
- Docker Engine and Docker Compose (for Postgres and redroid containers)
- Python 3.11+
- Node.js 20+
- An AI provider API key (Fireworks AI on AMD Instinct recommended, or any OpenAI-compatible provider)
Development (hot reload):
./dev.sh
Production (built frontend, no reload):
./start.sh
Both scripts start Postgres (if not running), run migrations, and launch the full stack. Ctrl-C
stops everything.
cd backend
cp .env.example .env
Edit .env
to set your AI_API_KEY
and any other settings.
docker compose up -d
Or manually:
docker run -d --name veta-pg \
-e POSTGRES_USER=postgres \
-e POSTGRES_PASSWORD=postgres \
-e POSTGRES_DB=veta \
-p 5432:5432 \
postgres:16-alpine
cd backend
pip install -e .
alembic upgrade head
uvicorn app.main:app --reload --host 0.0.0.0 --port 8000
The API starts at http://localhost:8000
. Docs at http://localhost:8000/docs
.
On first boot, the server auto-seeds admin user (
admin
/admin
) and default settings.
cd frontend
npm install
npm run dev
The dashboard starts at http://localhost:8080
. Login with any email/password.
Set
VITE_API_URL
to change the backend address (defaulthttp://localhost:8000
).
sudo modprobe binder_linux devices="binder,hwbinder,vndbinder"
adb devices
export REDROID_IMAGE=redroid/redroid:11.0.0-latest
| Variable | Default | Description |
|---|---|---|
DATABASE_URL |
||
postgresql+asyncpg://postgres:postgres@localhost:5432/veta |
||
| Postgres connection string | ||
REDROID_IMAGE |
||
redroid/redroid:11.0.0-latest |
||
| Android container image | ||
ADB_HOST |
||
localhost |
||
| ADB host address | ||
REDROID_PORT_START |
||
5555 |
||
| Start of port range for device ADB | ||
REDROID_PORT_END |
||
5655 |
||
| End of port range (up to ~100 devices) | ||
BINDER_SLOTS |
||
auto-detected from /dev |
||
| Override for binder slot count | ||
AI_API_KEY |
||
| -- | AI provider API key (also configurable via Settings UI) | |
AI_PROVIDER |
||
fireworks |
||
AI provider: fireworks , veta-ai , google-gemini , openrouter , deepseek , anthropic |
||
AI_MODEL |
||
| provider default | Model name override | |
MODEL_ENDPOINT |
||
| provider default | API base URL override |
veta/
Docs/ Design documents, wireframes, logo
server/ vLLM inference server (AMD ROCm)
veta_inference_server.ipynb Notebook: serve vision models on AMD GPUs via vLLM
bin/ cloudflared binary for tunnel
backend/ FastAPI backend API
app/
main.py FastAPI entrypoint + lifespan (auto-seed, stale cleanup, dispatch + fleet loops)
config.py pydantic-settings (.env / env vars)
database.py Async SQLAlchemy engine + session factory
dispatcher.py Session dispatch loop + fleet health loop
device_manager.py Redroid container lifecycle, port/binder allocation, warm pool, health checks
device_events.py SSE device state change notifications
apk_utils.py APK metadata extraction (aapt + apkutils)
screenshot_store.py Per-step screenshot downscale + JPEG storage
models/ SQLAlchemy ORM models (session, step, log_entry, device, settings, session_analysis, user)
schemas/ Pydantic request/response schemas (camelCase JSON)
routers/ REST + WebSocket endpoint handlers (sessions, steps, logs, devices, auth, settings, stream)
services/
session_analysis.py Build + store session analysis (bridges reporter sub-agent to DB)
adapters/ Port implementations (ApkInspectorImpl, etc.)
alembic/ Database migrations
scripts/
seed.py Seed admin user + default settings
uploads/ Uploaded APKs + step screenshot storage
agent/ AI agent sidecar
agents/
base.py Shared retry logic (generate_with_retries)
planner.py Pre-execution checkpoint builder
executor.py Per-step action decision
verifier.py Post-decision verification
reporter.py Post-verdict write-up
core/
adb.py adbutils device handle
device.py Screenshots, action execution, marker extraction, activity tracking
hierarchy.py XML hierarchy parsing, element signature hashing
model_client.py Multi-provider AI client (Gemini SDK + OpenAI-compat) with retry + JSON extraction
apk.py APK metadata extraction, install/launch on device
cancellation.py Thread-safe session cancellation
thinking_hooks.py Real-time thinking indicator callbacks
modes.py Mode configs (review/run) + allowed action groups
skill_text.py Loads agent playbook into system prompt
ports/ Protocol interfaces for dependency injection
apk_inspector.py ApkInspector protocol (get_all_activities, get_apk_details)
skills/ Agent playbook (loaded into system context)
SKILL.md Master playbook — decision loop, confidence/targeting rules, verdict discipline
actions.md Full action vocabulary with semantics for each action type
elements.md UI element taxonomy — how to recognize and interact with different UI components
scenarios.md Step-by-step playbooks for authentication, permissions, onboarding, forms, search
debugging.md Recovery strategies for stuck states, loop detection, unexpected screens
memory.py SessionMemory — action history, loop detection, activity tracking, screen stack
orchestrator.py Session state machine wiring all sub-agents together (booting -> ... -> passed/failed)
phases.py Phase-based lifecycle (SetupPhase, ReviewPhase, RunPhase)
runner.py Compatibility entrypoint (delegates to orchestrator)
frontend/ TanStack Start web dashboard
src/
routes/ File-based routing
__root.tsx Root layout + sidebar + auth
index.tsx Fleet Dashboard (stat bar, search, filter, sort, pagination, session cards)
session.$id.tsx Session Detail (live mirror, log stream, replay player, analysis panel, export, filmstrip)
new-session.tsx New Session form (APK/URL toggle, file drag-drop, context attachment, device profile)
devices.tsx Device Farm (card grid, provision dialog, warm pool controls, binder tracking)
history.tsx Run History (table view, sort, filter)
settings.tsx Settings (workspace, execution, API key, notifications, integrations)
login.tsx / signup.tsx Auth pages
components/
session-card.tsx Fleet dashboard session card
session-analysis-panel.tsx Analysis tab UI
ui/ shadcn/ui primitives (50+ components)
lib/
api.ts API client (fetch + WebSocket wrappers, all endpoints)
auth.ts Session storage auth
session-report.ts PDF + CSV export (jsPDF with professional layout)
public/ Static assets (favicon, logo)
Located in agent/agents/
. Each is independently promptable and can be retried or swapped without affecting the others.
| Agent | File | Purpose |
|---|---|---|
| planner | ||
planner.py |
||
| Builds 3-6 concrete checkpoints from task + first screenshot (+ APK manifest). Runs once before execution. | ||
| executor | ||
executor.py |
||
| Per-step decision: screenshot + hierarchy + history -> one action. Retries on bad generation (3 attempts). | ||
| verifier | ||
verifier.py |
||
| Checkpoint-by-checkpoint review against action history. Can send session back for more exploration. | ||
| reporter | ||
reporter.py |
||
| Post-verdict structured write-up. Falls back to checkpoint-derived summary on model failure. |
Located in agent/core/
.
| Module | Purpose |
|---|---|
device |
|
| Screenshots, action execution dispatch, activity tracking, marker extraction | |
hierarchy |
|
| UiAutomator hierarchy dump + parsing + element signature hashing | |
model_client |
|
| Multi-provider HTTP client (Gemini SDK + OpenAI-compat, Fireworks on AMD Instinct), JSON extraction, retry logic | |
apk |
|
| APK install, package discovery, app launch on device | |
adb |
|
| Shared adbutils device handle | |
cancellation |
|
| Thread-safe session cancellation registry | |
modes |
|
| Mode configs (review/run) + action group definitions | |
skill_text |
|
| Agent playbook loaded into the executor's system prompt |
| Provider | SDK | Backend hardware | Models | Status |
|---|---|---|---|---|
| Fireworks AI | ||||
openai |
||||
| AMD Instinct MI250/MI300X | ||||
qwen3.7-plus , firefunction-v2 , Gemma 4 |
||||
| ✅ Default (100% AMD) | ||||
| Veta AI (self-hosted) | ||||
openai |
||||
| AMD ROCm (vLLM) | ||||
| Any HF vision model (Qwen2.5-VL, Gemma, Llama) | ✅ 100% AMD | |||
| Google Gemini | ||||
google-genai |
||||
| Google TPU | gemma-4-26b-a4b-it |
|||
| ⬜ Optional | ||||
| OpenRouter | ||||
openai |
||||
| Varies | openai/gpt-4o-mini , any routed model |
|||
| ⬜ Optional | ||||
| Deepseek | ||||
openai |
||||
| Varies | deepseek-v4-flash |
|||
| ⬜ Optional | ||||
| Anthropic | ||||
openai |
||||
| Varies | Anthropic models via OpenAI-compat endpoint | ⬜ Optional |
100% of inference runs on AMD out of the box.The two AMD-backed providers (Fireworks + self-hosted ROCm) handle every agent call. Gemini, OpenRouter, Deepseek, and Anthropic are optional alternatives — none are enabled by default.
The agent playbook (agent/skills/
) is loaded into the executor's system context:
| File | Purpose |
|---|---|
SKILL.md |
|
| Master playbook — decision loop, confidence/targeting rules, verdict discipline | |
actions.md |
|
| Full action vocabulary with semantics for each action type | |
elements.md |
|
| UI element taxonomy — recognizing and interacting with buttons, sliders, toggles, dropdowns, dialogs | |
scenarios.md |
|
| Step-by-step playbooks for authentication, permissions, onboarding, forms, search | |
debugging.md |
|
| Recovery strategies for stuck states, loop detection, unexpected screens |
| Layer | Technology |
|---|---|
| Backend | |
| Python 3.11+, FastAPI, SQLAlchemy (async), Alembic, PostgreSQL | |
| Agent loop | |
| Python, httpx, Pillow, google-genai SDK, openai SDK | |
| Frontend | |
| React 19, TanStack Start, TanStack Router, Tailwind CSS v4, shadcn/ui | |
| AI inference (cloud, default) | |
| Fireworks AI — AMD Instinct MI250/MI300X (100% AMD) | |
| AI inference (self-hosted) | |
| vLLM + AMD ROCm (100% AMD) | |
| Android runtime | |
| Docker, redroid (Android 11), ADB, adbutils, uiautomator | |
| Infrastructure | |
| Docker, Docker Compose (dev), Kubernetes (target) | |
| Real-time | |
| WebSocket (live frame stream + events bus), SSE fallback | |
| Export | |
| jsPDF + jspdf-autotable (PDF), CSV |
| Approach | Cost per session (~50 steps) | When to use |
|---|---|---|
| Fireworks API (AMD Instinct) | ||
| ~$0.02-0.05 | Development, low-volume QA | |
| Self-hosted vLLM on ROCm | ||
| GPU electricity only | Production, high-volume CI/CD | |
| Gemini CachedContent | ||
| ~$0.01 (95% cached) | Optional fallback / multi-provider |