You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This benchmark suite compares DroidPath (Shadow DOM + Layer 3 A Spatial Graph Router)* against the two dominant Android automation paradigms used by autonomous AI agents:
Pure Vision (Screenshot + VLM loop) — captures a full-resolution PNG screenshot on every step and sends it to a Vision-Language Model to predict (x, y) coordinates.
Appium / uiautomator dump XML loop — runs uiautomator dump over ADB on every step and feeds the raw XML hierarchy (~25–35 KB) to an LLM.
DroidPath (Shadow DOM + Layer 3 A)* — streams incremental NDJSON accessibility diffs over a persistent loopback TCP socket (android-daemon) and executes learned multi-hop paths locally in Rust via A* without intermediate LLM turns.
Transparency Note: Hardware primitive latencies and End-to-End (E2E) Settings navigation timings below were measured live over USB ADB on an unlocked Google Pixel 10a (61061JEA307110, Android com.android.settings root screen, 54 live UI nodes) using node scripts/benchmark_agentic.mjs --mode live --json against compiled target/release/droidpath-mcp. Multi-step Pure Vision and Appium/XML comparisons in the replay table run real droidpath-mcp stdio JSON-RPC calls against a 60-node loopback Shadow DOM daemon paired with explicit token and cost formulas documented under Methodology & Token Model.
- Live Hardware Probe (Unlocked Google Pixel 10a · com.android.settings, 54 Nodes)
Primitive / Method
Status
Measured Latency
Payload Size
Notes
DroidPath android_get_ui_tree (steady-state) ok
0.32 ms
10,742 bytes (~2,686 tok) 54 live UI nodes from local Shadow DOM stream (7,575× faster than uiautomator)
DroidPath android_get_ui_tree (cold connect)
ok
41.89 ms
10,742 bytes (~2,686 tok) First TCP connect + live snapshot (58× faster than uiautomator)
DroidPath Hybrid Vision (include_vision: true) ok
452.33 ms
38,581 bytes JPEG (540×1212) 54 live UI nodes + compressed JPEG (q45, 6.1× smaller than raw PNG)
adb exec-out screencap -p (raw PNG) ok
658.1 ms
235,258 bytes PNG
Visual capture only — no structured UI tree
adb exec-out uiautomator dump /dev/tty (XML) ok
2,423.9 ms
29,357 bytes (~7,340 tok) Requires stopping UiAutomation daemon; 2.7× more text tokens than Shadow DOM
- Live End-to-End Hardware Navigation (Google Pixel 10a · Real Seed Hydration → Warm A*)
Task (Settings → Target)
Cold-Start (Seed Hydration)
Cold Status
Warm A* Graph
Warm Status
Live Success Rate
Settings → Display
378.6 ms
arrived
296.2 ms(1 step) arrived
100%
Settings → Battery
655.9 ms
arrived
642.8 ms(1 step) arrived
100%
Settings → About phone(scroll required) 775.7 ms
arrived
744.1 ms(1 step) arrived
100%
Average across live suite
603 ms
100% (3/3) 561 ms
100% (3/3) 100%
- Local MCP Runtime Micro-Benchmarks (Measured) Metric
Measured Value
tools/list JSON-RPC latency (9 tools) 0.25 ms
android_graph_stats JSON-RPC latency
0.42 ms
Pre-hydrated Seed Graph loaded at boot
20 vertices / 19 edges
Compact Shadow DOM tree (60-node mock Settings screen)
Text tokens:ceil(utf8_bytes / 4) (standard BPE approximation for JSON/XML).
Cost / 1,000 tasks: Computed transparently at $3.00 / 1M input tokens (standard Claude Sonnet-class API pricing): (avg_tokens_per_task * 1,000 / 1,000,000) * $3.00.
Vision image tokens: Anthropic's published (W × H) / 750 formula for a 1080×2400 Android screenshot scaled to 1568px long edge (706 × 1568 / 750 = 1,477 image tokens + ~350 prompt/tool tokens = 1,827 tokens/step).
Opt-in Hybrid Vision (include_vision: true): Downscaled 540px-wide JPEG (quality: 45, measured 38.6 KB on unlocked Pixel 10a Settings vs 235.3 KB raw PNG) is strictly opt-in and only used when an agent explicitly needs visual disambiguation (e.g., canvas/icons), remaining strictly blocked on FLAG_SECURE screens.