{"slug": "reflex-demonstrate-a-gui-workflow-once-replay-it-with-zero-llm-calls", "title": "Reflex: Demonstrate a GUI workflow once, replay it with zero LLM calls", "summary": "REFLEX, a new open-source tool, replays demonstrated GUI workflows with zero LLM calls per action, achieving 100 perfect runs out of 100 in a benchmark with randomized window positions and row orders, and a median replay time of 6.5 seconds versus 14.4 seconds for demonstration. Developed by an unnamed team, REFLEX compiles screen recordings into a visual state machine using OpenCV template matching, and includes a repair mechanism that uses a single LLM call only when a run is blocked, then writes the fix back into the skill.", "body_md": "**Demonstrate a GUI workflow once. REFLEX compiles the recording into a visual state\nmachine and replays it faster than the demonstration — with zero LLM calls per action —\nagainst a window that moved, rows that shuffled, and a theme that changed color.**\n\nComputer-use agents re-pay a model call for every click, which makes them slow, expensive, and nondeterministic on workflows a human could teach in fifteen seconds. RPA tools solved replay decades ago but break on anything that moves. REFLEX sits between them:\n\n**Record**: a screen ring buffer (~12fps,`mss`\n\n) + global input hooks (`pynput`\n\n) capture a human demonstration as frames + actions.**Compile**: each click becomes a state — target crop, tight band, context crop, and a label strip, all cut from the frame the human aimed at. No coordinates are stored as truth; crops are the truth.**Replay**: each state matches its crops on the*live*screen (OpenCV template matching with a pyramid search), clicks at the*matched*position, and verifies the screen actually responded before advancing. Timeouts pause the run as BLOCKED — the runtime never guesses.**Repair**(prototype): a BLOCKED run builds a compact repair package and asks a model for ONE plan; a successful repair is written back into the skill as a new version, so the same exception never costs a second model call.\n\nThe benchmark app (`reflex/mock_app.py`\n\n, tkinter) randomizes **window position, row order,\naccent color, and spinner duration** per launch and writes its own ground-truth log of what\nwas actually clicked and typed — replays are scored against reality, not against the\nruntime's opinion of itself.\n\nEach run: fresh demonstration → compile → relaunch randomized → replay from crops only.\n\n| outcome | count |\n|---|---|\n| PERFECT (all fields exact, correct shuffled row, DONE) | 100 |\n| MISROUTE (text in a wrong field) | 0 |\n| BLOCKED (honest refusal, nothing wrong typed) | 0 |\n| CRASH/other | 0 |\n\nMedian replay: 6.5s (demonstration: 14.4s). LLM calls per replay: 0.\n\nWindow position AND form field order randomized per replay; ground truth is the page itself confessing which fields received which text (document.title).\n\n| outcome | count |\n|---|---|\n| DEMO-INVALID | 1 |\n| PERFECT | 29 |\n\nValid reps (contention/demo-invalid excluded): 29.\n\nEvery rule in `matcher.py`\n\nexists because a measured failure demanded it:\n\n| Rule | The failure that earned it |\n|---|---|\n| Change detection counts moved pixels, not mean diff | a menu opening changed 0.2% of the screen and mean-diff called it \"no change\" |\n| Featureless crops are refused, not matched | a blank patch of one app matched a blank patch of a different app at confidence 1.0 and 33 characters went into the wrong window |\n| Tight band beats square crop in lists | reordered rows dragged old neighbors into the crop → 0.81 confidence on the right row |\n| Label strip through the cursor row, matched globally | three pixel-identical empty form fields are distinguishable only by the labels beside them; every cursor-local crop excluded the label and text landed in the wrong field |\n| Twin margin: the best row must beat the runner-up by ≥0.02 | a strip win inside noise of a sibling row is a coin flip, not a localization |\n| Edge-domain rescue (raw float Sobel magnitude) | a randomized accent color dropped grayscale correlation to 0.72–0.85 on the same button; gradient structure survives color |\n| Focus clicks verify via the next type-state's visible text | a caret is 1px — invisible to change detection; the honest evidence that focus landed is that typing appears |\n\n`reflex/matcher.py`\n\njournals every decision (context confidence, strip margin, fallbacks\ntaken, final box) to `diag_find.jsonl`\n\n— when a run misbehaves, the evidence chain is one\nread away.\n\nWindows, Python 3.13+, `pip install mss pynput opencv-python numpy`\n\n.\n\n```\npython reflex/flow.py        # one full cycle: demo → compile → randomized replay → verdict\npython reflex/bench100.py    # the unattended 100-run benchmark (waits for an idle desktop)\n```\n\n`kernel/motor.py`\n\nis the only component that emits input (SendInput, DPI-proof 0–65535\nvirtual-desktop space). It honors a `SAFE_STOP`\n\nfile before every emission — drop the file\nnext to `journal/`\n\nand everything stops.\n\n- The benchmark is a controlled mock app, not a production suite. It randomizes what real apps change (position, order, theme, timing) but it is one workflow on one machine.\n- The repair path is a working prototype, not benchmarked yet.\n- Windows-only (SendInput, virtual-desktop coordinates).\n- Recording quality matters: crops are cut from the frame the human aimed at; a demo performed during heavy screen churn will compile weaker anchors.", "url": "https://wpnews.pro/news/reflex-demonstrate-a-gui-workflow-once-replay-it-with-zero-llm-calls", "canonical_source": "https://github.com/MARCCHERGGI/reflex", "published_at": "2026-08-10 03:30:26+00:00", "updated_at": "2026-08-10 03:36:07.945154+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "computer-vision", "developer-tools"], "entities": ["REFLEX", "OpenCV", "Python", "tkinter", "mss", "pynput"], "alternates": {"html": "https://wpnews.pro/news/reflex-demonstrate-a-gui-workflow-once-replay-it-with-zero-llm-calls", "markdown": "https://wpnews.pro/news/reflex-demonstrate-a-gui-workflow-once-replay-it-with-zero-llm-calls.md", "text": "https://wpnews.pro/news/reflex-demonstrate-a-gui-workflow-once-replay-it-with-zero-llm-calls.txt", "jsonld": "https://wpnews.pro/news/reflex-demonstrate-a-gui-workflow-once-replay-it-with-zero-llm-calls.jsonld"}}