A measured study of multi-provider LLM consensus as a locator-healing signal — and why Automation Sandbox keeps the LLM out of the decision. Originally published on the project site.
When a UI refactor deletes an element, a self-healing engine must decline and ask a human — not latch onto a neighbouring button. I tested whether asking several independent LLMs and requiring them to agree can act as that "the element is gone" detector.
Across four live multi-provider runs (2026-08-16 to 2026-08-18, 133 usable scenarios):
The consensus check is real protection — but it comes from providers disagreeing with each other, not from any model recognising that the element is gone. When they happen to agree on a deleted element, they are unanimously, confidently wrong. That is why in Automation Sandbox the LLM is an opt-in fallback gated by an independent-agreement quorum, and a heal is only ever committed after the retried action actually succeeds — the model is never the decision maker.
Full data, methodology, and the trade-off curves live in the Benchmark & Calibration guide (§3, §4, §6). This post is the standalone story; that page is the source of truth for every number.
Locator healing handles the easy 90%: an AutomationId
changes, a label is reworded, a control moves 100px in a layout pass. The engine re-resolves the element from structural evidence — control type, parent, sibling position, name similarity, geometry — and retries.
The dangerous case is the other kind of change: the element is deleted outright. A checkout button is removed; the test that clicked it should now fail loudly so a human looks at it. What it must not do is quietly heal onto the "Cancel" button next to where "Submit" used to be, pass green, and hide a real regression — a "false heal".
A pure-heuristic scorer cannot solve this case on its own, and I proved that rigorously: a surviving sibling in a deleted control's container is structurally indistinguishable from a control that genuinely moved next to that sibling. Their similarity-score distributions overlap ([0.665, 0.955]
for deleted-element decoys vs [0.749, 0.874]
for true compound drift), so no confidence threshold, runner-up margin, cluster-density, or control-type filter can draw a line between them.
That negative result is what motivated the LLM experiment. Semantic reasoning asks a different question than geometry does — maybe independent models, forced to agree, could tell "moved" from "gone".
Natural locator drift across real releases is too sparse to benchmark. So I invert it: take a real, organically-built application UI tree and systematically mutate its authored locators.
Leakage protection. Every candidate AutomationId
in a mutated tree is rewritten to the same opaque format (ablation-
- SHA-256 hex), not just the target. Otherwise a model could solve the scenario by spotting the "odd one out". IDs carry no semantic hint — which means these scores are a lower bound: in production, a descriptive
btnSaveDocument
gives the model more to work with than a synthetic hash does.
LLM evaluation protocol. Temperature 0. Every provider's raw vote recorded alongside the agreement telemetry. A scenario counts as usable only when ≥ 2 providers actually returned an answer — one opinion can neither agree nor disagree with anything. Token cost is managed by targeting the two informative subsets: 25 compound-drift + 42 removed-element scenarios.
Running the deterministic scorer at the shipped default (MinimumConfidence = 0.50
) over all 176 HandBrake scenarios:
| Mutation tier | n | Correct | False heals | Missed (review) | Score range |
|---|---|---|---|---|---|
| Renamed AutomationId | 42 | 40 | 0 | 2 | [1.000, 1.000] |
| Name drift | 25 | 23 | 0 | 2 | [0.877, 0.950] |
| Position shift | 42 | 34 | 0 | 8 | [0.832, 0.955] |
| Compound drift | 25 | 6 | 2 | 17 | [0.749, 0.874] |
| Removed element | |||||
| 42 | |||||
| 25 | |||||
| 17 | |||||
| 0 | |||||
[0.344, 0.955] |
40.5% of deleted elements are false-healed at the default threshold. On ShareX the same figure is 57.1%. Raising the threshold trades this away against auto-heal recall linearly — at 0.90
the false-heal rate on removed elements is still 5.5% while three-quarters of all locators are forced to manual review. There is no free setting.
This is the gap the LLM was asked to close.
Four runs, 2026-08-16 to 2026-08-18, as the provider pool grew from 3 to 7 and Groq's model was swapped twice:
| Run | Usable (n) | Compound-drift unanimous | Removed-element unanimous |
|---|---|---|---|
| 1 (08-16) | 19 | 6 / 7 | 3 / 12 |
| 2 (08-16) | 39 | 16 / 17 | 10 / 22 |
| 3 (08-16) | 33 | 14 / 14 | 11 / 19 |
| 4 (08-18) | 42 | 16 / 17 | 10 / 25 |
| Total | |||
| 133 | |||
| 52 / 55 | |||
| 34 / 78 |
Providers reached unanimous agreement on 94.5% of scenarios where a successor existed, and on 43.6% where the element was gone. That gap is real — it is the only signal in the entire project that separates the two populations at all.
But unanimity is not a safe acceptance gate:
Every one of the
52unanimous verdicts on a surviving element was correct.
Every one of the34unanimous verdicts on a deleted element was a false heal.
Zero exceptions in either direction, across four runs with four different provider sets.
An earlier draft of this analysis reported "33%" wrong — that figure pooled both mutation types into one denominator and understated the real rate. Read per type, the deleted-element rate is 100%: agreement never once happened to be right about an absence.
The hypothesis predicted two safe outcomes on a deleted element — providers decline, or providers scatter across different decoys. In practice:
Widening the pool made this clearer, not better. One run recorded 10 unanimous false heals; 7 of those 10 had three independent model families agreeing on the same non-existent element — three vendors, three architectures, one wrong answer, unanimously. Going 3 → 7 providers moved the removed-element agreement rate 25% → 45% → 58% → 40% with no downward trend and 100% wrong throughout.
The protection is a byproduct of independence, and independence alone does not bound the failure rate: a genuinely capable, independent reasoner finds a deleted control's surviving neighbour a convincing answer often enough that adding more reasoners does not reliably break the tie. It also cannot be strengthened by asking for more confidence — the failing cases are already maximally confident.
Automation Sandbox is built around this result rather than despite it:
| Design choice | Why |
|---|---|
| Heuristic-first, deterministic. A pure C# structural scorer decides on its own, ~23ms for 3,000 controls, zero tokens. | |
| The LLM is never on the default path. Most healing never touches a model. | |
| LLM is opt-in and quorum-gated (≥ 2 providers must independently name the same candidate). | |
| A single model's confidence is worthless here. Agreement is permission to consider a pick — never evidence it is correct. | |
| A heal commits only after the retried action succeeds; the shipped default mode changes no locators and only routes candidates to a report. | |
| A wrong pick that cannot actually perform the test step is caught before it is persisted. | |
| Every decision is written to an audit report (JSON + HTML) — which signal contributed what weight, which providers voted, what the outcome was. | |
| "The AI healed it" is not an acceptable answer. You can see exactly why. |
The honest bottom line: unassisted absence detection is mathematically bounded by the structural score overlap. Automation Sandbox does not claim to have solved it. It makes the failure visible and declinable instead of silent and green.
From a clean clone (needs the .NET SDK; the LLM run needs provider API keys):
dotnet test TestAutomation/ScenarioRunner/ScenarioRunner.csproj \
--filter "FullyQualifiedName~LocatorAblationTests.HandBrakeFixture_RunsEndToEndAndReportsMetrics"
dotnet test TestAutomation/ScenarioRunner/ScenarioRunner.csproj \
--filter "FullyQualifiedName~LocatorAblationTests.HandBrakeFixture_ThresholdSweep"
dotnet run --project samples/CalibrationCli -- <your-tree.json> --app YourApp
dotnet test TestAutomation/ScenarioRunner/ScenarioRunner.csproj \
--filter "FullyQualifiedName~LocatorAblationTests.HandBrakeFixture_LlmConsensus_LiveEvaluation"
Every claim is guarded by a committed regression test — see the "Regression guards" lines throughout the Benchmark & Calibration guide.
Automation Sandbox is MIT, pure C#/.NET, seven packages on nuget.org. It does locator self-healing and intent-driven test generation for Windows desktop (FlaUI/UIA3) and web (Playwright). If you maintain UI tests that break on every refactor, I'd genuinely like your feedback — especially from anyone who has shipped locator self-healing and hit the deleted-element problem.