# Can You Trust an LLM to Fix a Broken Locator? I Measured It.

> Source: <https://dev.to/mustafa_sercan_sak/can-you-trust-an-llm-to-fix-a-broken-locator-i-measured-it-4och>
> Published: 2026-08-30 11:37:50+00:00

*A measured study of multi-provider LLM consensus as a locator-healing signal — and why Automation Sandbox keeps the LLM out of the decision. Originally published on the project site.*

When a UI refactor **deletes** an element, a self-healing engine must decline and ask a human — not latch onto a neighbouring button. I tested whether asking several independent LLMs and requiring them to agree can act as that "the element is gone" detector.

Across **four live multi-provider runs** (2026-08-16 to 2026-08-18, 133 usable scenarios):

**The consensus check is real protection — but it comes from providers disagreeing with each other, not from any model recognising that the element is gone.** When they happen to agree on a deleted element, they are unanimously, confidently wrong. That is why in Automation Sandbox the LLM is an opt-in fallback gated by an independent-agreement quorum, and a heal is only ever committed after the retried action actually succeeds — the model is never the decision maker.

Full data, methodology, and the trade-off curves live in the [Benchmark & Calibration guide](https://mustafasercansak.github.io/automation-sandbox/docs/benchmark-calibration.html) (§3, §4, §6). This post is the standalone story; that page is the source of truth for every number.

Locator healing handles the easy 90%: an `AutomationId`

changes, a label is reworded, a control moves 100px in a layout pass. The engine re-resolves the element from structural evidence — control type, parent, sibling position, name similarity, geometry — and retries.

The dangerous case is the other kind of change: the element is **deleted outright**. A checkout button is removed; the test that clicked it should now *fail loudly* so a human looks at it. What it must not do is quietly heal onto the "Cancel" button next to where "Submit" used to be, pass green, and hide a real regression — a "false heal".

A pure-heuristic scorer cannot solve this case on its own, and [I proved that rigorously](https://mustafasercansak.github.io/automation-sandbox/docs/benchmark-calibration.html#5-offline-absence-signal-investigation-95): a surviving sibling in a deleted control's container is structurally **indistinguishable** from a control that genuinely moved next to that sibling. Their similarity-score distributions overlap (`[0.665, 0.955]`

for deleted-element decoys vs `[0.749, 0.874]`

for true compound drift), so **no confidence threshold, runner-up margin, cluster-density, or control-type filter can draw a line between them.**

That negative result is what motivated the LLM experiment. Semantic reasoning asks a different question than geometry does — maybe independent models, forced to agree, could tell "moved" from "gone".

Natural locator drift across real releases is too sparse to benchmark. So I invert it: take a real, organically-built application UI tree and **systematically mutate** its authored locators.

**Leakage protection.** Every candidate `AutomationId`

in a mutated tree is rewritten to the *same* opaque format (`ablation-`

+ SHA-256 hex), not just the target. Otherwise a model could solve the scenario by spotting the "odd one out". IDs carry no semantic hint — which means these scores are a **lower bound**: in production, a descriptive `btnSaveDocument`

gives the model more to work with than a synthetic hash does.

**LLM evaluation protocol.** Temperature 0. Every provider's raw vote recorded alongside the agreement telemetry. A scenario counts as **usable** only when ≥ 2 providers actually returned an answer — one opinion can neither agree nor disagree with anything. Token cost is managed by targeting the two informative subsets: 25 compound-drift + 42 removed-element scenarios.

Running the deterministic scorer at the shipped default (`MinimumConfidence = 0.50`

) over all 176 HandBrake scenarios:

| Mutation tier | n | Correct | False heals | Missed (review) | Score range |
|---|---|---|---|---|---|
| Renamed AutomationId | 42 | 40 | 0 | 2 | `[1.000, 1.000]` |
| Name drift | 25 | 23 | 0 | 2 | `[0.877, 0.950]` |
| Position shift | 42 | 34 | 0 | 8 | `[0.832, 0.955]` |
| Compound drift | 25 | 6 | 2 | 17 | `[0.749, 0.874]` |
Removed element |
42 |
25 |
17 |
0 |
`[0.344, 0.955]` |

**40.5% of deleted elements are false-healed** at the default threshold. On ShareX the same figure is **57.1%**. Raising the threshold trades this away against auto-heal recall linearly — at `0.90`

the false-heal rate on removed elements is still 5.5% while three-quarters of *all* locators are forced to manual review. There is no free setting.

This is the gap the LLM was asked to close.

Four runs, 2026-08-16 to 2026-08-18, as the provider pool grew from 3 to 7 and Groq's model was swapped twice:

| Run | Usable (n) | Compound-drift unanimous | Removed-element unanimous |
|---|---|---|---|
| 1 (08-16) | 19 | 6 / 7 | 3 / 12 |
| 2 (08-16) | 39 | 16 / 17 | 10 / 22 |
| 3 (08-16) | 33 | 14 / 14 | 11 / 19 |
| 4 (08-18) | 42 | 16 / 17 | 10 / 25 |
Total |
133 |
52 / 55 |
34 / 78 |

Providers reached unanimous agreement on **94.5%** of scenarios where a successor existed, and on **43.6%** where the element was gone. That gap is real — it is the *only* signal in the entire project that separates the two populations at all.

**But unanimity is not a safe acceptance gate:**

Every one of the

52unanimous verdicts on a surviving element was correct.

Every one of the34unanimous verdicts on a deleted element was a false heal.

Zero exceptions in either direction, across four runs with four different provider sets.

An earlier draft of this analysis reported "33%" wrong — that figure pooled both mutation types into one denominator and understated the real rate. Read per type, **the deleted-element rate is 100%: agreement never once happened to be right about an absence.**

The hypothesis predicted two safe outcomes on a deleted element — providers decline, or providers scatter across different decoys. In practice:

Widening the pool made this clearer, not better. One run recorded 10 unanimous false heals; **7 of those 10 had three independent model families agreeing on the same non-existent element** — three vendors, three architectures, one wrong answer, unanimously. Going 3 → 7 providers moved the removed-element agreement rate 25% → 45% → 58% → 40% with no downward trend and 100% wrong throughout.

The protection is a **byproduct of independence**, and independence alone does not bound the failure rate: a genuinely capable, independent reasoner finds a deleted control's surviving neighbour a *convincing* answer often enough that adding more reasoners does not reliably break the tie. It also cannot be strengthened by asking for more confidence — the failing cases are already maximally confident.

Automation Sandbox is built around this result rather than despite it:

| Design choice | Why |
|---|---|
Heuristic-first, deterministic. A pure C# structural scorer decides on its own, ~23ms for 3,000 controls, zero tokens. |
The LLM is never on the default path. Most healing never touches a model. |
LLM is opt-in and quorum-gated (≥ 2 providers must independently name the same candidate). |
A single model's confidence is worthless here. Agreement is permission to consider a pick — never evidence it is correct. |
A heal commits only after the retried action succeeds; the shipped default mode changes no locators and only routes candidates to a report. |
A wrong pick that cannot actually perform the test step is caught before it is persisted. |
Every decision is written to an audit report (JSON + HTML) — which signal contributed what weight, which providers voted, what the outcome was. |
"The AI healed it" is not an acceptable answer. You can see exactly why. |

**The honest bottom line:** unassisted absence detection is mathematically bounded by the structural score overlap. Automation Sandbox does not claim to have solved it. It makes the failure *visible and declinable* instead of silent and green.

From a clean clone (needs the .NET SDK; the LLM run needs provider API keys):

```
# 1. The deterministic heuristic baseline and threshold sweep - no keys, no tokens
dotnet test TestAutomation/ScenarioRunner/ScenarioRunner.csproj \
  --filter "FullyQualifiedName~LocatorAblationTests.HandBrakeFixture_RunsEndToEndAndReportsMetrics"

dotnet test TestAutomation/ScenarioRunner/ScenarioRunner.csproj \
  --filter "FullyQualifiedName~LocatorAblationTests.HandBrakeFixture_ThresholdSweep"

# 2. Calibrate against your own captured UI tree
dotnet run --project samples/CalibrationCli -- <your-tree.json> --app YourApp

# 3. The live multi-provider consensus evaluation (set provider keys as env vars first)
dotnet test TestAutomation/ScenarioRunner/ScenarioRunner.csproj \
  --filter "FullyQualifiedName~LocatorAblationTests.HandBrakeFixture_LlmConsensus_LiveEvaluation"
```

Every claim is guarded by a committed regression test — see the "Regression guards" lines throughout the [Benchmark & Calibration guide](https://mustafasercansak.github.io/automation-sandbox/docs/benchmark-calibration.html).

**Automation Sandbox** is MIT, pure C#/.NET, seven packages on nuget.org. It does locator self-healing and intent-driven test generation for Windows desktop (FlaUI/UIA3) and web (Playwright). If you maintain UI tests that break on every refactor, I'd genuinely like your feedback — especially from anyone who has shipped locator self-healing and hit the deleted-element problem.