{"slug": "can-you-trust-an-llm-to-fix-a-broken-locator-i-measured-it", "title": "Can You Trust an LLM to Fix a Broken Locator? I Measured It.", "summary": "A developer measured whether requiring multiple independent LLMs to agree can detect when a UI element has been deleted, finding that consensus among providers is real protection but stems from disagreement rather than any model recognizing the absence. The study, part of the Automation Sandbox project, shows that when LLMs agree on a deleted element, they are unanimously wrong, so the system keeps the LLM as an opt-in fallback gated by an independent-agreement quorum and only commits a heal after the retried action succeeds.", "body_md": "*A measured study of multi-provider LLM consensus as a locator-healing signal — and why Automation Sandbox keeps the LLM out of the decision. Originally published on the project site.*\n\nWhen a UI refactor **deletes** an element, a self-healing engine must decline and ask a human — not latch onto a neighbouring button. I tested whether asking several independent LLMs and requiring them to agree can act as that \"the element is gone\" detector.\n\nAcross **four live multi-provider runs** (2026-08-16 to 2026-08-18, 133 usable scenarios):\n\n**The consensus check is real protection — but it comes from providers disagreeing with each other, not from any model recognising that the element is gone.** When they happen to agree on a deleted element, they are unanimously, confidently wrong. That is why in Automation Sandbox the LLM is an opt-in fallback gated by an independent-agreement quorum, and a heal is only ever committed after the retried action actually succeeds — the model is never the decision maker.\n\nFull data, methodology, and the trade-off curves live in the [Benchmark & Calibration guide](https://mustafasercansak.github.io/automation-sandbox/docs/benchmark-calibration.html) (§3, §4, §6). This post is the standalone story; that page is the source of truth for every number.\n\nLocator healing handles the easy 90%: an `AutomationId`\n\nchanges, a label is reworded, a control moves 100px in a layout pass. The engine re-resolves the element from structural evidence — control type, parent, sibling position, name similarity, geometry — and retries.\n\nThe dangerous case is the other kind of change: the element is **deleted outright**. A checkout button is removed; the test that clicked it should now *fail loudly* so a human looks at it. What it must not do is quietly heal onto the \"Cancel\" button next to where \"Submit\" used to be, pass green, and hide a real regression — a \"false heal\".\n\nA pure-heuristic scorer cannot solve this case on its own, and [I proved that rigorously](https://mustafasercansak.github.io/automation-sandbox/docs/benchmark-calibration.html#5-offline-absence-signal-investigation-95): a surviving sibling in a deleted control's container is structurally **indistinguishable** from a control that genuinely moved next to that sibling. Their similarity-score distributions overlap (`[0.665, 0.955]`\n\nfor deleted-element decoys vs `[0.749, 0.874]`\n\nfor true compound drift), so **no confidence threshold, runner-up margin, cluster-density, or control-type filter can draw a line between them.**\n\nThat negative result is what motivated the LLM experiment. Semantic reasoning asks a different question than geometry does — maybe independent models, forced to agree, could tell \"moved\" from \"gone\".\n\nNatural locator drift across real releases is too sparse to benchmark. So I invert it: take a real, organically-built application UI tree and **systematically mutate** its authored locators.\n\n**Leakage protection.** Every candidate `AutomationId`\n\nin a mutated tree is rewritten to the *same* opaque format (`ablation-`\n\n+ SHA-256 hex), not just the target. Otherwise a model could solve the scenario by spotting the \"odd one out\". IDs carry no semantic hint — which means these scores are a **lower bound**: in production, a descriptive `btnSaveDocument`\n\ngives the model more to work with than a synthetic hash does.\n\n**LLM evaluation protocol.** Temperature 0. Every provider's raw vote recorded alongside the agreement telemetry. A scenario counts as **usable** only when ≥ 2 providers actually returned an answer — one opinion can neither agree nor disagree with anything. Token cost is managed by targeting the two informative subsets: 25 compound-drift + 42 removed-element scenarios.\n\nRunning the deterministic scorer at the shipped default (`MinimumConfidence = 0.50`\n\n) over all 176 HandBrake scenarios:\n\n| Mutation tier | n | Correct | False heals | Missed (review) | Score range |\n|---|---|---|---|---|---|\n| Renamed AutomationId | 42 | 40 | 0 | 2 | `[1.000, 1.000]` |\n| Name drift | 25 | 23 | 0 | 2 | `[0.877, 0.950]` |\n| Position shift | 42 | 34 | 0 | 8 | `[0.832, 0.955]` |\n| Compound drift | 25 | 6 | 2 | 17 | `[0.749, 0.874]` |\nRemoved element |\n42 |\n25 |\n17 |\n0 |\n`[0.344, 0.955]` |\n\n**40.5% of deleted elements are false-healed** at the default threshold. On ShareX the same figure is **57.1%**. Raising the threshold trades this away against auto-heal recall linearly — at `0.90`\n\nthe false-heal rate on removed elements is still 5.5% while three-quarters of *all* locators are forced to manual review. There is no free setting.\n\nThis is the gap the LLM was asked to close.\n\nFour runs, 2026-08-16 to 2026-08-18, as the provider pool grew from 3 to 7 and Groq's model was swapped twice:\n\n| Run | Usable (n) | Compound-drift unanimous | Removed-element unanimous |\n|---|---|---|---|\n| 1 (08-16) | 19 | 6 / 7 | 3 / 12 |\n| 2 (08-16) | 39 | 16 / 17 | 10 / 22 |\n| 3 (08-16) | 33 | 14 / 14 | 11 / 19 |\n| 4 (08-18) | 42 | 16 / 17 | 10 / 25 |\nTotal |\n133 |\n52 / 55 |\n34 / 78 |\n\nProviders reached unanimous agreement on **94.5%** of scenarios where a successor existed, and on **43.6%** where the element was gone. That gap is real — it is the *only* signal in the entire project that separates the two populations at all.\n\n**But unanimity is not a safe acceptance gate:**\n\nEvery one of the\n\n52unanimous verdicts on a surviving element was correct.\n\nEvery one of the34unanimous verdicts on a deleted element was a false heal.\n\nZero exceptions in either direction, across four runs with four different provider sets.\n\nAn earlier draft of this analysis reported \"33%\" wrong — that figure pooled both mutation types into one denominator and understated the real rate. Read per type, **the deleted-element rate is 100%: agreement never once happened to be right about an absence.**\n\nThe hypothesis predicted two safe outcomes on a deleted element — providers decline, or providers scatter across different decoys. In practice:\n\nWidening the pool made this clearer, not better. One run recorded 10 unanimous false heals; **7 of those 10 had three independent model families agreeing on the same non-existent element** — three vendors, three architectures, one wrong answer, unanimously. Going 3 → 7 providers moved the removed-element agreement rate 25% → 45% → 58% → 40% with no downward trend and 100% wrong throughout.\n\nThe protection is a **byproduct of independence**, and independence alone does not bound the failure rate: a genuinely capable, independent reasoner finds a deleted control's surviving neighbour a *convincing* answer often enough that adding more reasoners does not reliably break the tie. It also cannot be strengthened by asking for more confidence — the failing cases are already maximally confident.\n\nAutomation Sandbox is built around this result rather than despite it:\n\n| Design choice | Why |\n|---|---|\nHeuristic-first, deterministic. A pure C# structural scorer decides on its own, ~23ms for 3,000 controls, zero tokens. |\nThe LLM is never on the default path. Most healing never touches a model. |\nLLM is opt-in and quorum-gated (≥ 2 providers must independently name the same candidate). |\nA single model's confidence is worthless here. Agreement is permission to consider a pick — never evidence it is correct. |\nA heal commits only after the retried action succeeds; the shipped default mode changes no locators and only routes candidates to a report. |\nA wrong pick that cannot actually perform the test step is caught before it is persisted. |\nEvery decision is written to an audit report (JSON + HTML) — which signal contributed what weight, which providers voted, what the outcome was. |\n\"The AI healed it\" is not an acceptable answer. You can see exactly why. |\n\n**The honest bottom line:** unassisted absence detection is mathematically bounded by the structural score overlap. Automation Sandbox does not claim to have solved it. It makes the failure *visible and declinable* instead of silent and green.\n\nFrom a clean clone (needs the .NET SDK; the LLM run needs provider API keys):\n\n```\n# 1. The deterministic heuristic baseline and threshold sweep - no keys, no tokens\ndotnet test TestAutomation/ScenarioRunner/ScenarioRunner.csproj \\\n  --filter \"FullyQualifiedName~LocatorAblationTests.HandBrakeFixture_RunsEndToEndAndReportsMetrics\"\n\ndotnet test TestAutomation/ScenarioRunner/ScenarioRunner.csproj \\\n  --filter \"FullyQualifiedName~LocatorAblationTests.HandBrakeFixture_ThresholdSweep\"\n\n# 2. Calibrate against your own captured UI tree\ndotnet run --project samples/CalibrationCli -- <your-tree.json> --app YourApp\n\n# 3. The live multi-provider consensus evaluation (set provider keys as env vars first)\ndotnet test TestAutomation/ScenarioRunner/ScenarioRunner.csproj \\\n  --filter \"FullyQualifiedName~LocatorAblationTests.HandBrakeFixture_LlmConsensus_LiveEvaluation\"\n```\n\nEvery claim is guarded by a committed regression test — see the \"Regression guards\" lines throughout the [Benchmark & Calibration guide](https://mustafasercansak.github.io/automation-sandbox/docs/benchmark-calibration.html).\n\n**Automation Sandbox** is MIT, pure C#/.NET, seven packages on nuget.org. It does locator self-healing and intent-driven test generation for Windows desktop (FlaUI/UIA3) and web (Playwright). If you maintain UI tests that break on every refactor, I'd genuinely like your feedback — especially from anyone who has shipped locator self-healing and hit the deleted-element problem.", "url": "https://wpnews.pro/news/can-you-trust-an-llm-to-fix-a-broken-locator-i-measured-it", "canonical_source": "https://dev.to/mustafa_sercan_sak/can-you-trust-an-llm-to-fix-a-broken-locator-i-measured-it-4och", "published_at": "2026-08-30 11:37:50+00:00", "updated_at": "2026-08-30 12:23:47.839375+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-agents", "developer-tools"], "entities": ["Automation Sandbox", "HandBrake", "Mustafa Sercan Sak"], "alternates": {"html": "https://wpnews.pro/news/can-you-trust-an-llm-to-fix-a-broken-locator-i-measured-it", "markdown": "https://wpnews.pro/news/can-you-trust-an-llm-to-fix-a-broken-locator-i-measured-it.md", "text": "https://wpnews.pro/news/can-you-trust-an-llm-to-fix-a-broken-locator-i-measured-it.txt", "jsonld": "https://wpnews.pro/news/can-you-trust-an-llm-to-fix-a-broken-locator-i-measured-it.jsonld"}}