# Your AI testing dashboards are green. That’s the problem > Source: > Published: 2026-09-17 09:00:00+00:00 The green dashboard is one of the most comforting objects in software engineering. It says the build passed, the tests passed, the service is healthy and the incident queue is quiet. When an AI agent is writing code or repairing tests, though, those green signals can become dangerously local. Each layer can report success while the system’s actual behavior drifts away from what a human intended. I have seen this most clearly in test automation. A locator breaks. An automated repair finds a nearby element, changes the selector and reruns the test. The test passes. The dashboard turns green. But the repaired selector may now point at a different control, a different row or an invisible duplicate. The test has not recovered; it has substituted a new behavior and hidden the substitution. An AI-assisted delivery system usually has at least three observers. The model observes its own task state: it generated a patch, called a tool or declared the work complete. The test harness observes execution: a command returned zero, an assertion passed or a retry succeeded. The production system observes runtime state: requests are flowing, error rates are within thresholds and infrastructure is available. These are useful signals, but they are not the same claim. The problem is not that one of these layers is lying. The problem is that none of them has enough context to detect a mismatch between the claims. A model can complete the requested action against the wrong file. A harness can pass after a locator repair that targets the wrong element. A service can remain healthy while a critical user journey silently stops being exercised. Traditional observability gives us a good vocabulary for signals, traces and logs. [OpenTelemetry’s observability primer](https://opentelemetry.io/docs/concepts/observability-primer/) is a useful starting point. But an AI testing pipeline needs one more question: do the model’s claim, the harness result and the system state describe the same event? Without that correlation, a dashboard is a collection of green opinions rather than evidence. My own work on cross-layer observability began with a practical failure mode: automated locator healing. The useful question was not simply whether a broken locator could be repaired. It was whether the repair selected the intended target. That distinction changes the metric. A system can recover a working locator and still be wrong. In the benchmark behind my current preprint, the LLM-based healer recovered many broken locators, but false-heals occurred roughly one-quarter of the time. A false-heal means the test continues to run while checking the wrong target. The result is especially dangerous because the failure is not loud. There is no red build, no obvious exception and often no spike in application telemetry. The control plane records success precisely when the assurance claim has weakened. The broader lesson applies beyond selectors. An agent can update a timeout until a flaky test passes, remove an assertion that blocks a deployment or map a requirement to a superficially similar implementation. A test suite can remain healthy as an executable artifact while losing its connection to the behavior it was supposed to protect. The [test pyramid](https://martinfowler.com/articles/practical-test-pyramid.html) is still a useful way to think about feedback speed and scope, but AI-generated changes add an oracle problem: who or what is independently checking that the test still means what it used to mean? This is why a vendor’s self-healing rate is an incomplete procurement metric. Buyers should ask for the wrong-target rate, the abstention rate and the evidence generated when a repair changes the semantic target. If a tool cannot show those numbers, its success percentage may be measuring continuity of execution rather than preservation of intent. The fix is not another larger dashboard. It is a cross-layer evidence record that joins the model’s action, the harness’s result and the system’s observed state. Each automated change should carry the original target, the proposed target, the evidence used to make the substitution, the confidence score, the resulting assertion and an explicit human-review status. For an evaluation or procurement conversation, I would ask five questions. First, can the tool distinguish a successful repair from a successful execution against the wrong target? Second, does it measure false-heals on adversarially perturbed locators rather than only measuring whether a test reruns? Third, can an engineer reproduce the decision from an audit record? Fourth, does the tool abstain when evidence is weak, or does it optimize for a green build? Fifth, can its events be correlated with the application’s runtime traces and the model’s tool calls? I would also require a staged operating mode. Autonomous repair can propose a change, but high-impact changes should enter assisted triage until the organization has evidence that the repair preserves meaning. This is similar in spirit to [chaos engineering’s emphasis on disciplined, observable experiments](https://principlesofchaos.org/): the system should reveal its failure modes under controlled conditions before it is trusted in an uncontrolled one. The practical design is modest. Put a shared event identifier across the model trace, test-run record and application trace. Store the before-and-after target. Record the evidence used by the repair. Compare the expected semantic state with the observed state. Alert on disagreement, not merely on exceptions. And make abstention a successful safety outcome when the evidence is insufficient. I have described the architecture and the limits of the measurements in my SSRN preprint, [Cross-Layer Observability for LLM-Assisted Test Automation: A Reference Architecture and Web Feasibility Study](https://papers.ssrn.com/sol3/papers.cfm?abstract_id=6963910). It is a preprint, not peer-reviewed research, and it should be read as a formative feasibility study rather than a universal performance claim. The point is not that one metric solves autonomous testing. The point is that teams need to measure whether their green signals still refer to the same behavior. AI-assisted testing will improve. The organizations that benefit most will not be the ones with the most green checks. They will be the ones that can explain what each green check means, which layer produced it and what independent evidence confirms it. A green dashboard is valuable only when the system has earned the right to be believed. There is also a governance implication. A procurement team should not accept a single aggregate accuracy number as evidence that an autonomous testing product is safe. Ask to see the denominator, the failure taxonomy and the cases where the tool declined to act. Ask whether the benchmark includes changed page structure, ambiguous targets, duplicate controls and permissions that alter what a user can see. Ask how the tool behaves when the model’s rationale is plausible but the target is wrong. Those questions move the conversation from impressive demos to operational accountability. The same discipline belongs in internal engineering metrics. Track the percentage of automated changes that receive independent review, the number of assertions removed or weakened, and the rate at which a supposedly repaired test later requires human correction. Review these numbers by application and by change type. A low failure count may mean the system is reliable, or it may mean the organization is not looking for silent failures. Cross-layer evidence gives leaders a way to tell the difference.