The redesign shipped. The smoke test kept walking the old UI.

A developer at nlqdb describes how acceptance tests that pin literal UI strings caught real product changes but also generated false positives that required manual triage. The team learned that while literal assertions are valuable for detecting drift, they must be self-explaining and time-boxed to avoid masking actual failures.

Originally published at nlqdb.com/blog Our acceptance walkers pin literal UI strings on purpose. A walker asserts the homepage placeholder reads exactly what we ship, that a heading says the words we wrote, that the query composer is where we put it. Drift fails the walk loudly — and a loud failure the moment the surface changes underneath you is precisely the regression detector you want. Silent tolerance is how a broken flow ships green. Then three things happened in one week. A homepage redesign moved the goal input to a different page. A copy edit reworded a heading. The MCP catalog additively grew two tools. None of these broke the product. All three broke the walkers — which dutifully reported 0/9 , and kept reporting it for a week. The reflex is to blame the pinned strings and loosen them into fuzzy matches. That's the wrong lesson. The literals did their job: the surface changed, the walk went red. The actual cost is that a red which mixes "the product broke" with "the test went stale" takes a full manual triage to disentangle — and ours contained both at once. Two flows were red from pure test-drift. One flow was red from a real production wall. Same 0/9. You cannot tell which is which from the number, so every red costs you the same expensive human read regardless of whether anything is actually wrong. placeholder was null, expected "Ask your data anything" is decidable from the artifact alone — you know instantly it's drift, not breakage. failed at step 2 forces you to re-run the whole walk by hand to find out.This is a testing-hygiene pattern, not a product feature: it's for anyone whose end-to-end suite asserts real rendered copy rather than test-ids. The literals are worth keeping — they catch the drift you'd otherwise ship. Just make the red self-explaining and triage it on a clock, or the detector quietly becomes a week of noise that hides the one failure that mattered. nlqdb is a database you query in plain English; this is one of the measurement lessons from keeping our stranger-walk honest as the product underneath it moved.