An LLM judge passed renders with defects I planted

A developer's LLM-judged UI pipeline produced nondeterministic verdicts: scoring one unchanged render eight times with gpt-5.6-terra at medium reasoning effort and seed 42, seven runs cleared the 0.85 pass threshold and one landed at 0.82, despite hashed and asserted-identical inputs. Running one component through the full pipeline three times yielded final scores of 0.85 (pass), 0.83 (fail) and 0.85 (pass), which the developer says makes the 0.03-to-0.09 score gaps behind four reported button regressions indistinguishable from reviewer noise. The developer also found a real regression in a nav component (0.81, 0.68, 0.76, 0.87) caused by the refiner receiving none of the component's seven labelled designs while the writer got all seven, and fixed it so the refiner now receives all seven.

The pipeline I wrote about in the harness post https://yamanahlawat.in/articles/harness-around-llms started reporting regressions that weren't there. A button would pass on one refinement attempt and fail on the next. Four times. I hashed the code for its Secondary variant from each attempt's output, and in two of those four cases the code was byte for byte the same. The refiner had changed exactly what its changes made list said it changed, which didn't touch that block, and the score had gone down anyway. Quick recap if you haven't read the earlier post. The pipeline turns a Figma design into a React component, renders it, and asks a vision model to compare the render with the design. The model scores four things layout, colors, typography, spacing up to 0.25 each, and the component passes when the total is at least 0.85 and a handful of hard checks are green. If it fails, a refinement agent edits the code and the whole thing runs again. scoring-one-render-eight-times Scoring one render eight times If the code didn't change and the score did, whatever moved it came after the code, so I took the refiner and the writer out of it entirely. I took one run's stored artefacts for that same button, which comes from Untitled UI's Figma kit: the code, the rendered screenshot and the Figma reference image. Then I had the reviewer score the same version eight times. Before each call the inputs were hashed and asserted identical, so nothing upstream could have changed. A fixed seed was already in place. The pipeline passes seed=settings.AI AGENT.SEED to every agent, with SEED set to 42, and sets no temperature. The reviewer at the time was gpt-5.6-terra at medium reasoning effort. Here's what came back: Seven of the eight runs cleared 0.85 and one landed at 0.82, so the same code, the same screenshot and the same design both passed and failed depending on which call you happened to look at. The yes-or-no fields the reviewer also returns whether the schema is valid, whether the structure is complete were identical on all eight runs. I also ran one component through the whole pipeline three times, with the same design and the same pipeline code each time: | Run | Trajectory | Final | Verdict | |---|---|---|---| | 1 | 0.72 → 0.58 → 0.85 | 0.85 | pass | | 2 | 0.00 → 0.49 → 0.68 → 0.83 | 0.83 | fail | | 3 | 0.75 → 0.79 → 0.85 | 0.85 | pass | The loop stops at the first attempt that clears the gate, so a passing score tends to sit right at 0.85, inside the band the reviewer wanders around in. That undid a lot of my own reasoning. Most of the comparisons in my build log at the time were between scores 0.03 to 0.09 apart, including the four button regressions I started with, and none of them can be told apart from the reviewer disagreeing with itself. The harness post had the same problem: it read dips of 0.01 to 0.02 in refinement trajectories as refinement going backwards, and I've added a note there. Not every drop was noise, though. One nav went 0.81, 0.68, 0.76, 0.87 across four attempts because the refiner really had broken a version: its instructions said "A fix for one version must not regress a version that already passes", and the trace showed it received none of the component's seven labelled designs, while the writer got all seven on every call. I changed that, so the refiner now gets all seven too. computing-the-verdict-in-code-wasnt-enough Computing the verdict in code wasn't enough In the harness post I made a point of deriving the verdict in code so the model never gets to decide it. This is the current version: Every line in that property held steady across the eight runs except one. The llm review score line is the model inventing a number on the spot, and the code compares it with a constant, so the verdict really is computed in code and it still inherits all of the model's variance through that one comparison. The loop also picks which attempt to keep by sorting on review.passed, review.llm review score , and the noise reaches both parts of that key. I'd picked 0.85 after running a number of Figma designs through the pipeline, because above it the output mostly looked good, and it's still the gate today. the-same-study-on-the-newer-reviewer The same study on the newer reviewer Since then every agent in the pipeline has moved to gpt-6.1-sol at high reasoning effort, so I ran the eight-call study again. This time I used a nav from Untitled UI's free Figma community kit, so I can show the images, and two of its versions: the closed mobile header, and the open panel, the menu opened with its subnav expanded, whose render is over 4,000 pixels tall. | Reviewer | Closed header | Open panel | |---|---|---| | gpt-6.1-sol , high effort | 0.92 to 0.96, passed 8/8 | 0.90 to 0.93, passed 8/8 | | gpt-5.6-terra , medium effort | 0.95 to 0.99, passed 8/8 | 0.89 to 0.99, passed 8/8 | The terra rows are the old reviewer on the same code and the same two images. On the open panel its spread was 0.10, about what I saw the first time, and sol's was 0.03. On the closed header both spreads were 0.04. I changed the model and the effort together, so I can't say which of the two did it. The top row of that image is the design and the render, and they're very close, so a steady score at or above 0.90 was the right answer. When I gave the reviewer the wrong design to compare against the closed header's render against the design of a third version, the open menu with its subnav collapsed , it scored 0.46 to 0.54 and failed all three runs. ten-planted-defects Ten planted defects That covers a render that's right and one that's completely wrong. The refinement loop mostly deals with the cases in between, so I planted some small defects whose answers I knew. Each defect is one Tailwind class changed in the generated styles, font-semibold to font-normal on the nav labels, px-4 to px-10 on the rows, a hidden on the menu button and so on. Each one was rendered in the same sandbox as the real thing and scored three times by the reviewer. The bottom row of the image above shows two of them. Freshly rendered and clean, the two versions scored 0.93 to 0.95, and the 30 defect runs scored 0.82 to 0.90: | Render | Production scores | Runs failed | |---|---|---| | Closed header, clean | 0.93, 0.95, 0.94 | 0/3 | | Grey header background | 0.84, 0.87, 0.89 | 1/3 | | Wider side padding | 0.83, 0.83, 0.82 | 3/3 | | Menu button missing | 0.84, 0.84, 0.83 | 3/3 | | Shorter header | 0.84, 0.83, 0.83 | 3/3 | | Open panel, clean | 0.93, 0.94, 0.93 | 0/3 | | Red nav labels | 0.86, 0.89, 0.84 | 1/3 | | Regular-weight nav labels | 0.89, 0.90, 0.88 | 0/3 | | Larger item titles | 0.86, 0.87, 0.84 | 1/3 | | Wider icon gap | 0.84, 0.84, 0.84 | 3/3 | | Wider row padding | 0.89, 0.87, 0.89 | 0/3 | | All-posts link missing | 0.89, 0.88, 0.86 | 0/3 | The reviewer noticed every defect, in the sense that the score went down. The gate failed all three runs on only 4 of the 10, and never failed the regular-weight labels, the wider row padding or the missing "All blog posts" link. A defect costs 0.04 to 0.11, and how much it costs doesn't follow how visible it is: the regular-weight labels are obvious in the image and cost about 0.04. On these runs a gate at 0.91 would have separated every clean run from every defect run, and I took that number forward without noticing that the eight-run study above had already scored the same clean panel 0.90 once. Planting known defects to test a judge isn't new. Finding Blind Spots in Evaluator LLMs with Interpretable Checklists opens in a new tab https://arxiv.org/abs/2406.13439 Doddapaneni and others, EMNLP 2024 built 2,400 perturbed text answers across 22 kinds of perturbation, and the evaluator LLMs they tested "failed to identify quality drops in over 50% of cases on average". The result looks similar, since the 0.85 gate here let about half of the defect runs through, but the cause is different: their judges didn't notice the drop, and my reviewer's score went down every time and the threshold let it through. Their paper also found judges did better with a reference answer to compare against, which is how my reviewer works. yes-or-no-checks-scored-in-code Yes-or-no checks, scored in code The plan after the first study was to stop asking the reviewer for a number, because its yes-or-no fields had held still. I wrote 15 checks across the same four dimensions every element present, text colors match, font weights match, padding matches and so on , asked gpt-6.1-sol with the same settings to answer each one true or false with a sentence of evidence, and computed the score in code with each dimension still worth 0.25. Over eight runs on the open panel it scored exactly 0.50 every time, failing the same 7 of 15 checks. On the closed header it moved between 0.646 and 0.792, with two checks flipping between runs. The open panel's failures point at differences that are real but small, like this one: The render uses neutral black and grey text where the reference uses dark navy and blue-grey. The instructions say a visible difference is a fail, and on a render this close the checks fail anyway on differences too small to matter. So it caught only 4 of the 10 defects, the ones whose check had passed on the clean render. Two of those four, the regular-weight labels and the missing link, are defects the 0.85 gate never failed. cloudflare-clef Cloudflare Clef I tried Cloudflare's Clef next because it's built for this kind of classification and it's fast. It takes the two images, a sentence naming the version and the same 15 checks, and returns a probability of yes for each check in 0.7 to 2.7 seconds, with the same answers on every call. It doesn't get the code or the Figma data, and any image over 300,000 pixels has to be scaled down to fit its window, so it sees the open panel at a much lower resolution than the reviewer does. On the nav it was sure about three defects: the grey background, the missing menu button and the red labels, at 0.05 to 0.16, where the lowest answer it gave any clean render was 0.476. It missed the rest, and on the missing link it was confidently wrong, at 0.858. four-more-components Four more components By this point I had two ideas from one nav: raise the gate to 0.91, and fail a render whenever Clef answers any check below 0.2, which I'll call the Clef veto. I wrote both down, then took four more components from the same Untitled UI kit a desktop FAQ, a desktop hero, a call-to-action strip from a homepage and the open mobile menu of a different nav and planted 27 defects of the same kinds in them. The reviewer scored each clean render five times and each defect three times, and Clef scored each render once. The veto held up, and 0.91 turned out to be too high. The hero scored 0.73 to 0.78 with nothing planted in it, so I compared it with the design: The cards are flipped and stacked the wrong way round, and the reviewer said so: "The illustration differs substantially: the purple card slopes upward instead of downward". The older reviewer had passed this render at exactly 0.85, the new one fails it every time, and Clef answered every check above 0.55. A broken render underneath says nothing about the defects planted on top of it, so I left the hero out of the table, a call I made after seeing its scores. The table covers every clean run from the current reviewer except the hero's 37, across the nav's two versions and the clean FAQ, CTA and menu and the 60 defect runs on the FAQ, CTA and menu: | Rule | Clean runs passed | Defect runs failed | |---|---|---| | Score ≥ 0.85 today | 37/37 | 32/60 | | Score ≥ 0.91 | 36/37 | 47/60 | | Score ≥ 0.85, Clef veto | 37/37 | 42/60 | | Score ≥ 0.91, Clef veto | 36/37 | 53/60 | | Score ≥ 0.88, Clef veto, picked afterwards | 37/37 | 45/60 | - The 0.91 gate fails the 0.90 clean run from the eight-run study, which is a good render failing on noise. - The veto never fired on a clean render and caught 10 more defect runs at 0.85. It fired on 7 of the 20 defects, mostly changed colors and missing elements, including a red subheading that only moved the reviewer's score from 0.99 to 0.95 or 0.96. - It missed both grey backgrounds, the FAQ's missing description, two of the three padding changes, and every change to a size, weight or gap. These scores are the reviewer's alone. The experiment doesn't run the TypeScript compiler, so the reviewer guesses ts compiles and the hard checks in passed aren't the ones production would apply. what-i-would-change What I would change On the nav, the FAQ, the CTA and the menu, every defect run scored below every fresh clean run of the render it was planted in, one by only 0.02, less than the 0.03 to 0.04 the score moved on one clean render scored eight times. So the score reacted to every defect, by amounts one fixed gate can't separate from its own noise. Clef's veto caught 6 of the 8 color changes and 3 of the 5 missing elements, and never fired on a clean render. With the gate at 0.88, which I picked after seeing these scores, the two together fail 45 of the 60 defect runs on the FAQ, CTA and menu and pass all 37 clean runs, which come from seven renders. - Run Clef on every version next to the reviewer, and fail the version when any check comes back under 0.2. The check that failed goes to the refiner as an issue, so it knows, say, that the text colors are wrong. - Raise VALIDATION PASS THRESHOLD from 0.85 to 0.88. The lowest clean run in 37 was 0.90, on a panel that averaged 0.915 with a standard deviation of 0.009, and 0.88 is the highest gate more than three deviations below that average. I picked it after seeing the data, so the next components the pipeline generates are its test. - Keep the reviewer as the main judge. Clef missed the hero's flipped illustration, and every size, weight and gap change on the FAQ, CTA and menu. - Rerun the planted defects whenever the reviewer model changes. I set 0.85 by eye on earlier models, and on the same closed header the old reviewer scored 0.95 to 0.99 where the new one scores 0.92 to 0.96. What gets through 0.88 with the veto is all on the FAQ and the menu: bolder questions, a larger heading, larger footer links, regular-weight rows and two grey backgrounds. The clean FAQ scored 0.99 on all five runs and its bolder questions 0.92 to 0.94, so the reviewer saw that change, and one absolute gate can't sit right for a component that starts at 0.99 and one whose clean runs dip to 0.90. Gating on the drop from a component's own clean score would need a known-good render to compare with, and while the pipeline is generating a component it doesn't have one. Comparing two images might work better than scoring one. In MLLM-as-a-Judge opens in a new tab https://arxiv.org/abs/2402.04788 Chen and others, ICML 2024 , multimodal judges "demonstrate remarkable human-like discernment in Pair Comparison" while their scores diverge from people's. The loop already has two renders to compare whenever it picks which attempt to keep, and I haven't tried asking the reviewer which one is closer to the design. The other limits: five components from one free kit, and 37 defects that I designed, 7 of them on the hero and left out of the rules table. I haven't measured what the higher gate costs in extra refinement attempts, or what a Clef call costs next to a reviewer call. The veto's clean record is 37 runs but only seven renders, since Clef gives the same answer every time. And Clef saw every desktop render, the open panel and the other nav's open menu scaled down to 300,000 pixels, so I don't know how it does on those cut into full-size pieces.