# Should the LLM see the "nice to have" findings? I measured it.

> Source: <https://dev.to/aliaksandrliapin/should-the-llm-see-the-nice-to-have-findings-i-measured-it-1eao>
> Published: 2026-11-12 06:48:44+00:00

[CloudArchitect Lite](https://github.com/aliaksandr-liapin/cloudarchitect-lite) turns a plain-English app description into an AWS architecture diagram and checks it with 25 deterministic rules. For the cheaper models there's one repair turn: the validator's findings go back to the model and it returns a corrected design.

Only errors and warnings go back. The info findings (no encryption at rest, no monitoring, Lambda talking to RDS without RDS Proxy, components not linked to any requirement, cost drivers nobody asked for) are never shown to the model. So they stay in the final design. GPT-6 Luna, for example, left encryption at rest out of 32 of 44 designs in the last benchmark.

Under my post on [how I stop an LLM from inventing cloud services](https://dev.to/aliaksandrliapin/how-i-stop-an-llm-from-inventing-cloud-services-3lpn), a reader, Gautam K, asked the obvious question in two parts: would sending the info findings clean those designs up, and would it make the model start changing things nobody asked it to change? I didn't know, so I measured it.

The full report with every number is [in the repository](https://github.com/aliaksandr-liapin/cloudarchitect-lite/blob/main/docs/product/info-repair-experiment.md).

The naive way is to run the generator twice, once with info findings and once without, and compare. That mostly measures luck: two first drafts from the same prompt differ more than any repair turn does.

So every job makes **one** first draft, then runs **two** repairs from that same draft:

One info rule is never sent: "component not in the verified catalog". An unresolved component is often the honest answer, and asking a model to "fix" it is an invitation to invent a service.

Three economical models (Claude Haiku 5.5, GPT-6 Luna, Gemini 3.5 Flash-Lite), 22 scenarios, 2 repeats: 132 jobs, 264 repairs.

This is the part of the question that's easy to hand-wave, so I made it mechanical. Each repaired design is diffed against its draft by element id: nodes added or removed, a service swapped, edges added, removed or rewired, nodes moved between containers, requirement links dropped.

A change is **targeted** if it touches something a finding named, or adds what the finding asked for (CloudWatch for "no monitoring"). Everything else is **collateral**. I also counted **new findings**: problems in the repaired design that weren't in the draft.

Collateral changes happen anyway, so I needed a baseline. When a draft has no info finding to send, both arms send the *identical* request. The difference between those two answers is pure chance, and that's the noise floor.

Before spending anything, I wrote down what would make me change the app, per model:

"Clearly" means a 90% paired-bootstrap interval that excludes zero. Writing the bar first matters more than it sounds: it's the only thing that stopped me from re-reading a borderline result in my favor later.

| Model | Info findings per design (W → I) | Collateral changes | New findings | Blind score (W → I) | Verdict | 
|---|---|---|---|---|---|
| GPT-6 Luna | 1.23 → **0.00** | no clear difference | no clear difference | 2.36 → 2.43 | **adopt** | 
| Claude Haiku 5.5 | 0.40 → **0.00** | no clear difference | no clear difference | 2.43 → 2.36 | keep as is | 
| Gemini 3.5 Flash-Lite | 0.07 → 0.00 | no clear difference | no clear difference | 1.93 → 1.93 | keep as is | 

**First question: does it clean them up?** Yes, completely. Luna left encryption at rest out of 31 of 44 final designs with today's repair turn, and out of none when the finding was in the message. Haiku went from 0.40 info findings per design to zero.

**Second question: does it change things nobody asked for?** Not measurably. No model showed a clear rise in collateral changes or in new findings. The extra cost is under $0.001 per design. One number to keep an eye on: Luna's "load balancer in a single Availability Zone" warning went from 6 to 10 of 44 designs. That's inside the noise, but it's the first thing I'll check in the next benchmark.

The same panel as in the [0.3 benchmark](https://dev.to/aliaksandrliapin/i-let-claude-gpt-and-gemini-grade-each-others-aws-architectures-1a4l) (one Claude, one GPT and one Gemini reviewer, designs anonymized, 7-dimension rubric) compared arm W and arm I of the same job: 10 pairs per model.

Luna passed every criterion. Flash-Lite almost never trips the info rules (2 of 44 drafts), so there was nothing to measure.

Haiku is the interesting one. Its median blind score came out one step lower with info findings, 2.36 against 2.43. Yet when the reviewers picked a winner pair by pair, they preferred the info version **8 times out of 10**. Those two numbers point in opposite directions, and with 10 pairs one design moves the median by a step.

I'd written "the blind score is not lower", and it was lower. So Haiku keeps today's behavior, and the report says why. If I had let myself re-argue it after seeing the head-to-head numbers, the bar would have been worth nothing.

A per-model setting now lists which info rules the repair turn sends. It's set only for GPT-6 Luna. Haiku, Flash-Lite and the best-quality models (which have no repair turn) are unchanged. It's on main and goes out in the next release.

Smoke run, 132 jobs and the review with one retry: **$6.88** in API calls.

You need Docker and an API key from Anthropic, OpenAI or Google:

```
curl -fsSLO https://raw.githubusercontent.com/aliaksandr-liapin/cloudarchitect-lite/v0.3.1/compose.yaml && docker compose up
```

There's also a [clickable demo](https://aliaksandr-liapin.github.io/cloudarchitect-lite/) with no install. The best questions so far have come from the comments, so if there's something about the pipeline you'd want measured, ask.

Thanks, Gautam, for a question that turned into a better experiment than I'd have designed on my own.