How Well Do LLMs Write Infrastructure Modules? iac-bench 2026-10 Results

Upcoming Workshop: Citizen Developer Agentic Software Factory on AWS 11.4.26 @ 12PM PST

Register Here!

We asked ten language models to write reusable OpenTofu modules for AWS, GCP and Azure, then scored 90 modules against real plans and a calibrated judge. Here is how they did and the patterns behind the scores.

iac-bench release 2026-10 · run 2026-10-22 · 10 models · 3 clouds · 90 attempts

We asked ten language models to write reusable OpenTofu modules for three common cloud services intended for a specific audience:

Each request was a few sentences describing the high level use case of the module and the intended audience. We never listed the specific security, durability or hygiene requirements a senior reviewer would expect.

We scored the 90 resulting modules on two components:

tofu plan output. This article reports the results and the patterns behind them:

H13). H11). B1).

Composite = 0.70 × mechanical + 0.30 × judge. Fixture columns are medians of three attempts. ✗ = attempts whose module did not plan; ⚠ = attempts failing at least one mandatory security item. Costs are estimates at list prices on 2026-10-23 and exclude judging.

Figure 1. Overall composite (mean of the three per-fixture medians). Whiskers show the bootstrap 95% CI; the right-hand labels give the 95% rank interval.

How to read the ranking. Three samples per model per fixture is enough to separate groups but not neighbors:

Claude Haiku 5.5's interval is the widest in the release. It is driven by one fixture where two of three modules failed to plan. Pairwise win probabilities are in the data release.

Figure 2. Median composite of three attempts, per model and fixture.

Fixture scores are not interchangeable. G001 (S3) punishes omission: the brief's audience is developers who "don't know S3 well," so every hygiene decision falls to the module. The models that lose the most here:

G002 (Cloud SQL) rewards locking down a specialist's interface without taking their controls away. G003 (App Service) depends heavily on two settings the Azure provider leaves insecure by default.

Some criteria were missed by nearly every model, regardless of vendor, size or overall rank. These are the clearest signal in the release, because they do not depend on which model you pick.

Figure 3. Share of attempts earning full credit on the rubric items most often missed, by model. Item definitions are in Scoring criteria.

These criteria were missed by most of the field.

Access logs that cannot be delivered (G001 H13, 30/30 missed).

H9). Databases dropped when removed from the input (G002 H11, 24/28 missed).

google_sql_database defaults to deleting the database, and its data, when the resource is removed.databases list therefore drops that database's data on the next apply.deletion_policy = "ABANDON" or tying it to deletion protection. Backups that cannot be turned off (G002 H4/ H5). This one splits by rank:

E1, 0/28). By default Cloud SQL deletes them with the instance. Provider constraints that admit the next major version (B1).

>= 5.0, or a window such as >= 5.0, < 7.0 that spans two majors.~> 4.0. The habit looks provider-specific rather than model-specific: the same models that bound azurerm correctly left aws and google open.

Expertise that only a few models showed:

roles/cloudsql.instanceUser, G002 E6): 13/28 granted it for the IAM database users they created. Without it, those users cannot log in.E8): 12/28. E5): 6/28.

Figure 4. Attempts with at least one mandatory security failure, and attempts that did not plan, out of 9 per model.

Each fixture has two mandatory items: its most severe security failures. Twenty-five of 90 attempts failed at least one.

Secure default, open override. The doctrine treats this as a defect even when the default is safe. A module's security posture has to hold for every consumer, and an input that can silently weaken it will eventually be used to do so. Models in this pattern often document the risk in the same breath: one description reads "Do not weaken this without a security review" above a validation list that allows exactly that.

Insecure provider default left in place. This failure cannot be seen by reading the source, because nothing is written down. The most-missed mandatory item in the release is G003 H2 (basic-auth publishing, 10/28). Several of these modules set HTTPS-only, TLS 1.2 and FTPS-off correctly in the same block, and still miss the two publishing flags the Azure provider enables by default.

Failed gates. Four attempts produced modules that did not plan:

Figure 5. Mechanical-only and judge-only scores per model (mean of per-fixture medians).

The ranking holds without the judge. Ranked by the mechanical score alone, the ten models fall in exactly the composite order. The judge's 30% shifts scores but never reorders the leaderboard.

The two components disagree on individual modules, and the disagreements are systematic.

What the judge misses. It reads source, so it cannot see:

What the rubric misses. It checks planned outcomes, so it cannot see:

The judge scored Documentation 0 in 26 of 90 attempts. It found, for example, a databases input that fails as soon as one database is declared, which the rubric credited.

Disagreement about restraint. The largest single gap is GPT-6 Astra on S3:

A reader who rejects the rubric's view can read the judge-only column, and the reverse.

Self-preference. The judge is Claude Opus 5.5. Claude models take three of the top four judge-only ranks, alongside GPT-6 Astra. But the composite order, and Claude Opus 5.5's first place, are identical under the mechanical score alone, where no model judges any other.

Figure 6. Overall composite against the estimated cost of one attempt at each fixture, at list prices on 2026-10-23. The dashed line is the Pareto frontier: models no other model beats on both cost and score.

Generation cost varies by a factor of about 280 across the field, and it is only loosely related to score.

The Pareto frontier, the models no other model beats on both cost and score, runs through:

Claude Fable 5.1 costs about four times as much as Claude Opus 5.5 for a slightly lower score. Claude Haiku 5.5 is the second-cheapest model and would sit near the frontier, but for its two failed Cloud SQL attempts.

Costs are estimates: token counts multiplied by list prices. They exclude judging and any provider-specific caching discounts beyond a flat cache-read and cache-write multiplier.

Documentation is the judged section most often scored 0: in 26 of 90 attempts. Nearly every module has documentation. The zeros come from documentation that is wrong:

Under the doctrine, any wrong statement scores 0, however small.

Longer documentation produces more of these errors. The models that write long READMEs (Claude Fable 5.1, GPT-6 Astra, Kimi K3) give the judge more claims to check:

The same model, prompt and fixture can produce very different modules. Nine of the 30 model-fixture cells spread by more than 10 points across their three attempts. Four spread by more than 20:

Two consequences:

Every model in the top group has all three attempts within 11 points on every fixture. Claude Opus 5.5's widest spread is 8.5 points.

The main limitations of this release:

Full discussion: Methodology §7–8.

The data release (results/2026-10/ in the repository) contains:

It is licensed CC BY 4.0. The evaluation code, fixtures, rubrics, reference modules and doctrine are in the repository.

@misc{iacbench2026,
  title  = {iac-bench: Evaluating LLM-authored OpenTofu modules},
  author = {{Massdriver}},
  year   = {2026},
  month  = {10},
  note   = {Release 2026-10},
  url    = {https://github.com/massdriver-cloud/iac-bench}
}

How Claude Fable 5.1 did on iac-bench 2026-10: an overall composite of 86.9, scores per fixture, strengths, weaknesses and the patterns in its OpenTofu modules.

How Claude Haiku 5.5 did on iac-bench 2026-10: an overall composite of 68.0, scores per fixture, strengths, weaknesses and the patterns in its OpenTofu modules.

How Claude Opus 5.5 did on iac-bench 2026-10: an overall composite of 88.9, scores per fixture, strengths, weaknesses and the patterns in its OpenTofu modules.