Upcoming Workshop: Citizen Developer Agentic Software Factory on AWS 11.4.26 @ 12PM PST
How the iac-bench 2026-10 results were produced: the briefs, the evaluation setup, mechanical scoring against real tofu plans, the calibrated LLM judge, statistics and limitations.
Release 2026-10 · run 2026-10-22, judged 2026-10-23
This document describes how the 2026-10 iac-bench results were produced: what the models were asked to do, how their output was scored, how the scores were aggregated, and what the numbers cannot tell you. The results are in the 2026-10 results article. Every scored item is defined in the scoring criteria reference.
iac-bench measures how well a language model writes reusable infrastructure-as-code modules for OpenTofu (the open-source Terraform fork). A model gets a short request of the kind a platform engineer receives from a colleague, and must deliver a module another team could adopt.
Producing configuration that plans is the minimum. The benchmark measures what a senior reviewer would look for:
The 2026-10 release covers module generation only, on three fixtures, one per major cloud.
| Fixture | Cloud and service | Audience named in the brief | Interface budget |
|---|---|---|---|
| G001 | AWS · S3 bucket | Application developers who "fill in a couple of obvious inputs" | 8 inputs (zero credit at 16) |
| G002 | GCP · Cloud SQL for PostgreSQL | A DBA team that "knows PostgreSQL and Cloud SQL deeply" | 25 inputs (zero credit at 35) |
| G003 | Azure · App Service (Linux, containers) | An SRE team that "knows Azure well" | 20 inputs (zero credit at 30) |
Each fixture is defined by a brief, a rubric, and a reference matrix. All three are
published in the repository under fixtures/generation/ and testdata/solutions/.
Briefs are short, realistic requests of two to five sentences. They state what is needed, who will use it, and the one or two facts about the environment that settle the design: for G002, that each environment already has its own VPC with private services access, and that every engineer and workload has a Google identity.
Briefs do not list security, durability, or hygiene requirements. Real requests rarely do, and listing them would turn the most valuable signal (does the model know to do these things?) into an instruction-following check. The rubric's criteria are therefore hidden from the model, but each core criterion is held to a rule: it must follow either from the brief or from practice that is universal for the service. A model is never penalized for failing to guess a preference the brief leaves open. Service-specific touches that go beyond universal practice are scored separately as expertise (§4.3).
Each model works as a tool-using agent in an isolated working directory, using the
Inspect framework. Every model gets the same system prompt
(prompts/generate-module.md), the same tools, and the same limits.
| Setting | Value |
|---|---|
| Tools | list_files ,read_file ,write_file ,tofu_validate ,tofu_plan ,tofu_fmt ,submit |
| Deliverable | A root module ( *.tf ) plusminimal.tfvars , which gives a literal value for every variable without a default and must plan cleanly |
| Remote modules | Not allowed (calling one fails the evaluation); local submodules allowed |
| Limits | 150 messages, 30 minutes wall clock per attempt |
| API retries | Up to 3 per request, then the attempt ends as an error (none occurred) |
| Planning | tofu plan against real cloud accounts (AWS, GCP, Azure sandboxes). Nothing is ever applied. |
| OpenTofu | 1.12.5 |
| Providers | Each module's own version constraints, resolved at run time |
The tofu_plan tool plans against the same real cloud accounts used for scoring, so a
model can verify that its module plans before submitting.
| Model | API identifier | Access | Weights |
|---|---|---|---|
| Claude Opus 5.5 | claude-opus-5-5 |
Anthropic API | Closed |
| Claude Fable 5.1 | claude-fable-5-1 |
Anthropic API | Closed |
| Claude Sonnet 5.5 | claude-sonnet-5-5 |
Anthropic API | Closed |
| Claude Haiku 5.5 | claude-haiku-5-5 |
Anthropic API | Closed |
| GPT-6 Astra | gpt-6-astra |
OpenAI API (Responses) | Closed |
| GPT-6 Luna | gpt-6-luna |
OpenAI API (Responses) | Closed |
| Gemini 3.1 Pro | gemini-3.1-pro-preview |
Google Gemini API | Closed |
| Gemini 3.6 Flash | gemini-3.6-flash |
Google Gemini API | Closed |
| Kimi K3 | moonshotai/kimi-k3 |
OpenRouter | Open |
| DeepSeek V4 Pro | deepseek/deepseek-v4-pro-0813 |
OpenRouter | Open |
All models ran with their providers' default sampling and reasoning settings. No temperature, reasoning-effort, or token-budget overrides were set. The GPT-6 models were called through OpenAI's Responses API, because its Chat Completions API rejects tool use combined with reasoning for these models. It changes only the API used, and no model setting.
Pinning. Except for DeepSeek V4 Pro (-0813), the providers' APIs returned these
identifiers as undated aliases, and none reported a dated snapshot. Results are therefore
tied to the run date: all generations ran on 2026-10-22, and all judging on
2026-10-23. Hosted models can change behind an alias without notice.
Each model ran each fixture three times, as independent attempts with no shared state: 10 models × 3 fixtures × 3 samples = 90 attempts. Every attempt is scored and published. No attempt was re-run, discarded, or chosen after the fact.
Each attempt passes through four layers.
| Layer | What it measures | Nature | Weight |
|---|---|---|---|
| 1. Gate | The module plans as a root module with its own minimal.tfvars |
Mechanical, pass/fail | Prerequisite for layer 2 |
| 2. Rubric and expertise | Fixture-specific consequences: security, data protection, audience fit, service expertise | Mechanical, itemized and weighted | 70% |
| 3. Judge | Design and craft against a published quality doctrine | LLM judge, calibrated against a human | 30% |
| 4. Statistics | Inputs, SLOC, tokens, time, estimated cost | Mechanical, descriptive | 0% (reported only) |
Composite = 0.70 × mechanical + 0.30 × judge grade, on a 0–100 scale.
The runner keeps only the variables that have no default from the model's own
minimal.tfvars, then runs tofu plan on the module as a root module. A module that does
not plan fails the gate:
Each fixture's rubric.yaml declares weighted items, checked by a declarative check engine
(docs/generation-check-engine.md). The engine never looks at variable names. It
evaluates outcomes in two places:
count/ for_each, dynamic blocks and each.*.
The perturbation sweep. For every documented input, the runner plans the module once
more with that one input changed: booleans flipped, and enumerated values from contains()
validations tried (up to five alternatives). The sweep reveals what an input actually
controls without the engine knowing what the model named it. A setting reached through a
direct toggle, a value inferred from an environment, or a derived local all score the same.
Collection-typed and free-string inputs are not perturbed.
The solver. When an item fails at defaults and in the sweep, a solver reads the module's source and searches for inputs that would reach the item: up to three inputs changed at once, with values drawn from the module's own comparisons, validations and map keys, plus a few well-formed sample values per fixture (such as a resource ID). Inputs that would fail a validation or precondition are rejected. The runner then plans each proposed input set, and only a successful plan that shows the item counts. The source proposes and the plan decides.
Items take one of five postures, depending on what the item protects:
| Posture | Meaning | Passes when |
|---|---|---|
| Hardcoded | Security posture. An input that can weaken it is the defect, even if the default is safe. | Every value the source can produce is secure (a literal, or an input confined to secure values, such as a choice between two encrypted modes), the resource exists unconditionally, and it stays secure in every sweep plan |
| On by default | Safe by default, with a legitimate reason to turn it off | On at defaults with a real way to turn it off. Always-on with no opt-out earns partial credit (0.75); off by default but reachable earns 0.5. |
| Offered | A capability the audience should be able to reach | Present at defaults or in at least one sweep plan |
| Graded | Better and worse mechanisms that both work | Credit for the best mechanism present |
| Budget | Interface size fits the audience | Full credit at or under the soft count, falling linearly to zero at the hard count |
Items are grouped by role:
Two H items per fixture are mandatory: the most severe security failure modes. Failing one is a security finding (⚠). It does not cap the score, but it marks the attempt unshippable.
Each fixture also lists expertise items. These are service-specific touches an experienced engineer adds without being asked, such as S3 bucket-owner-enforced ownership or IAM database authentication on Cloud SQL. They are checked mechanically and pooled into the mechanical score at weight 1 each, against rubric weights of 2–18. The rubric therefore decides whether a module meets the bar, and expertise separates modules that do.
Expertise items come from observation. Candidates come from differences between models and from the judge's rationales. Each is approved by the doctrine owner and records where it came from. An item must not reward inventing requirements the brief never stated.
mechanical = (Σ rubric weight × credit + Σ expertise credit) / (Σ rubric weight + number of expertise items)
The rubric weight totals are 86 (G001), 97 (G002) and 72 (G003), including the baseline items. Each fixture has 7–9 expertise items.
Each fixture ships hand-written reference modules that the rubric must classify correctly:
ref_lean is opinionated, with few inputs and choices inferred from a concept such as
environment.ref_flexible is more configurable.bad_passthrough: every setting is an input. bad_minimal: a bare resource. bad_unsafe: one critical setting is exposed as an input. bad_open_slot: a locked-down app with an open deployment slot.
The self-test runs this matrix against real cloud plans. If a deliberately good design fails the rubric, the rubric is wrong. The matrix exists so that the rubric rewards outcomes rather than one author's preferred shape.
The judge is Claude Opus 5.5. It grades each module against the benchmark's quality
doctrine (docs/good-opentofu.md, edition af3ae1dd9fdc1c0c) on 14 sections, scoring
each 0 (no credit), 1 (partial) or 2 (full):
The weighted mean is scaled to 0–100.
The judge grades only what the rubric cannot. The doctrine sections the rubric already scores are given to the judge as context but are not graded: insecure configurations, destructive defaults, unstated assumptions, tagging, versions, and provider declarations. A defect therefore never costs a module in both components.
Blind to authorship. The judge sees the brief, the doctrine, and the module's files: HCL first, then tfvars, then documentation. It never sees which model wrote the module. The files are marked as untrusted data, so instructions inside a module are not followed.
Sampling. Each module is graded three times independently. Each section takes the median of the three grades, and the judge returns one sentence of evidence per grade.
A judge grade counts toward the composite only after the judge is calibrated against the doctrine owner (the human who maintains the doctrine), fixture by fixture:
A second judge (GPT-5.5) graded the same modules during calibration. The doctrine owner reviewed in detail the cells where the two judges disagreed. Agreement for Claude Opus 5.5:
| Fixture | Cells | Exact agreement | Lowest section |
|---|---|---|---|
| G001 | 112 | 0.929 | 0.75 (Interface design, Scope, File layout, Documentation) |
| G002 | 112 | 0.973 | 0.875 (Interface design, File layout, Documentation) |
| G003 | 112 | 0.982 | 0.75 (Documentation) |
GPT-5.5 failed the gate on G001 (Interface design) and G003 (Documentation, Sensitive values). A single judge is used across all fixtures, so that grades are comparable between fixtures.
The final modules. The 2026-10 modules were graded with the same judge model, judge code, doctrine edition and sampling as the calibration. Calibration was carried over on that basis; the doctrine owner did not re-review the final modules. Each grade file records where its calibration came from.
tofu plan. It can miss a
setting that the source omits and the provider defaults insecurely, and it can grade a
module that does not plan. The rubric covers both cases. The two components are meant to
complement each other.
Within a fixture the median is used, so one broken attempt out of three is reported as one broken attempt instead of being averaged into a result that hides it.
Three samples per cell is a small sample, so the overall score is reported with a bootstrap 95% confidence interval:
Each model also gets a rank interval: the 2.5th to 97.5th percentile of its rank across joint bootstrap draws, following the LMArena convention. A ranking claim is made only where intervals separate. Pairwise win probabilities are published in the data release.
With n = 3, the bootstrap is coarse, and intervals are wide for models with one outlier attempt. Read them as a statement of how far the data supports a ranking, not as precise coverage.
A module that fails the gate keeps its judge grade, so its composite is about 25–30 rather than 0. It is flagged ✗ and counted in the results tables. The judge grade is kept because the judge grades design and craft, which a plan failure does not erase. The flag shows that the attempt was unshippable. Security findings are handled the same way: flagged ⚠ and counted, but no score cap.
Estimated cost per attempt uses list prices from OpenRouter, dated 2026-10-23, for every model, including closed models routed through their own APIs. Cache reads are charged at 0.1× and cache writes at 1.25× the input price. Judge calls are excluded. Token counts, wall-clock time, SLOC and input counts come from the eval logs. None of them affects any score.
iac-bench is developed by Massdriver, which builds infrastructure tooling. Massdriver wrote the briefs, rubrics, references and doctrine, and its staff acted as the calibrating human.
The results were scored by the code tagged
2026-10 in the repository.
At that tag, the repository contains everything needed to reproduce or audit a score:
src/iacbench/. prompts/generate-module.md. fixtures/ and testdata/. docs/good-opentofu.md. composite-scoring-policy.yaml. prices.yaml.
The 2026-10 data release (results/2026-10/ in the repository) contains:
The identifiers of the sandbox cloud accounts (AWS account, GCP project and Azure subscription) are replaced with placeholders wherever they appear; the release lists every redacted file. Results are licensed under CC BY 4.0.
To reproduce one cell:
uv run inspect eval src/iacbench/task.py@generate_module -T fixture=G002 \
--model anthropic/claude-sonnet-5-5 --epochs 3
uv run iacbench judge --grade --samples 3 --fixture G002 \
--candidates <collected modules> --model anthropic/claude-opus-5-5
@misc{iacbench2026,
title = {iac-bench: Evaluating LLM-authored OpenTofu modules},
author = {{Massdriver}},
year = {2026},
month = {10},
note = {Release 2026-10},
url = {https://github.com/massdriver-cloud/iac-bench}
}
How Claude Fable 5.1 did on iac-bench 2026-10: an overall composite of 86.9, scores per fixture, strengths, weaknesses and the patterns in its OpenTofu modules.
How Claude Haiku 5.5 did on iac-bench 2026-10: an overall composite of 68.0, scores per fixture, strengths, weaknesses and the patterns in its OpenTofu modules.
How Claude Opus 5.5 did on iac-bench 2026-10: an overall composite of 88.9, scores per fixture, strengths, weaknesses and the patterns in its OpenTofu modules.