{"slug": "iac-bench-2026-10-methodology", "title": "iac-bench 2026-10 Methodology", "summary": "Iac-bench's 2026-10 release, run on 2026-10-22 and judged on 2026-10-23, measures how well language models write reusable OpenTofu infrastructure-as-code modules across three cloud fixtures: AWS S3 (8-input budget, zero credit at 16), GCP Cloud SQL for PostgreSQL (25 inputs, zero credit at 35) and Azure App Service on Linux containers (20 inputs, zero credit at 30). Each model runs as a tool-using agent under the Inspect framework with identical prompts, tools and limits — 150 messages and 30 minutes wall clock per attempt — and is scored mechanically against real tofu plans on live AWS, GCP and Azure sandboxes using OpenTofu 1.12.5, with nothing ever applied. The benchmark covers module generation only, hides its rubric from the model, and forbids remote modules while allowing local submodules.", "body_md": "Upcoming Workshop: Citizen Developer Agentic Software Factory on AWS 11.4.26 @ 12PM PST\n\n[Register Here!](https://streamyard.com/watch/umMggUFkVMNu)\n\nHow the iac-bench 2026-10 results were produced: the briefs, the evaluation setup, mechanical scoring against real tofu plans, the calibrated LLM judge, statistics and limitations.\n\n**Release 2026-10 · run 2026-10-22, judged 2026-10-23**\n\nThis document describes how the 2026-10 iac-bench results were produced: what the models\nwere asked to do, how their output was scored, how the scores were aggregated, and what\nthe numbers cannot tell you. The results are in [the 2026-10 results article](https://www.massdriver.cloud/blogs/iac-bench-2026-10-results).\nEvery scored item is defined in [the scoring criteria reference](https://www.massdriver.cloud/blogs/iac-bench-2026-10-scoring-criteria).\n\niac-bench measures how well a language model writes **reusable infrastructure-as-code\nmodules** for OpenTofu (the open-source Terraform fork). A model gets a short request of\nthe kind a platform engineer receives from a colleague, and must deliver a module another\nteam could adopt.\n\nProducing configuration that plans is the minimum. The benchmark measures what a senior reviewer would look for:\n\nThe 2026-10 release covers **module generation only**, on three fixtures, one per major\ncloud.\n\n| Fixture | Cloud and service | Audience named in the brief | Interface budget | \n|---|---|---|---|\n| G001 | AWS · S3 bucket | Application developers who \"fill in a couple of obvious inputs\" | 8 inputs (zero credit at 16) | \n| G002 | GCP · Cloud SQL for PostgreSQL | A DBA team that \"knows PostgreSQL and Cloud SQL deeply\" | 25 inputs (zero credit at 35) | \n| G003 | Azure · App Service (Linux, containers) | An SRE team that \"knows Azure well\" | 20 inputs (zero credit at 30) | \n\nEach fixture is defined by a brief, a rubric, and a reference matrix. All three are\npublished in the repository under `fixtures/generation/` and `testdata/solutions/`.\n\nBriefs are short, realistic requests of two to five sentences. They state what is needed, who will use it, and the one or two facts about the environment that settle the design: for G002, that each environment already has its own VPC with private services access, and that every engineer and workload has a Google identity.\n\nBriefs **do not list security, durability, or hygiene requirements.** Real requests rarely\ndo, and listing them would turn the most valuable signal (does the model know to do these\nthings?) into an instruction-following check. The rubric's criteria are therefore hidden\nfrom the model, but each core criterion is held to a rule: it must follow either from the\nbrief or from practice that is universal for the service. A model is never penalized for\nfailing to guess a preference the brief leaves open. Service-specific touches that go\nbeyond universal practice are scored separately as *expertise* (§4.3).\n\nEach model works as a tool-using agent in an isolated working directory, using the\n[Inspect](https://inspect.aisi.org.uk/) framework. Every model gets the same system prompt\n(`prompts/generate-module.md`), the same tools, and the same limits.\n\n| Setting | Value | \n|---|---|\n| Tools | `list_files` ,`read_file` ,`write_file` ,`tofu_validate` ,`tofu_plan` ,`tofu_fmt` ,`submit` | \n| Deliverable | A root module ( `*.tf` ) plus`minimal.tfvars` , which gives a literal value for every variable without a default and must plan cleanly | \n| Remote modules | Not allowed (calling one fails the evaluation); local submodules allowed | \n| Limits | 150 messages, 30 minutes wall clock per attempt | \n| API retries | Up to 3 per request, then the attempt ends as an error (none occurred) | \n| Planning | `tofu plan` against real cloud accounts (AWS, GCP, Azure sandboxes). Nothing is ever applied. | \n| OpenTofu | 1.12.5 | \n| Providers | Each module's own version constraints, resolved at run time | \n\nThe `tofu_plan` tool plans against the same real cloud accounts used for scoring, so a\nmodel can verify that its module plans before submitting.\n\n| Model | API identifier | Access | Weights | \n|---|---|---|---|\n| Claude Opus 5.5 | `claude-opus-5-5` | Anthropic API | Closed | \n| Claude Fable 5.1 | `claude-fable-5-1` | Anthropic API | Closed | \n| Claude Sonnet 5.5 | `claude-sonnet-5-5` | Anthropic API | Closed | \n| Claude Haiku 5.5 | `claude-haiku-5-5` | Anthropic API | Closed | \n| GPT-6 Astra | `gpt-6-astra` | OpenAI API (Responses) | Closed | \n| GPT-6 Luna | `gpt-6-luna` | OpenAI API (Responses) | Closed | \n| Gemini 3.1 Pro | `gemini-3.1-pro-preview` | Google Gemini API | Closed | \n| Gemini 3.6 Flash | `gemini-3.6-flash` | Google Gemini API | Closed | \n| Kimi K3 | `moonshotai/kimi-k3` | OpenRouter | Open | \n| DeepSeek V4 Pro | `deepseek/deepseek-v4-pro-0813` | OpenRouter | Open | \n\nAll models ran with their providers' default sampling and reasoning settings. No temperature, reasoning-effort, or token-budget overrides were set. The GPT-6 models were called through OpenAI's Responses API, because its Chat Completions API rejects tool use combined with reasoning for these models. It changes only the API used, and no model setting.\n\n**Pinning.** Except for DeepSeek V4 Pro (`-0813`), the providers' APIs returned these\nidentifiers as undated aliases, and none reported a dated snapshot. Results are therefore\ntied to the run date: all generations ran on **2026-10-22**, and all judging on\n**2026-10-23**. Hosted models can change behind an alias without notice.\n\nEach model ran each fixture **three times**, as independent attempts with no shared state:\n10 models × 3 fixtures × 3 samples = **90 attempts**. Every attempt is scored and\npublished. No attempt was re-run, discarded, or chosen after the fact.\n\nEach attempt passes through four layers.\n\n| Layer | What it measures | Nature | Weight | \n|---|---|---|---|\n| 1. Gate | The module plans as a root module with its own `minimal.tfvars` | Mechanical, pass/fail | Prerequisite for layer 2 | \n| 2. Rubric and expertise | Fixture-specific consequences: security, data protection, audience fit, service expertise | Mechanical, itemized and weighted | 70% | \n| 3. Judge | Design and craft against a published quality doctrine | LLM judge, calibrated against a human | 30% | \n| 4. Statistics | Inputs, SLOC, tokens, time, estimated cost | Mechanical, descriptive | 0% (reported only) | \n\n**Composite = 0.70 × mechanical + 0.30 × judge grade**, on a 0–100 scale.\n\nThe runner keeps only the variables that have no default from the model's own\n`minimal.tfvars`, then runs `tofu plan` on the module as a root module. A module that does\nnot plan **fails the gate**:\n\nEach fixture's `rubric.yaml` declares weighted items, checked by a declarative check engine\n(`docs/generation-check-engine.md`). The engine never looks at variable names. It\nevaluates outcomes in two places:\n\n`count`/` for_each`, dynamic blocks and `each.*`.\n**The perturbation sweep.** For every documented input, the runner plans the module once\nmore with that one input changed: booleans flipped, and enumerated values from `contains()`\nvalidations tried (up to five alternatives). The sweep reveals what an input actually\ncontrols without the engine knowing what the model named it. A setting reached through a\ndirect toggle, a value inferred from an environment, or a derived local all score the same.\nCollection-typed and free-string inputs are not perturbed.\n\n**The solver.** When an item fails at defaults and in the sweep, a solver reads the module's\nsource and searches for inputs that would reach the item: up to three inputs changed at\nonce, with values drawn from the module's own comparisons, validations and map keys, plus a\nfew well-formed sample values per fixture (such as a resource ID). Inputs that would fail a\nvalidation or precondition are rejected. The runner then plans each proposed input set, and\nonly a successful plan that shows the item counts. The source proposes and the plan\ndecides.\n\nItems take one of five **postures**, depending on what the item protects:\n\n| Posture | Meaning | Passes when | \n|---|---|---|\n| Hardcoded | Security posture. An input that can weaken it is the defect, even if the default is safe. | Every value the source can produce is secure (a literal, or an input confined to secure values, such as a choice between two encrypted modes), the resource exists unconditionally, and it stays secure in every sweep plan | \n| On by default | Safe by default, with a legitimate reason to turn it off | On at defaults with a real way to turn it off. Always-on with no opt-out earns partial credit (0.75); off by default but reachable earns 0.5. | \n| Offered | A capability the audience should be able to reach | Present at defaults or in at least one sweep plan | \n| Graded | Better and worse mechanisms that both work | Credit for the best mechanism present | \n| Budget | Interface size fits the audience | Full credit at or under the soft count, falling linearly to zero at the hard count | \n\nItems are grouped by role:\n\nTwo H items per fixture are **mandatory**: the most severe security failure modes. Failing\none is a **security finding** (⚠). It does not cap the score, but it marks the attempt\nunshippable.\n\nEach fixture also lists **expertise items**. These are service-specific touches an\nexperienced engineer adds without being asked, such as S3 bucket-owner-enforced ownership\nor IAM database authentication on Cloud SQL. They are checked mechanically and pooled into\nthe mechanical score at weight 1 each, against rubric weights of 2–18. The rubric therefore\ndecides whether a module meets the bar, and expertise separates modules that do.\n\nExpertise items come from observation. Candidates come from differences between models and from the judge's rationales. Each is approved by the doctrine owner and records where it came from. An item must not reward inventing requirements the brief never stated.\n\nmechanical = (Σ rubric weight × credit + Σ expertise credit) / (Σ rubric weight + number of expertise items)\n\nThe rubric weight totals are 86 (G001), 97 (G002) and 72 (G003), including the baseline items. Each fixture has 7–9 expertise items.\n\nEach fixture ships hand-written reference modules that the rubric must classify correctly:\n\n`ref_lean` is opinionated, with few inputs and choices inferred from a concept such as\nenvironment.`ref_flexible` is more configurable.`bad_passthrough`: every setting is an input.` bad_minimal`: a bare resource.` bad_unsafe`: one critical setting is exposed as an input.` bad_open_slot`: a locked-down app with an open deployment slot.\nThe self-test runs this matrix against real cloud plans. If a deliberately good design fails the rubric, the rubric is wrong. The matrix exists so that the rubric rewards outcomes rather than one author's preferred shape.\n\nThe judge is **Claude Opus 5.5**. It grades each module against the benchmark's quality\ndoctrine (`docs/good-opentofu.md`, edition `af3ae1dd9fdc1c0c`) on **14 sections**, scoring\neach 0 (no credit), 1 (partial) or 2 (full):\n\nThe weighted mean is scaled to 0–100.\n\n**The judge grades only what the rubric cannot.** The doctrine sections the rubric already\nscores are given to the judge as context but are not graded: insecure configurations,\ndestructive defaults, unstated assumptions, tagging, versions, and provider declarations.\nA defect therefore never costs a module in both components.\n\n**Blind to authorship.** The judge sees the brief, the doctrine, and the module's files:\nHCL first, then tfvars, then documentation. It never sees which model wrote the module.\nThe files are marked as untrusted data, so instructions inside a module are not followed.\n\n**Sampling.** Each module is graded three times independently. Each section takes the\nmedian of the three grades, and the judge returns one sentence of evidence per grade.\n\nA judge grade counts toward the composite only after the judge is calibrated against the doctrine owner (the human who maintains the doctrine), fixture by fixture:\n\nA second judge (GPT-5.5) graded the same modules during calibration. The doctrine owner reviewed in detail the cells where the two judges disagreed. Agreement for Claude Opus 5.5:\n\n| Fixture | Cells | Exact agreement | Lowest section | \n|---|---|---|---|\n| G001 | 112 | 0.929 | 0.75 (Interface design, Scope, File layout, Documentation) | \n| G002 | 112 | 0.973 | 0.875 (Interface design, File layout, Documentation) | \n| G003 | 112 | 0.982 | 0.75 (Documentation) | \n\nGPT-5.5 failed the gate on G001 (Interface design) and G003 (Documentation, Sensitive values). A single judge is used across all fixtures, so that grades are comparable between fixtures.\n\n**The final modules.** The 2026-10 modules were graded with the same judge model, judge\ncode, doctrine edition and sampling as the calibration. Calibration was carried over on that\nbasis; the doctrine owner did not re-review the final modules. Each grade file records where\nits calibration came from.\n\n`tofu plan`. It can miss a\nsetting that the source omits and the provider defaults insecurely, and it can grade a\nmodule that does not plan. The rubric covers both cases. The two components are meant to\ncomplement each other.\nWithin a fixture the median is used, so one broken attempt out of three is reported as one broken attempt instead of being averaged into a result that hides it.\n\nThree samples per cell is a small sample, so the overall score is reported with a\n**bootstrap 95% confidence interval**:\n\nEach model also gets a **rank interval**: the 2.5th to 97.5th percentile of its rank across\njoint bootstrap draws, following the LMArena convention. A ranking claim is made only where\nintervals separate. Pairwise win probabilities are published in the data release.\n\nWith n = 3, the bootstrap is coarse, and intervals are wide for models with one outlier attempt. Read them as a statement of how far the data supports a ranking, not as precise coverage.\n\nA module that fails the gate keeps its judge grade, so its composite is about 25–30 rather than 0. It is flagged ✗ and counted in the results tables. The judge grade is kept because the judge grades design and craft, which a plan failure does not erase. The flag shows that the attempt was unshippable. Security findings are handled the same way: flagged ⚠ and counted, but no score cap.\n\nEstimated cost per attempt uses list prices from OpenRouter, dated 2026-10-23, for every model, including closed models routed through their own APIs. Cache reads are charged at 0.1× and cache writes at 1.25× the input price. Judge calls are excluded. Token counts, wall-clock time, SLOC and input counts come from the eval logs. None of them affects any score.\n\niac-bench is developed by Massdriver, which builds infrastructure tooling. Massdriver wrote the briefs, rubrics, references and doctrine, and its staff acted as the calibrating human.\n\nThe results were scored by the code tagged\n[`2026-10`](https://github.com/massdriver-cloud/iac-bench/tree/2026-10) in the repository.\nAt that tag, the repository contains everything needed to reproduce or audit a score:\n\n`src/iacbench/`.` prompts/generate-module.md`.` fixtures/` and `testdata/`.` docs/good-opentofu.md`.` composite-scoring-policy.yaml`.` prices.yaml`.\nThe [2026-10 data release](https://github.com/massdriver-cloud/iac-bench/tree/2026-10/results/2026-10) (`results/2026-10/` in the repository) contains:\n\nThe identifiers of the sandbox cloud accounts (AWS account, GCP project and Azure subscription) are replaced with placeholders wherever they appear; the release lists every redacted file. Results are licensed under CC BY 4.0.\n\nTo reproduce one cell:\n\n```\nuv run inspect eval src/iacbench/task.py@generate_module -T fixture=G002 \\\n    --model anthropic/claude-sonnet-5-5 --epochs 3\nuv run iacbench judge --grade --samples 3 --fixture G002 \\\n    --candidates <collected modules> --model anthropic/claude-opus-5-5\n@misc{iacbench2026,\n  title  = {iac-bench: Evaluating LLM-authored OpenTofu modules},\n  author = {{Massdriver}},\n  year   = {2026},\n  month  = {10},\n  note   = {Release 2026-10},\n  url    = {https://github.com/massdriver-cloud/iac-bench}\n}\n```\n\nHow Claude Fable 5.1 did on iac-bench 2026-10: an overall composite of 86.9, scores per fixture, strengths, weaknesses and the patterns in its OpenTofu modules.\n\nHow Claude Haiku 5.5 did on iac-bench 2026-10: an overall composite of 68.0, scores per fixture, strengths, weaknesses and the patterns in its OpenTofu modules.\n\nHow Claude Opus 5.5 did on iac-bench 2026-10: an overall composite of 88.9, scores per fixture, strengths, weaknesses and the patterns in its OpenTofu modules.", "url": "https://wpnews.pro/news/iac-bench-2026-10-methodology", "canonical_source": "https://www.massdriver.cloud/blogs/iac-bench-2026-10-methodology", "published_at": "2026-11-04 00:00:00+00:00", "updated_at": "2026-11-04 22:21:08.688702+00:00", "lang": "en", "topics": ["ai-research", "large-language-models", "ai-agents", "ai-tools"], "entities": ["iac-bench", "OpenTofu", "Inspect", "AWS", "GCP", "Azure", "Terraform"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/iac-bench-2026-10-methodology", "markdown": "https://wpnews.pro/news/iac-bench-2026-10-methodology.md", "text": "https://wpnews.pro/news/iac-bench-2026-10-methodology.txt", "jsonld": "https://wpnews.pro/news/iac-bench-2026-10-methodology.jsonld"}}