{"slug": "how-well-do-llms-write-infrastructure-modules-iac-bench-2026-10-results", "title": "How Well Do LLMs Write Infrastructure Modules? iac-bench 2026-10 Results", "summary": "Anthropic's Claude Opus 5.5 ranked first among 10 language models in the iac-bench 2026-10 release, scoring 88.9 (95% CI 87.5–91.5) on 90 OpenTofu module attempts across AWS, GCP and Azure, with zero failed plans and zero mandatory security misses in 9 attempts. OpenAI's GPT-6 Astra posted the best single-fixture medians, 91.3 on Cloud SQL and 95.9 on App Service, while Gemini 3.6 Flash finished last at 54.3 and failed at least one mandatory security item in 8 of 9 attempts. The benchmark scored modules as 0.70 × mechanical checks plus 0.30 × a calibrated judge, and found the most-missed rubric item was log delivery permitted (H13), missed in 30 of 30 S3 attempts that planned.", "body_md": "Upcoming Workshop: Citizen Developer Agentic Software Factory on AWS 11.4.26 @ 12PM PST\n\n[Register Here!](https://streamyard.com/watch/umMggUFkVMNu)\n\nWe asked ten language models to write reusable OpenTofu modules for AWS, GCP and Azure, then scored 90 modules against real plans and a calibrated judge. Here is how they did and the patterns behind the scores.\n\n**iac-bench release 2026-10 · run 2026-10-22 · 10 models · 3 clouds · 90 attempts**\n\nWe asked ten language models to write reusable OpenTofu modules for three common cloud services intended for a specific audience:\n\nEach request was a few sentences describing the high level use case of the module and the intended audience. We never listed the specific security, durability or hygiene requirements a senior reviewer would expect.\n\nWe scored the 90 resulting modules on two components:\n\n`tofu plan`\noutput.\nThis article reports the results and the patterns behind them:\n\n`H13`).` H11`).` B1`).\n| Rank | Model | Developer | Weights | Overall (95% CI) | Rank interval | G001 · S3 | G002 · Cloud SQL | G003 · App Service | ✗ gate | ⚠ security | Cost, 9 attempts | \n|---|---|---|---|---|---|---|---|---|---|---|---|\n| 1 | [Claude Opus 5.5](https://www.massdriver.cloud/blogs/iac-bench-2026-10-claude-opus-5-5) | Anthropic | Closed | **88.9** (87.5–91.5) | 1–3 | 87.9 | 88.6 | 90.2 | 0/9 | 0/9 | $3.58 | \n| 2 | [Claude Fable 5.1](https://www.massdriver.cloud/blogs/iac-bench-2026-10-claude-fable-5-1) | Anthropic | Closed | **86.9** (83.7–90.2) | 1–5 | 85.6 | 85.6 | 89.5 | 0/9 | 0/9 | $13.97 | \n| 3 | [GPT-6 Astra](https://www.massdriver.cloud/blogs/iac-bench-2026-10-gpt-6-astra) | OpenAI | Closed | **86.7** (86.0–89.8) | 1–5 | 72.8 | 91.3 | 95.9 | 0/9 | 0/9 | $7.26 | \n| 4 | [Claude Sonnet 5.5](https://www.massdriver.cloud/blogs/iac-bench-2026-10-claude-sonnet-5-5) | Anthropic | Closed | **86.1** (84.8–88.2) | 2–5 | 84.8 | 90.6 | 82.8 | 0/9 | 2/9 | $1.52 | \n| 5 | [Kimi K3](https://www.massdriver.cloud/blogs/iac-bench-2026-10-kimi-k3) | Moonshot AI | Open | **82.4** (57.3–84.7) | 5–9 | 81.1 | 79.2 | 86.9 | 1/9 | 4/9 | $3.96 | \n| 6 | [GPT-6 Luna](https://www.massdriver.cloud/blogs/iac-bench-2026-10-gpt-6-luna) | OpenAI | Closed | **81.5** (79.8–82.1) | 5–7 | 74.3 | 84.5 | 85.7 | 0/9 | 0/9 | $0.05 | \n| 7 | [Gemini 3.1 Pro](https://www.massdriver.cloud/blogs/iac-bench-2026-10-gemini-3-1-pro-preview) |  | Closed | **73.0** (69.2–76.2) | 6–9 | 74.2 | 79.3 | 65.4 | 0/9 | 5/9 | $2.26 | \n| 8 | [DeepSeek V4 Pro](https://www.massdriver.cloud/blogs/iac-bench-2026-10-deepseek-v4-pro-0813) | DeepSeek | Open | **69.2** (64.4–79.3) | 6–9 | 82.5 | 62.1 | 63.2 | 0/9 | 5/9 | $0.88 | \n| 9 | [Claude Haiku 5.5](https://www.massdriver.cloud/blogs/iac-bench-2026-10-claude-haiku-5-5) | Anthropic | Closed | **68.0** (66.1–90.6) | 1–9 | 87.8 | 26.8 | 89.4 | 2/9 | 1/9 | $0.11 | \n| 10 | [Gemini 3.6 Flash](https://www.massdriver.cloud/blogs/iac-bench-2026-10-gemini-3-6-flash) |  | Closed | **54.3** (40.0–61.5) | 10–10 | 62.8 | 49.2 | 51.0 | 1/9 | 8/9 | $3.14 | \n\nComposite = 0.70 × mechanical + 0.30 × judge. Fixture columns are medians of three attempts. ✗ = attempts whose module did not plan; ⚠ = attempts failing at least one mandatory security item. Costs are estimates at list prices on 2026-10-23 and exclude judging.\n\n*Figure 1. Overall composite (mean of the three per-fixture medians). Whiskers show the\nbootstrap 95% CI; the right-hand labels give the 95% rank interval.*\n\n**How to read the ranking.** Three samples per model per fixture is enough to separate\ngroups but not neighbors:\n\nClaude Haiku 5.5's interval is the widest in the release. It is driven by one fixture where two of three modules failed to plan. Pairwise win probabilities are in the data release.\n\n*Figure 2. Median composite of three attempts, per model and fixture.*\n\n| Fixture | Field median | Best median | Most-missed rubric item (attempts that planned) | Most-missed mandatory item | \n|---|---|---|---|---|\n| G001 · S3 | 81.8 | 87.9 (Claude Opus 5.5) | `H13` log delivery permitted: 30/30 missed | `H1` public access block: 6/30 | \n| G002 · Cloud SQL | 82.2 | 91.3 (GPT-6 Astra) | `H11` databases dropped on removal: 24/28 | `H2` encrypted connections: 9/28 | \n| G003 · App Service | 86.2 | 95.9 (GPT-6 Astra) | `A7` interface budget: 19/28 | `H2` basic-auth publishing: 10/28 | \n\nFixture scores are not interchangeable. **G001 (S3)** punishes omission: the brief's\naudience is developers who \"don't know S3 well,\" so every hygiene decision falls to the\nmodule. The models that lose the most here:\n\n**G002 (Cloud SQL)** rewards locking down a specialist's interface without taking their\ncontrols away. **G003 (App Service)** depends heavily on two settings the Azure provider\nleaves insecure by default.\n\nSome criteria were missed by nearly every model, regardless of vendor, size or overall rank. These are the clearest signal in the release, because they do not depend on which model you pick.\n\n*Figure 3. Share of attempts earning full credit on the rubric items most often missed,\nby model. Item definitions are in [Scoring criteria](https://www.massdriver.cloud/blogs/iac-bench-2026-10-scoring-criteria).*\n\nThese criteria were missed by most of the field.\n\n**Access logs that cannot be delivered (G001 `H13`, 30/30 missed).**\n\n`H9`).\n**Databases dropped when removed from the input (G002 `H11`, 24/28 missed).**\n\n`google_sql_database` defaults to deleting the database, and its data, when the resource\nis removed.`databases` list therefore drops that database's\ndata on the next apply.`deletion_policy = \"ABANDON\"` or tying it to deletion\nprotection.\n**Backups that cannot be turned off (G002 `H4`/` H5`).** This one splits by rank:\n\n`E1`, 0/28). By default Cloud SQL deletes them with the instance.\n**Provider constraints that admit the next major version (`B1`).**\n\n| Cloud | Attempts with such a constraint | \n|---|---|\n| AWS | 19/30 | \n| GCP | 22/28 | \n| Azure | 2/28 | \n\n`>= 5.0`, or a window such as\n`>= 5.0, < 7.0` that spans two majors.`~> 4.0`.\nThe habit looks provider-specific rather than model-specific: the same models that bound\n`azurerm` correctly left `aws` and `google` open.\n\n**Expertise that only a few models showed:**\n\n`roles/cloudsql.instanceUser`, G002 `E6`): 13/28 granted it for\nthe IAM database users they created. Without it, those users cannot log in.`E8`): 12/28.` E5`): 6/28.\n\n*Figure 4. Attempts with at least one mandatory security failure, and attempts that did\nnot plan, out of 9 per model.*\n\nEach fixture has two **mandatory** items: its most severe security failures. Twenty-five of\n90 attempts failed at least one.\n\n| Pattern | Attempts | What it looks like | Models | \n|---|---|---|---|\n| **Secure default, open override** | 15 | The secure value is the default, but the setting is an input any consumer can change. For example, `ssl_mode` defaults to`ENCRYPTED_ONLY` while the validation also accepts`ALLOW_UNENCRYPTED_AND_ENCRYPTED` . | Gemini 3.6 Flash, Kimi K3, DeepSeek V4 Pro | \n| **Insecure provider default left in place** | 10 | The module never sets the attribute, and the provider's default is insecure. On Azure, basic-auth publishing for FTP and Web Deploy is on unless disabled. | Gemini 3.1 Pro, Claude Sonnet 5.5, Claude Haiku 5.5, plus some attempts by DeepSeek V4 Pro and Gemini 3.6 Flash | \n\n**Secure default, open override.** The doctrine treats this as a defect even when the\ndefault is safe. A module's security posture has to hold for every consumer, and an input\nthat can silently weaken it will eventually be used to do so. Models in this pattern often\ndocument the risk in the same breath: one description reads \"Do not weaken this without a\nsecurity review\" above a validation list that allows exactly that.\n\n**Insecure provider default left in place.** This failure cannot be seen by reading the\nsource, because nothing is written down. The most-missed mandatory item in the release is\nG003 `H2` (basic-auth publishing, 10/28). Several of these modules set HTTPS-only, TLS 1.2\nand FTPS-off correctly in the same block, and still miss the two publishing flags the Azure\nprovider enables by default.\n\n**Failed gates.** Four attempts produced modules that did not plan:\n\n*Figure 5. Mechanical-only and judge-only scores per model (mean of per-fixture medians).*\n\n**The ranking holds without the judge.** Ranked by the mechanical score alone, the ten models\nfall in exactly the composite order. The judge's 30% shifts scores but never reorders the\nleaderboard.\n\n| Model | Mechanical only | Judge only | Composite rank | \n|---|---|---|---|\n| Claude Opus 5.5 | 86.4 (#1) | 94.7 (#1) | 1 | \n| Claude Fable 5.1 | 85.0 (#2) | 91.2 (#5) | 2 | \n| GPT-6 Astra | 83.2 (#3) | 94.7 (#2) | 3 | \n| Claude Sonnet 5.5 | 82.5 (#4) | 94.7 (#3) | 4 | \n| Kimi K3 | 80.6 (#5) | 86.0 (#6) | 5 | \n| GPT-6 Luna | 79.7 (#6) | 84.2 (#7) | 6 | \n| Gemini 3.1 Pro | 71.9 (#7) | 71.9 (#8) | 7 | \n| DeepSeek V4 Pro | 69.5 (#8) | 67.5 (#9) | 8 | \n| Claude Haiku 5.5 | 59.2 (#9) | 91.2 (#4) | 9 | \n| Gemini 3.6 Flash | 50.9 (#10) | 62.3 (#10) | 10 | \n\nThe two components disagree on individual modules, and the disagreements are systematic.\n\n**What the judge misses.** It reads source, so it cannot see:\n\n**What the rubric misses.** It checks planned outcomes, so it cannot see:\n\nThe judge scored *Documentation* 0 in 26 of 90 attempts. It found, for example, a databases\ninput that fails as soon as one database is declared, which the rubric credited.\n\n**Disagreement about restraint.** The largest single gap is GPT-6 Astra on S3:\n\nA reader who rejects the rubric's view can read the judge-only column, and the reverse.\n\n**Self-preference.** The judge is Claude Opus 5.5. Claude models take three of the top four\njudge-only ranks, alongside GPT-6 Astra. But the composite order, and Claude Opus 5.5's\nfirst place, are identical under the mechanical score alone, where no model judges any\nother.\n\n*Figure 6. Overall composite against the estimated cost of one attempt at each fixture, at\nlist prices on 2026-10-23. The dashed line is the Pareto frontier: models no other model\nbeats on both cost and score.*\n\nGeneration cost varies by a factor of about 280 across the field, and it is only loosely related to score.\n\n| Model | Overall | Estimated cost, 9 attempts | \n|---|---|---|\n| GPT-6 Luna | 81.5 | $0.05 | \n| Claude Haiku 5.5 | 68.0 | $0.11 | \n| DeepSeek V4 Pro | 69.2 | $0.88 | \n| Claude Sonnet 5.5 | 86.1 | $1.52 | \n| Gemini 3.1 Pro | 73.0 | $2.26 | \n| Gemini 3.6 Flash | 54.3 | $3.14 | \n| Claude Opus 5.5 | 88.9 | $3.58 | \n| Kimi K3 | 82.4 | $3.96 | \n| GPT-6 Astra | 86.7 | $7.26 | \n| Claude Fable 5.1 | 86.9 | $13.97 | \n\nThe Pareto frontier, the models no other model beats on both cost and score, runs through:\n\nClaude Fable 5.1 costs about four times as much as Claude Opus 5.5 for a slightly lower score. Claude Haiku 5.5 is the second-cheapest model and would sit near the frontier, but for its two failed Cloud SQL attempts.\n\nCosts are estimates: token counts multiplied by list prices. They exclude judging and any provider-specific caching discounts beyond a flat cache-read and cache-write multiplier.\n\n*Documentation* is the judged section most often scored 0: in 26 of 90 attempts. Nearly every\nmodule has documentation. The zeros come from **documentation that is wrong**:\n\nUnder the doctrine, any wrong statement scores 0, however small.\n\n| Pattern | Examples | \n|---|---|\n| Limits misstated | 1–35 days of WAL retention described for an edition that allows 1–7 | \n| Inverted meaning | A deletion-protection input described as \"whether or not to allow Terraform to destroy the instance\" | \n| Features that do not exist | \"Least-requests load balancing\" that is never configured; a staging-traffic input that feeds only an output | \n\n**Longer documentation produces more of these errors.** The models that write long READMEs\n(Claude Fable 5.1, GPT-6 Astra, Kimi K3) give the judge more claims to check:\n\n**The same model, prompt and fixture can produce very different modules.** Nine of the 30\nmodel-fixture cells spread by more than 10 points across their three attempts. Four spread\nby more than 20:\n\n| Model · fixture | Range | Cause | \n|---|---|---|\n| Claude Haiku 5.5 · Cloud SQL | 25.3–91.0 | Two attempts did not plan | \n| Kimi K3 · App Service | 25.3–89.4 | One attempt did not plan | \n| Gemini 3.6 Flash · App Service | 15.0–62.4 | One attempt did not plan | \n| DeepSeek V4 Pro · App Service | 57.8–81.8 | Basic-auth flags set in one attempt, absent in two | \n\nTwo consequences:\n\nEvery model in the top group has all three attempts within 11 points on every fixture. Claude Opus 5.5's widest spread is 8.5 points.\n\nThe main limitations of this release:\n\nFull discussion: [Methodology §7–8](https://www.massdriver.cloud/blogs/iac-bench-2026-10-methodology#7-limitations).\n\nThe [data release](https://github.com/massdriver-cloud/iac-bench/tree/2026-10/results/2026-10) (`results/2026-10/` in the repository) contains:\n\nIt is licensed CC BY 4.0. The evaluation code, fixtures, rubrics, reference modules and doctrine are in the repository.\n\n```\n@misc{iacbench2026,\n  title  = {iac-bench: Evaluating LLM-authored OpenTofu modules},\n  author = {{Massdriver}},\n  year   = {2026},\n  month  = {10},\n  note   = {Release 2026-10},\n  url    = {https://github.com/massdriver-cloud/iac-bench}\n}\n```\n\nHow Claude Fable 5.1 did on iac-bench 2026-10: an overall composite of 86.9, scores per fixture, strengths, weaknesses and the patterns in its OpenTofu modules.\n\nHow Claude Haiku 5.5 did on iac-bench 2026-10: an overall composite of 68.0, scores per fixture, strengths, weaknesses and the patterns in its OpenTofu modules.\n\nHow Claude Opus 5.5 did on iac-bench 2026-10: an overall composite of 88.9, scores per fixture, strengths, weaknesses and the patterns in its OpenTofu modules.", "url": "https://wpnews.pro/news/how-well-do-llms-write-infrastructure-modules-iac-bench-2026-10-results", "canonical_source": "https://www.massdriver.cloud/blogs/iac-bench-2026-10-results", "published_at": "2026-11-04 00:00:00+00:00", "updated_at": "2026-11-04 20:21:49.341450+00:00", "lang": "en", "topics": ["large-language-models", "ai-research", "ai-tools", "ai-infrastructure"], "entities": ["Anthropic", "Claude Opus 5.5", "OpenAI", "GPT-6 Astra", "Moonshot AI", "Kimi K3", "DeepSeek", "OpenTofu"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/how-well-do-llms-write-infrastructure-modules-iac-bench-2026-10-results", "markdown": "https://wpnews.pro/news/how-well-do-llms-write-infrastructure-modules-iac-bench-2026-10-results.md", "text": "https://wpnews.pro/news/how-well-do-llms-write-infrastructure-modules-iac-bench-2026-10-results.txt", "jsonld": "https://wpnews.pro/news/how-well-do-llms-write-infrastructure-modules-iac-bench-2026-10-results.jsonld"}}