Qwen3.8 2.4T-A95B on iac-bench 2026-10

Alibaba's Qwen3.8 2.4T-A95B, an open-weight model accessed via OpenRouter and priced at $2.00/$6.00 per million input/output tokens, scored an overall composite of 81.6 on the iac-bench 2026-10 infrastructure-as-code evaluation, ranking 11th of 16 models with a 95% rank interval of 7–13, according to Massdriver's LLM rankings. The model planned all 9 attempts (3 fixtures × 3 samples) at an estimated total cost of $4.41, but 4 of 9 attempts carried a security finding — one on the S3 fixture and three on App Service — and its App Service result (71.7, 13th) was its weakest, versus Cloud SQL (89.0, fifth) as its strongest and the best of the open-weight models.

New: LLM Rankings are live See how the top LLMs rank on real infrastructure tasks https://www.massdriver.cloud/llm-rankings How Qwen3.8 2.4T-A95B did on iac-bench 2026-10: an overall composite of 81.6, scores per fixture, strengths, weaknesses and the patterns in its OpenTofu modules. Every model evaluated on the 2026-10 fixtures side by side, with a sortable leaderboard, picks by use case and the cost of each: LLM rankings https://www.massdriver.cloud/llm-rankings . | Developer | Alibaba | | API identifier | qwen/qwen3.8-2.4t-a95b | | Access | OpenRouter, pinned to Alibaba's endpoint | | Weights | Open | | List price USD / M tokens, in / out | 2.00 / 6.00 OpenRouter, 2026-11-10 | | Evaluated | 2026-11-10 generation and judge | | Attempts | 9 3 fixtures × 3 samples | | Overall composite | 81.6 95% CI 71.7–86.2 | | Rank | 11 of 16 95% rank interval 7–13 | | Mechanical only / judge only | 78.6 rank 12 / 86.8 rank 11 | | Failed gates · insecure attempts | 0/9 · 4/9 | | Estimated cost, all 9 attempts | $4.41 | Qwen3.8 2.4T-A95B places eleventh of 16 by composite 81.6 , between GLM-5.3 82.5 and GPT-6 Luna 81.5 , with a 95% rank interval of 7–13. Against its neighbours the bootstrap is close to a coin flip: it finishes ahead of Kimi K3 in 48% of resamples, GPT-6 Luna in 45% and GLM-5.3 in 43%, and ahead of Kimi K3.1 Pro in 2%. All 9 attempts cost $4.40, against $3.52 for Kimi K3.1 Pro, $3.96 for Kimi K3, $8.46 for GLM-5.3 and $0.88 for DeepSeek V4 Pro. All 9 attempts planned. 4 of 9 carry a security finding: one on S3 and three on App Service. Cloud SQL G002 is its strongest fixture, fifth in the field by median composite 89.0 and the highest of the open-weight models. App Service G003 is its weakest, thirteenth 71.7 . | Fixture | Composite median range | Rubric median | Expertise | Judge median | ✗ gate | ⚠ security | Median inputs | Median output tokens | Median time | Median cost | |---|---|---|---|---|---|---|---|---|---|---| | G001 · S3 | 84.1 66.8–91.0 | 0.791 | 6/7 | 94.7 | 0/3 | 1/3 H1 | 9 | 27,441 | 766 s | $0.326 | | G002 · Cloud SQL | 89.0 85.3–89.6 | 0.897 | 5/9 | 89.5 | 0/3 | 0/3 | 20 | 37,210 | 1055 s | $0.434 | | G003 · App Service | 71.7 59.4–78.7 | 0.706 | 3/5 | 76.3 | 0/3 | 3/3 H1 , H2 | 29 | 42,862 | 1383 s | $0.523 | Rubric items not at full credit, counted over attempts that planned: G001: A5 2/3, B1 3/3, H1 1/3, H9 2/3, H12 3/3, H13 3/3, H15 2/3; G002: A4 1/3, B1 2/3, H4 3/3, H5 3/3, H11 2/3; G003: A6 1/3, A7 3/3, H1 2/3, H2 2/3, H3 1/3, H4 1/3. Definitions are in Scoring criteria https://www.massdriver.cloud/blogs/iac-bench-2026-10-scoring-criteria . ipv4 enabled = false and ssl mode = "ENCRYPTED ONLY" are literals, where Kimi K3 lets ssl mode be weakened in 3/3. The private network is derived from environment apps-<environment . cloudsql.iam authentication , log connections and log disconnections over the DBA's flags, rejects any input that sets them, and refuses to disable deletion protection in prod . H9 appear in 3/3. H1 on S3 e1 ; on App Service H2 e1 , google sql user names that PostgreSQL rejects until the .gserviceaccount.com suffix is trimmed 3/3 . The judge caught it; the rubric did not. A7 partial or zero, 3/3 . The 41-input attempt also exposes http2 enabled A6 . H13 , 3/3 ; logging H15 each appear in 1/3; H4 / H5 at 0.75, 3/3 , and backups never outlive the instance E1, 0/3 ; deletion policy in either attempt that declares databases H11 . B1 : = 5.0 or = 6.0 on S3 3/3 , and = 5.30.0, < 7.0.0 spanning two majors on Cloud SQL 2/3 . Wide spread within a fixture. Mechanical scores run 0.57–0.87 on S3 and 0.53–0.80 on App Service, against a spread of 0.05 on Cloud SQL, all from modules that plan. Each sample hardcodes a different subset of the security posture and exposes the rest: allow public access , which turns off all four public-access-block settings https only and the managed identity as inputs minimum tls version and ftps state with validations that accept "1.0" and "AllAllowed" H3 . Kimi K3.1 Pro varies by at most 0.075 on any fixture. Kimi K3 and GLM-5.3 have wider App Service spreads, but theirs come from a failed plan. Long transcripts. Median 37,210 output tokens per attempt, second in the field after GLM-5.3. The longest App Service e2: 68,729 tokens, $1.02, 41 inputs is also its lowest-scoring attempt 59.4 . G003/qwen3.8-2.4t-a95b-e3/main.tf lines 48–67 https only = var.https only public network access enabled = var.public network access enabled virtual network subnet id = var.virtual network subnet id HTTP APIs are stateless; session affinity only pins users to instances that may be recycled, so keep it off. client affinity enabled = false Images arrive from a container registry, never over password-based publishing endpoints. ftp publish basic authentication enabled = false webdeploy publish basic authentication enabled = false dynamic "identity" { for each = var.managed identity enabled ? "SystemAssigned" : content { type = identity.value } } One App Service resource, two postures. Basic-auth publishing is hardcoded off with a comment explaining why, while HTTPS-only and the managed identity are left to inputs https only , managed identity enabled . Both default to the secure setting, but either can be switched off, which costs this attempt H1 and H4 . Each App Service sample draws this line in a different place. How Claude Fable 5.1 did on iac-bench 2026-10: an overall composite of 86.9, scores per fixture, strengths, weaknesses and the patterns in its OpenTofu modules. How Claude Fable 5.2 did on iac-bench 2026-10: an overall composite of 93.9, scores per fixture, strengths, weaknesses and the patterns in its OpenTofu modules. How Claude Haiku 5.5 did on iac-bench 2026-10: an overall composite of 68.0, scores per fixture, strengths, weaknesses and the patterns in its OpenTofu modules.