{"slug": "qwen3-8-2-4t-a95b-on-iac-bench-2026-10", "title": "Qwen3.8 2.4T-A95B on iac-bench 2026-10", "summary": "Alibaba's Qwen3.8 2.4T-A95B, an open-weight model accessed via OpenRouter and priced at $2.00/$6.00 per million input/output tokens, scored an overall composite of 81.6 on the iac-bench 2026-10 infrastructure-as-code evaluation, ranking 11th of 16 models with a 95% rank interval of 7–13, according to Massdriver's LLM rankings. The model planned all 9 attempts (3 fixtures × 3 samples) at an estimated total cost of $4.41, but 4 of 9 attempts carried a security finding — one on the S3 fixture and three on App Service — and its App Service result (71.7, 13th) was its weakest, versus Cloud SQL (89.0, fifth) as its strongest and the best of the open-weight models.", "body_md": "New: LLM Rankings are live!\n\n[See how the top LLMs rank on real infrastructure tasks](https://www.massdriver.cloud/llm-rankings)\n\nHow Qwen3.8 2.4T-A95B did on iac-bench 2026-10: an overall composite of 81.6, scores per fixture, strengths, weaknesses and the patterns in its OpenTofu modules.\n\nEvery model evaluated on the 2026-10 fixtures side by side, with a sortable leaderboard, picks by use case and the cost of each: [LLM rankings](https://www.massdriver.cloud/llm-rankings).\n\n| Developer | Alibaba | \n| API identifier | `qwen/qwen3.8-2.4t-a95b` | \n| Access | OpenRouter, pinned to Alibaba's endpoint | \n| Weights | Open | \n| List price (USD / M tokens, in / out) | 2.00 / 6.00 (OpenRouter, 2026-11-10) | \n| Evaluated | 2026-11-10 (generation and judge) | \n| Attempts | 9 (3 fixtures × 3 samples) | \n| **Overall composite** | **81.6** (95% CI 71.7–86.2) | \n| Rank | 11 of 16 (95% rank interval 7–13) | \n| Mechanical only / judge only | 78.6 (rank 12) / 86.8 (rank 11) | \n| Failed gates · insecure attempts | 0/9 · 4/9 | \n| Estimated cost, all 9 attempts | $4.41 | \n\nQwen3.8 2.4T-A95B places eleventh of 16 by composite (81.6), between GLM-5.3 (82.5) and GPT-6 Luna (81.5), with a 95% rank interval of 7–13. Against its neighbours the bootstrap is close to a coin flip: it finishes ahead of Kimi K3 in 48% of resamples, GPT-6 Luna in 45% and GLM-5.3 in 43%, and ahead of Kimi K3.1 Pro in 2%. All 9 attempts cost $4.40, against $3.52 for Kimi K3.1 Pro, $3.96 for Kimi K3, $8.46 for GLM-5.3 and $0.88 for DeepSeek V4 Pro.\n\nAll 9 attempts planned. 4 of 9 carry a security finding: one on S3 and three on App Service. Cloud SQL (G002) is its strongest fixture, fifth in the field by median composite (89.0) and the highest of the open-weight models. App Service (G003) is its weakest, thirteenth (71.7).\n\n| Fixture | Composite median (range) | Rubric median | Expertise | Judge median | ✗ gate | ⚠ security | Median inputs | Median output tokens | Median time | Median cost | \n|---|---|---|---|---|---|---|---|---|---|---|\n| G001 · S3 | 84.1 (66.8–91.0) | 0.791 | 6/7 | 94.7 | 0/3 | 1/3 ( `H1` ) | 9 | 27,441 | 766 s | $0.326 | \n| G002 · Cloud SQL | 89.0 (85.3–89.6) | 0.897 | 5/9 | 89.5 | 0/3 | 0/3 | 20 | 37,210 | 1055 s | $0.434 | \n| G003 · App Service | 71.7 (59.4–78.7) | 0.706 | 3/5 | 76.3 | 0/3 | 3/3 ( `H1` ,`H2` ) | 29 | 42,862 | 1383 s | $0.523 | \n\nRubric items not at full credit, counted over attempts that planned: G001: `A5` 2/3, `B1` 3/3, `H1` 1/3, `H9` 2/3, `H12` 3/3, `H13` 3/3, `H15` 2/3; G002: `A4` 1/3, `B1` 2/3, `H4` 3/3, `H5` 3/3, `H11` 2/3; G003: `A6` 1/3, `A7` 3/3, `H1` 2/3, `H2` 2/3, `H3` 1/3, `H4` 1/3. Definitions are in [Scoring criteria](https://www.massdriver.cloud/blogs/iac-bench-2026-10-scoring-criteria).\n\n`ipv4_enabled = false` and\n`ssl_mode = \"ENCRYPTED_ONLY\"` are literals, where Kimi K3 lets `ssl_mode` be weakened in\n3/3. The private network is derived from `environment` (`apps-<environment>`).` cloudsql.iam_authentication`, `log_connections` and `log_disconnections` over the DBA's\nflags, rejects any input that sets them, and refuses to disable deletion protection in\n`prod`.` H9`) appear in 3/3.` H1` on S3 (e1); on App Service `H2` (e1), `google_sql_user` names that PostgreSQL rejects until the `.gserviceaccount.com` suffix\nis trimmed (3/3). The judge caught it; the rubric did not.`A7`\npartial or zero, 3/3). The 41-input attempt also exposes `http2_enabled` (`A6`).` H13`, 3/3); logging (` H15`) each appear in 1/3;` H4`/` H5` at 0.75, 3/3), and\nbackups never outlive the instance (E1, 0/3);`deletion_policy` in either attempt that declares databases (`H11`).` B1`): `>= 5.0` or\n`>= 6.0` on S3 (3/3), and `>= 5.30.0, < 7.0.0` spanning two majors on Cloud SQL (2/3).\n**Wide spread within a fixture.** Mechanical scores run 0.57–0.87 on S3 and 0.53–0.80 on\nApp Service, against a spread of 0.05 on Cloud SQL, all from modules that plan. Each\nsample hardcodes a different subset of the security posture and exposes the rest:\n\n`allow_public_access`, which turns off all four public-access-block settings\n(`https_only` and the managed\nidentity as inputs (`minimum_tls_version` and `ftps_state` with validations that accept `\"1.0\"` and\n`\"AllAllowed\"` (`H3`).\nKimi K3.1 Pro varies by at most 0.075 on any fixture. Kimi K3 and GLM-5.3 have wider App Service spreads, but theirs come from a failed plan.\n\n**Long transcripts.** Median 37,210 output tokens per attempt, second in the field after\nGLM-5.3. The longest (App Service e2: 68,729 tokens, $1.02, 41 inputs) is also its\nlowest-scoring attempt (59.4).\n\n`G003/qwen3.8-2.4t-a95b-e3/main.tf` lines 48–67\n\n```\n  https_only                    = var.https_only\n  public_network_access_enabled = var.public_network_access_enabled\n  virtual_network_subnet_id     = var.virtual_network_subnet_id\n\n  # HTTP APIs are stateless; session affinity only pins users to instances\n  # that may be recycled, so keep it off.\n  client_affinity_enabled = false\n\n  # Images arrive from a container registry, never over password-based\n  # publishing endpoints.\n  ftp_publish_basic_authentication_enabled       = false\n  webdeploy_publish_basic_authentication_enabled = false\n\n  dynamic \"identity\" {\n    for_each = var.managed_identity_enabled ? [\"SystemAssigned\"] : []\n\n    content {\n      type = identity.value\n    }\n  }\n```\n\nOne App Service resource, two postures. Basic-auth publishing is hardcoded off with a comment explaining why, while HTTPS-only and the managed identity are left to inputs (`https_only`, `managed_identity_enabled`). Both default to the secure setting, but either can be switched off, which costs this attempt `H1` and `H4`. Each App Service sample draws this line in a different place.\n\nHow Claude Fable 5.1 did on iac-bench 2026-10: an overall composite of 86.9, scores per fixture, strengths, weaknesses and the patterns in its OpenTofu modules.\n\nHow Claude Fable 5.2 did on iac-bench 2026-10: an overall composite of 93.9, scores per fixture, strengths, weaknesses and the patterns in its OpenTofu modules.\n\nHow Claude Haiku 5.5 did on iac-bench 2026-10: an overall composite of 68.0, scores per fixture, strengths, weaknesses and the patterns in its OpenTofu modules.", "url": "https://wpnews.pro/news/qwen3-8-2-4t-a95b-on-iac-bench-2026-10", "canonical_source": "https://www.massdriver.cloud/blogs/iac-bench-2026-10-qwen3-8-2-4t-a95b", "published_at": "2026-11-04 00:00:00+00:00", "updated_at": "2026-11-11 17:24:56.115868+00:00", "lang": "en", "topics": ["ai-research", "large-language-models", "ai-tools", "mlops"], "entities": ["Qwen3.8 2.4T-A95B", "Alibaba", "Massdriver", "OpenRouter", "iac-bench 2026-10", "GLM-5.3", "GPT-6 Luna", "Kimi K3"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/qwen3-8-2-4t-a95b-on-iac-bench-2026-10", "markdown": "https://wpnews.pro/news/qwen3-8-2-4t-a95b-on-iac-bench-2026-10.md", "text": "https://wpnews.pro/news/qwen3-8-2-4t-a95b-on-iac-bench-2026-10.txt", "jsonld": "https://wpnews.pro/news/qwen3-8-2-4t-a95b-on-iac-bench-2026-10.jsonld"}}