New: LLM Rankings are live!
See how the top LLMs rank on real infrastructure tasks
How Qwen3.8 2.4T-A95B did on iac-bench 2026-10: an overall composite of 81.6, scores per fixture, strengths, weaknesses and the patterns in its OpenTofu modules.
Every model evaluated on the 2026-10 fixtures side by side, with a sortable leaderboard, picks by use case and the cost of each: LLM rankings.
| Developer | Alibaba |
| API identifier | qwen/qwen3.8-2.4t-a95b |
| Access | OpenRouter, pinned to Alibaba's endpoint |
| Weights | Open |
| List price (USD / M tokens, in / out) | 2.00 / 6.00 (OpenRouter, 2026-11-10) |
| Evaluated | 2026-11-10 (generation and judge) |
| Attempts | 9 (3 fixtures × 3 samples) |
| Overall composite | 81.6 (95% CI 71.7–86.2) |
| Rank | 11 of 16 (95% rank interval 7–13) |
| Mechanical only / judge only | 78.6 (rank 12) / 86.8 (rank 11) |
| Failed gates · insecure attempts | 0/9 · 4/9 |
| Estimated cost, all 9 attempts | $4.41 |
Qwen3.8 2.4T-A95B places eleventh of 16 by composite (81.6), between GLM-5.3 (82.5) and GPT-6 Luna (81.5), with a 95% rank interval of 7–13. Against its neighbours the bootstrap is close to a coin flip: it finishes ahead of Kimi K3 in 48% of resamples, GPT-6 Luna in 45% and GLM-5.3 in 43%, and ahead of Kimi K3.1 Pro in 2%. All 9 attempts cost $4.40, against $3.52 for Kimi K3.1 Pro, $3.96 for Kimi K3, $8.46 for GLM-5.3 and $0.88 for DeepSeek V4 Pro.
All 9 attempts planned. 4 of 9 carry a security finding: one on S3 and three on App Service. Cloud SQL (G002) is its strongest fixture, fifth in the field by median composite (89.0) and the highest of the open-weight models. App Service (G003) is its weakest, thirteenth (71.7).
| Fixture | Composite median (range) | Rubric median | Expertise | Judge median | ✗ gate | ⚠ security | Median inputs | Median output tokens | Median time | Median cost |
|---|---|---|---|---|---|---|---|---|---|---|
| G001 · S3 | 84.1 (66.8–91.0) | 0.791 | 6/7 | 94.7 | 0/3 | 1/3 ( H1 ) |
9 | 27,441 | 766 s | $0.326 |
| G002 · Cloud SQL | 89.0 (85.3–89.6) | 0.897 | 5/9 | 89.5 | 0/3 | 0/3 | 20 | 37,210 | 1055 s | $0.434 |
| G003 · App Service | 71.7 (59.4–78.7) | 0.706 | 3/5 | 76.3 | 0/3 | 3/3 ( H1 ,H2 ) |
29 | 42,862 | 1383 s | $0.523 |
Rubric items not at full credit, counted over attempts that planned: G001: A5 2/3, B1 3/3, H1 1/3, H9 2/3, H12 3/3, H13 3/3, H15 2/3; G002: A4 1/3, B1 2/3, H4 3/3, H5 3/3, H11 2/3; G003: A6 1/3, A7 3/3, H1 2/3, H2 2/3, H3 1/3, H4 1/3. Definitions are in Scoring criteria.
ipv4_enabled = false and
ssl_mode = "ENCRYPTED_ONLY" are literals, where Kimi K3 lets ssl_mode be weakened in
3/3. The private network is derived from environment (apps-<environment>). cloudsql.iam_authentication, log_connections and log_disconnections over the DBA's
flags, rejects any input that sets them, and refuses to disable deletion protection in
prod. H9) appear in 3/3. H1 on S3 (e1); on App Service H2 (e1), google_sql_user names that PostgreSQL rejects until the .gserviceaccount.com suffix
is trimmed (3/3). The judge caught it; the rubric did not.A7
partial or zero, 3/3). The 41-input attempt also exposes http2_enabled (A6). H13, 3/3); logging ( H15) each appear in 1/3; H4/ H5 at 0.75, 3/3), and
backups never outlive the instance (E1, 0/3);deletion_policy in either attempt that declares databases (H11). B1): >= 5.0 or
>= 6.0 on S3 (3/3), and >= 5.30.0, < 7.0.0 spanning two majors on Cloud SQL (2/3).
Wide spread within a fixture. Mechanical scores run 0.57–0.87 on S3 and 0.53–0.80 on
App Service, against a spread of 0.05 on Cloud SQL, all from modules that plan. Each
sample hardcodes a different subset of the security posture and exposes the rest:
allow_public_access, which turns off all four public-access-block settings
(https_only and the managed
identity as inputs (minimum_tls_version and ftps_state with validations that accept "1.0" and
"AllAllowed" (H3).
Kimi K3.1 Pro varies by at most 0.075 on any fixture. Kimi K3 and GLM-5.3 have wider App Service spreads, but theirs come from a failed plan.
Long transcripts. Median 37,210 output tokens per attempt, second in the field after GLM-5.3. The longest (App Service e2: 68,729 tokens, $1.02, 41 inputs) is also its lowest-scoring attempt (59.4).
G003/qwen3.8-2.4t-a95b-e3/main.tf lines 48–67
https_only = var.https_only
public_network_access_enabled = var.public_network_access_enabled
virtual_network_subnet_id = var.virtual_network_subnet_id
client_affinity_enabled = false
ftp_publish_basic_authentication_enabled = false
webdeploy_publish_basic_authentication_enabled = false
dynamic "identity" {
for_each = var.managed_identity_enabled ? ["SystemAssigned"] : []
content {
type = identity.value
}
}
One App Service resource, two postures. Basic-auth publishing is hardcoded off with a comment explaining why, while HTTPS-only and the managed identity are left to inputs (https_only, managed_identity_enabled). Both default to the secure setting, but either can be switched off, which costs this attempt H1 and H4. Each App Service sample draws this line in a different place.
How Claude Fable 5.1 did on iac-bench 2026-10: an overall composite of 86.9, scores per fixture, strengths, weaknesses and the patterns in its OpenTofu modules.
How Claude Fable 5.2 did on iac-bench 2026-10: an overall composite of 93.9, scores per fixture, strengths, weaknesses and the patterns in its OpenTofu modules.
How Claude Haiku 5.5 did on iac-bench 2026-10: an overall composite of 68.0, scores per fixture, strengths, weaknesses and the patterns in its OpenTofu modules.