New: LLM Rankings are live!
See how the top LLMs rank on real infrastructure tasks
How GPT-6.1 Sol did on iac-bench 2026-10: an overall composite of 87.2, scores per fixture, strengths, weaknesses and the patterns in its OpenTofu modules.
Every model evaluated on the 2026-10 fixtures side by side, with a sortable leaderboard, picks by use case and the cost of each: LLM rankings.
| Developer | OpenAI |
| API identifier | gpt-6.1-sol |
| Access | OpenAI API (Responses) |
| Weights | Closed |
| List price (USD / M tokens, in / out) | 2.00 / 10.00 (OpenRouter, 2026-11-10) |
| Evaluated | 2026-11-10 (generation and judge) |
| Attempts | 9 (3 fixtures × 3 samples) |
| Overall composite | 87.2 (95% CI 84.9–92.3) |
| Rank | 3 of 16 (95% rank interval 2–8) |
| Mechanical only / judge only | 83.9 (rank 5) / 94.7 (rank 3) |
| Failed gates · insecure attempts | 0/9 · 0/9 |
| Estimated cost, all 9 attempts | $1.02 |
GPT-6.1 Sol places third of 16 by composite (87.2) and sits in the second group with Claude Opus 5.5, Claude Fable 5.1, GPT-6 Astra, Kimi K3.1 Pro and Claude Sonnet 5.5, with a 95% rank interval of 2–8. In the bootstrap it finishes ahead of Astra in 49% of resamples, ahead of Fable 5.1 in 54%, ahead of Sonnet in 71% and ahead of Kimi K3.1 Pro in 67%. It is the cheapest model in that group: $1.02 for all 9 attempts, against $1.52 for Sonnet and $7.26 for Astra.
All 9 attempts planned, with no security findings. App Service (G003) is its strongest fixture, third in the field by median composite. S3 (G001) is eleventh.
| Fixture | Composite median (range) | Rubric median | Expertise | Judge median | ✗ gate | ⚠ security | Median inputs | Median output tokens | Median time | Median cost |
|---|---|---|---|---|---|---|---|---|---|---|
| G001 · S3 | 78.1 (72.8–91.6) | 0.721 | 4/7 | 94.7 | 0/3 | 0/3 | 6 | 4,434 | 129 s | $0.066 |
| G002 · Cloud SQL | 89.3 (87.7–89.3) | 0.875 | 5/9 | 100.0 | 0/3 | 0/3 | 17 | 7,559 | 165 s | $0.110 |
| G003 · App Service | 94.1 (94.1–95.9) | 0.972 | 4/5 | 89.5 | 0/3 | 0/3 | 17 | 10,013 | 379 s | $0.168 |
Rubric items not at full credit, counted over attempts that planned: G001: A5 1/3, B1 2/3, H6 2/3, H7 2/3, H9 2/3, H12 3/3, H13 3/3, H14 1/3, H15 1/3; G002: B1 3/3, H4 3/3, H5 3/3, H11 3/3; G003: B2 2/3. Definitions are in Scoring criteria.
connector_enforcement = "REQUIRED" and validation that rejects overrides of IAM
authentication. On App Service, both basic-auth publishing flags are off on the app and
the slot; Claude Sonnet 5.5 leaves them on in 2/3 (H2). cloudsql.client and cloudsql.instanceUser (E6), the grant GPT-6 Luna omits.AcrPull goes to the identities the app and
the staging slot pull with, where GPT-6 Luna uses registry passwords.A7), which Sonnet exceeds in 3/3. H13, 3/3); H9 lost 2/3);H6, H7 lost 2/3). The
other two expire only incomplete multipart uploads.H11, 3/3). google_sql_database sets
deletion_policy. Backups and point-in-time recovery cannot be turned off ( H4/ H5 at
0.75 in 3/3).B1), including
>= 6.0, < 8.0 in all three Cloud SQL modules.cors_rules mirrors the provider's cors_rule block. registry_url that can disagree. iam_principals passes IAM_USER-style values to
google_sql_user.type, which expects CLOUD_IAM_USER. The judge caught it ( capacity object, and the S3 e3 attempt enables
access logging when an access_logging object is supplied. The rubric credits both
(WEBSITE_SWAP_WARMUP_PING_PATH
at the health-check path, so a swap waits for the staging slot to respond.
G001/gpt-6.1-sol-e1/main.tf lines 63–75
Statement = [{
Sid = "DenyInsecureTransport"
Effect = "Deny"
Principal = "*"
Action = "s3:*"
Resource = [aws_s3_bucket.assets.arn, "${aws_s3_bucket.assets.arn}/*"]
Condition = {
Bool = {
"aws:SecureTransport" = "false"
"aws:PrincipalIsAWSService" = "false"
}
}
}]
The TLS-deny statement carries a second condition that exempts AWS service principals, with the comment "Exempt direct AWS service principals as their network context can be redacted." It appears in all three S3 attempts and in no other model's modules.
How Claude Fable 5.1 did on iac-bench 2026-10: an overall composite of 86.9, scores per fixture, strengths, weaknesses and the patterns in its OpenTofu modules.
How Claude Fable 5.2 did on iac-bench 2026-10: an overall composite of 93.9, scores per fixture, strengths, weaknesses and the patterns in its OpenTofu modules.
How Claude Haiku 5.5 did on iac-bench 2026-10: an overall composite of 68.0, scores per fixture, strengths, weaknesses and the patterns in its OpenTofu modules.