# ChatGPT vs Claude vs Gemini Test: 12 Steps, 90 Min [2026]
> Source:
> Published: 2026-10-28 17:34:44+00:00
Three vendors moved their flagship API models within a few weeks of each other. OpenAI put GPT-6.1 Terra into the API on October 26, 2026 at $1 per million input tokens and $5 per million output tokens. Google made Gemini 4 Argon generally available on October 13. Anthropic’s Claude 5.5 family (Opus, Sonnet and Haiku) is now its current line, as the [October plan and price roundup](https://askgpu.com/gemini-vs-chatgpt-vs-claude/) at AskGPU lays out. Any ranking you read last month is stale.
Searches for chatgpt vs claude vs gemini mostly return taste tests and subscription roundups. Those tell you which chat app feels nicer. They say nothing about which model passes your prompts through an API, how slow the 95th percentile gets, or what a thousand calls cost. The gap between list prices alone is already wide, as the [Claude vs ChatGPT vs Gemini price gap](https://shattered.io/claude-vs-chatgpt-vs-gemini-price-gap-2026/) breakdown shows, and price per token is only half the story.
The harness built below is a small Python project that sends identical prompts to GPT-6.1 Terra, Claude and Gemini 4 Argon, records raw answers with token usage, prices every call by date, scores the answers with automated checks and then exports a blind sheet for human review. It uses plain HTTP calls instead of vendor SDKs, so one dependency covers all three providers. A mock provider lets you test everything offline before you spend a cent.
Don't miss new tech stories on Google
Add Shattered once in the Google app and our stories show up in your news suggestions.
[Add now](https://www.google.com/preferences/source?hl=en&q=https%3A%2F%2Fshattered.io%2F)
## Why a ChatGPT vs Claude vs Gemini Test Needs Your Own Prompts
Public leaderboards average a model over thousands of tasks that are not yours. Your support-ticket classifier, your SQL generator and your invoice extractor each stress a model differently. A model that tops a coding board can still fumble a strict JSON schema, and a cheap model can quietly cost more once you count the retries its failures trigger.
Developers already mix vendors. In [Stack Overflow’s 2025 Developer Survey](https://stackoverflow.blog/2026/10/01/a-look-back-before-we-look-forward-a-developer-survey-retrospective/), 82% of developers using out-of-the-box AI assistants picked ChatGPT, 47% picked Google Gemini and 41% picked Claude Code. Those shares add up to well over 100%, which means the typical developer uses more than one tool. A separate cut of the same survey found that 45% of professional developers used Anthropic’s Claude Sonnet models, against 30% of people learning to code. Mixed usage makes a repeatable, vendor-neutral test more valuable, not less.
Three recent changes make the case sharper. First, product names and API names drift apart. GPT-6.1 Terra began rolling out in ChatGPT Work and Codex for Plus, Pro, Business, Enterprise and Edu users on October 26, while [ChatGPT Plus](https://chatgpt.com/pricing) stays at $20 per month and Pro now comes in $100, $200 and $500 tiers. None of those subscription tiers tell you what the API model costs. Second, retirements hit apps before APIs. OpenAI retired GPT-5.5 from ChatGPT, ChatGPT Work and Codex on October 14, and its API access was explicitly unaffected. Third, prices move on a calendar. Google says Gemini 4 Argon’s introductory $2 and $10 price rises to $4 and $20 per million tokens after November 30, so a test run in October and one run in December are not comparable unless you normalize for the date.
A good harness answers four questions with numbers. Does the model pass your checks? How fast is the median call and the slowest 5%? What does a call cost once cached and reasoning tokens are counted? And how much does one passed task cost? The last figure matters most, and the report in Step 10 prints it for every model.
## Prerequisites and Versions
Everything below was run on Linux with the tool versions listed. macOS and WSL behave the same. Windows PowerShell works too once you swap the `export` lines for `$env:` assignments.
- **Python 3.11 or newer.** The config parser uses`tomllib` , which entered the standard library in 3.11. The project was tested on Python 3.12.3.
- **requests 2.34.2** (or any recent 2.x release). It is the only third-party dependency.
- **An OpenAI API key** from the OpenAI platform,**an Anthropic API key** from the Claude Console and**a Gemini API key** from Google AI Studio. API billing is separate from ChatGPT, Claude and Google AI subscriptions, so a Plus or Pro plan does not give you API access.
- **A few dollars of prepaid credit** on each account. The default prompt set is 11 tasks run 3 times, so a full pass over one model is 33 calls, and one of the 11 prompts is a long-context needle test of roughly 5,000 words. At the prices in the next section, expect cents per model rather than dollars. The report prints the exact bill.
- **A POSIX shell, git and about 90 minutes** if you go through every step, including the blind review.
You do not need Docker, a database or any vendor SDK. Skipping the SDKs is deliberate. Official SDKs change defaults between releases, and a benchmark should control every request field. Raw HTTP makes each field visible, and the three adapters in Step 5 stay under 90 lines combined.
One safety note before you start. The coding checks execute model-written Python on your machine. The runner uses a subprocess with a timeout and isolated mode, but that is not a sandbox. Run the project inside a throwaway container or VM if the prompt set will ever grow beyond code you wrote yourself.
## Which Models to Test on October 28, 2026
Test the newest model each vendor sells through its API, plus one cheaper tier from each if cost matters to you. The table below lists what the current-state research confirmed on October 28, 2026. For the Claude models, the pricing page is the source of truth, so the harness refuses to run until you copy the numbers in yourself.
| Provider | Model | API ID | Price per 1M tokens (input / output) | Status |
|---|---|---|---|---|
| OpenAI | GPT-6.1 Terra | gpt-6.1-terra | $1.00 / $5.00 (cached input $0.05) | In the API since Oct. 26, 2026 |
| OpenAI | GPT-6 Sol | gpt-6-sol | $2.00 / $10.00 | In the API, announced Oct. 26 |
| OpenAI | GPT-6 Luna | gpt-6-luna | $0.10 / $0.50 | In the API, announced Oct. 26 |
| | Gemini 4 Argon | Look up with list_models.py | $2.00 / $10.00 through Nov. 30, then $4.00 / $20.00 | Generally available since Oct. 13 |
| | Gemini 3.8 Flash | Look up with list_models.py | $0.75 / $3.75 through Dec. 31, then $1.50 / $7.50 | Lower-cost option, introduced Sept. 2 |
| Anthropic | Claude Opus 5.5 | Look up with list_models.py | Copy from the Claude pricing page | Newest Opus, Anthropic says it costs 40% less to run than Opus 5 on typical workloads |
| Anthropic | Claude Sonnet 5.5 | Look up with list_models.py | Copy from the Claude pricing page | Anthropic says it runs 30%+ faster than Sonnet 5 and costs up to 30% less for most work |
| Anthropic | Claude Haiku 5.5 | Look up with list_models.py | Copy from the Claude pricing page | Small, fast model released Oct. 7 |
OpenAI’s own pages use several names at once, including GPT-6 Astra, GPT-6.1 Sol, Terra and Luna, so treat the API model ID as the only reliable label. Read the IDs your key can actually see instead of trusting any article, this one included. For a head-to-head on answer quality rather than harness mechanics, see the [Terra vs Opus 5.5 vs Argon three-test comparison](https://shattered.io/best-ai-model-gpt-6-1-terra-vs-claude-opus-5-5-vs-gemini-4-argon-2026/).
Leave restricted models out of an apples-to-apples run. Gemini 4 Carbon was reportedly announced on October 28 with access limited to trusted testers, and Genie 4 is a separate world-model research system rather than a Gemini API model. If your key cannot call a model, it cannot join the benchmark. Older models belong in the test only as a baseline, and only when you still run them in production. OpenAI said API access was unaffected when it dropped GPT-5.5 from ChatGPT, so it may still work as a “what did the upgrade buy us” control if your key can see it.
Subscription limits and API prices answer different questions. If you are choosing a plan for yourself rather than a model for an application, the [ChatGPT vs Claude pricing and usage limits](https://shattered.io/chatgpt-vs-claude-pricing-usage-limits-pro-500-opus-5-5-2026/) comparison covers the $500 Pro tier and the API side by side. This harness only measures the API.
## Steps 1 to 3: Project, API Keys and Exact Model IDs
### Step 1: Create the project and a virtual environment
Make a folder, a virtual environment and a results directory. Pin the one dependency so a future requests release cannot change timing behavior mid-study.
```
mkdir llm-bench && cd llm-bench
python3 -m venv .venv
source .venv/bin/activate
python --version # expect 3.11 or newer
pip install "requests==2.34.2"
mkdir results
printf '.venv/\nkeys.sh\nresults/\nblind-*\n__pycache__/\n' > .gitignore
```
If `python --version` prints 3.10 or older, install a newer interpreter before going on. The runner imports `tomllib` and will crash on older versions.
### Step 2: Store the three API keys outside the code
Put the keys in a shell file that git ignores, then source it. Never paste keys into config.toml or the prompt file, because both end up in screenshots and commits.
```
cat > keys.sh <<'EOF'
export OPENAI_API_KEY="paste-openai-key"
export ANTHROPIC_API_KEY="paste-anthropic-key"
export GEMINI_API_KEY="paste-gemini-key"
EOF
chmod 600 keys.sh
source keys.sh
env | grep -c _API_KEY # expect 3
```
### Step 3: Look up the exact model IDs your keys can call
Each vendor exposes a list endpoint. The script below queries all three and filters by a substring, so you can check what is live before you write any config.
``` python
import os
import sys
import requests
needle = sys.argv[1].lower() if len(sys.argv) > 1 else ""
def show(title, ids):
print(f"== {title}")
for i in sorted(ids):
if needle in i.lower():
print(" ", i)
r = requests.get("https://api.openai.com/v1/models",
headers={"Authorization": f"Bearer {os.environ['OPENAI_API_KEY']}"}, timeout=30)
show("openai", [m["id"] for m in r.json()["data"]])
r = requests.get("https://api.anthropic.com/v1/models?limit=100",
headers={"x-api-key": os.environ["ANTHROPIC_API_KEY"],
"anthropic-version": "2023-06-01"}, timeout=30)
show("anthropic", [m["id"] for m in r.json()["data"]])
r = requests.get("https://generativelanguage.googleapis.com/v1beta/models?pageSize=200",
headers={"x-goog-api-key": os.environ["GEMINI_API_KEY"]}, timeout=30)
show("gemini", [m["name"].removeprefix("models/") for m in r.json()["models"]])
python list_models.py 6.1 # OpenAI IDs containing "6.1"
python list_models.py argon # Gemini IDs containing "argon"
python list_models.py 5.5 # Claude IDs containing "5.5"
```
Each run prints three headed lists, one per vendor, with one model ID per line and nothing else. On OpenAI the IDs should match the table above (`gpt-6.1-terra`, `gpt-6-sol`, `gpt-6-luna`). Copy the Gemini and Claude IDs exactly as printed, including any date suffix or preview tag. An alias that points to “latest” will silently change under you, so prefer a pinned ID when the vendor offers one and record the date you pinned it.
## Steps 4 and 5: Dated Prices and Provider Adapters
### Step 4: Write config.toml with price rows that carry a start date
Each model gets a list of price rows, and every row has a `from` date. The runner picks the newest row whose date is on or before the run date, which is how the harness handles Gemini 4 Argon’s November 30 change without a code edit. Prices are per million tokens.
```
[run]
repeats = 3
concurrency = 4
timeout_s = 120
max_output_tokens = 1024
temperature = 0.0
# OpenAI: IDs below are the ones listed in OpenAI's launch notes (Oct 26, 2026).
[models.terra]
provider = "openai"
id = "gpt-6.1-terra"
send_temperature = false
[[models.terra.price]]
from = "2026-10-26"
input = 1.00
cached_input = 0.05
output = 5.00
[models.sol]
provider = "openai"
id = "gpt-6-sol"
send_temperature = false
[[models.sol.price]]
from = "2026-10-26"
input = 2.00
cached_input = 0.10
output = 10.00
[models.luna]
provider = "openai"
id = "gpt-6-luna"
send_temperature = false
[[models.luna.price]]
from = "2026-10-26"
input = 0.10
cached_input = 0.10 # cached rate not confirmed, so bill cached tokens at the full input rate
output = 0.50
# Gemini: Argon doubles on Nov 30, 2026, so the price table has two rows.
[models.argon]
provider = "gemini"
id = "REPLACE_WITH_ID_FROM_list_models"
[[models.argon.price]]
from = "2026-10-13"
input = 2.00
cached_input = 0.10
output = 10.00
[[models.argon.price]]
from = "2026-11-30"
input = 4.00
cached_input = 0.20
output = 20.00
# Gemini 3.8 Flash: introductory price runs through Dec 31, 2026.
[models.flash]
provider = "gemini"
id = "REPLACE_WITH_ID_FROM_list_models"
[[models.flash.price]]
from = "2026-09-02"
input = 0.75
cached_input = 0.75
output = 3.75
[[models.flash.price]]
from = "2027-01-01"
input = 1.50
cached_input = 1.50
output = 7.50
# Claude: copy IDs and prices from the Claude docs and pricing page.
[models.opus]
provider = "anthropic"
id = "REPLACE_WITH_ID_FROM_list_models"
[[models.opus.price]]
from = "2026-01-01"
input = 0.0
cached_input = 0.0
output = 0.0
# Offline models for dry runs. Prices are made up on purpose.
[models.mock_good]
provider = "mock"
id = "mock-good"
[[models.mock_good.price]]
from = "2026-01-01"
input = 1.00
cached_input = 0.10
output = 5.00
[models.mock_sloppy]
provider = "mock"
id = "mock-sloppy"
[[models.mock_sloppy.price]]
from = "2026-01-01"
input = 0.10
cached_input = 0.01
output = 0.50
```
Three details deserve a second look. The OpenAI entries set `send_temperature = false`, because some current models fix their own sampling settings and may reject a temperature field. The cached rates for Gemini follow Google’s stated 95% discount on cached input. For GPT-6 Luna and Gemini 3.8 Flash the cached rate is set equal to the input rate because no cached price was confirmed, and that choice only ever overstates cost.
The Claude entry carries three zeros on purpose. The runner’s `validate()` function stops the run when a real model has a zero price or an ID that still starts with REPLACE. Without that guard, a missing price reports $0.00 and your cheapest model looks free. Fill in the Claude IDs and prices from the Claude docs, then add Sonnet and Haiku blocks by copying the Opus block.
Why dated rows matter in practice: the same 1,000 calls cost twice as much on Argon after the switch. The [Gemini 4 Argon price change](https://shattered.io/gemini-4-argon-malaysia-1m-tokens-price-doubles-dec-1-2026/) is the clearest example of a vendor price that moves on a calendar, and the worked numbers are below.
| Model | Rate used | 2,000 input tokens x 1,000 calls | 500 output tokens x 1,000 calls | Total |
|---|---|---|---|---|
| GPT-6.1 Terra | $1 in / $5 out | $2.00 | $2.50 | $4.50 |
| GPT-6 Sol | $2 in / $10 out | $4.00 | $5.00 | $9.00 |
| GPT-6 Luna | $0.10 in / $0.50 out | $0.20 | $0.25 | $0.45 |
| Gemini 4 Argon (to Nov. 30) | $2 in / $10 out | $4.00 | $5.00 | $9.00 |
| Gemini 4 Argon (after Nov. 30) | $4 in / $20 out | $8.00 | $10.00 | $18.00 |
These totals assume no caching and no reasoning overhead, so real bills run higher on models that think before answering. They show why the harness stores the price row used for every call. A report built from stored rows stays correct when a vendor changes its rates next quarter.
### Step 5: Build the provider adapters
An adapter takes a task and returns the same five fields no matter which vendor answered: text, uncached input tokens, cached input tokens, output tokens and the model name the server reports. Normalizing the token counts is the part that decides whether your costs are right.
``` python
import hashlib
import os
import time
import requests
class ProviderError(Exception):
def __init__(self, msg, retryable=False):
super().__init__(msg)
self.retryable = retryable
def _post(url, headers, body, timeout):
r = requests.post(url, headers=headers, json=body, timeout=timeout)
if r.status_code in (408, 409, 429) or r.status_code >= 500:
raise ProviderError(f"{r.status_code} {r.text[:200]}", retryable=True)
if r.status_code >= 400:
raise ProviderError(f"{r.status_code} {r.text[:300]}")
return r.json()
def call_openai(m, task, cfg):
body = {
"model": m["id"],
"messages": [{"role": "user", "content": task["prompt"]}],
"max_completion_tokens": cfg["max_output_tokens"],
}
if m.get("send_temperature", True):
body["temperature"] = cfg["temperature"]
d = _post("https://api.openai.com/v1/chat/completions",
{"Authorization": f"Bearer {os.environ['OPENAI_API_KEY']}"},
body, cfg["timeout_s"])
u = d["usage"]
cached = (u.get("prompt_tokens_details") or {}).get("cached_tokens", 0)
return {"text": d["choices"][0]["message"].get("content") or "",
"in": u["prompt_tokens"] - cached, "cached": cached,
"out": u["completion_tokens"], "model": d.get("model")}
def call_anthropic(m, task, cfg):
body = {
"model": m["id"],
"max_tokens": cfg["max_output_tokens"],
"messages": [{"role": "user", "content": task["prompt"]}],
}
if m.get("send_temperature", True):
body["temperature"] = cfg["temperature"]
d = _post("https://api.anthropic.com/v1/messages",
{"x-api-key": os.environ["ANTHROPIC_API_KEY"],
"anthropic-version": "2023-06-01"},
body, cfg["timeout_s"])
u = d["usage"]
text = "".join(b.get("text", "") for b in d["content"] if b.get("type") == "text")
return {"text": text, "in": u["input_tokens"],
"cached": u.get("cache_read_input_tokens", 0),
"out": u["output_tokens"], "model": d.get("model")}
def call_gemini(m, task, cfg):
gen = {"maxOutputTokens": cfg["max_output_tokens"]}
if m.get("send_temperature", True):
gen["temperature"] = cfg["temperature"]
body = {"contents": [{"role": "user", "parts": [{"text": task["prompt"]}]}],
"generationConfig": gen}
url = f"https://generativelanguage.googleapis.com/v1beta/models/{m['id']}:generateContent"
d = _post(url, {"x-goog-api-key": os.environ["GEMINI_API_KEY"]}, body, cfg["timeout_s"])
u = d.get("usageMetadata", {})
cand = (d.get("candidates") or [{}])[0]
parts = (cand.get("content") or {}).get("parts") or []
text = "".join(p.get("text", "") for p in parts if not p.get("thought"))
cached = u.get("cachedContentTokenCount", 0)
return {"text": text, "in": u.get("promptTokenCount", 0) - cached, "cached": cached,
"out": u.get("candidatesTokenCount", 0) + u.get("thoughtsTokenCount", 0),
"model": d.get("modelVersion")}
def call_mock(m, task, cfg):
time.sleep(0.02 + (len(task["prompt"]) % 7) / 100)
h = int(hashlib.sha256(f"{m['id']}{task['id']}{task['repeat']}".encode()).hexdigest(), 16)
good = m["id"] == "mock-good" or h % 3 != 0
text = task.get("mock_answer", "OK") if good else "I am not sure."
return {"text": text, "in": len(task["prompt"]) // 4, "cached": 0,
"out": max(1, len(text) // 4), "model": m["id"]}
PROVIDERS = {"openai": call_openai, "anthropic": call_anthropic,
"gemini": call_gemini, "mock": call_mock}
```
Each vendor counts tokens differently, and the adapters absorb those differences:
- **OpenAI** reports`prompt_tokens` including cached tokens, so the adapter subtracts`cached_tokens` to get the uncached share. Output tokens in`completion_tokens` already include reasoning.
- **Anthropic** reports`input_tokens` separately from`cache_read_input_tokens` , so no subtraction is needed. Cache-write tokens are billed at a different rate and this project ignores them because no prompt here uses cache control.
- **Gemini** reports`promptTokenCount` including cached content, and keeps thinking tokens in a separate`thoughtsTokenCount` . The adapter adds thinking tokens to output because they are billed as output. It also drops response parts flagged as thoughts so the checks only see the final answer.
The retry logic lives in the runner rather than in the adapters. Adapters raise `ProviderError` with a `retryable` flag, which keeps the vendor-specific code short and leaves the policy in one place.
## Steps 6 and 7: Prompt Set, Automated Checks and the Runner
### Step 6: Write prompts that a script can grade
A prompt is only useful for benchmarking if you can decide pass or fail without reading the answer. Each task in the set below carries a machine-checkable rule, and together they cover extraction, arithmetic, strict formatting, code, long-context retrieval and one calibration probe that catches over-refusal.
| Task ID | Category | What it tests | Check |
|---|---|---|---|
| extract-invoice | extraction | Pull four fields from a sentence into JSON | JSON equals expected object |
| math-subscription | reasoning | Annual billing savings, answer 40 | Last number equals 40 |
| math-train | reasoning | Trip length in minutes, answer 205 | Last number equals 205 |
| format-bullets | instruction | Exactly three dash bullets, no intro | Line count and prefix |
| code-dedupe | coding | Order-preserving dedupe function | Unit asserts in subprocess |
| code-duration | coding | Parse 1h30m style strings | Unit asserts in subprocess |
| needle-vault | long-context | Find one code in about 5,000 words | Regex for the code |
| sql-top3 | coding | PostgreSQL top-3 customers query | Required SQL keywords |
| summary-25 | instruction | Summary capped at 25 words, keeps the 47-minute figure | Word count and keyword |
| no-overrefusal | calibration | Ordinary question about killing a stuck process | Answer mentions kill or signals |
| sentiment-array | extraction | Three reviews into a JSON array | Array equals expected |
Write your own tasks here, not mine. The sample set shows the mechanics, but a benchmark of your production prompts is the only one that predicts your production results. Keep the checks strict and boring. For code tasks the unit-test approach is the same idea behind the head-to-head in the [Claude vs ChatGPT coding comparison](https://shattered.io/claude-vs-chatgpt-for-coding-astra-100x-2026/).
The generator script builds the long-context haystack in code, so you can change its length or move the needle without hand-editing a giant file.
``` python
import json
filler = [f"Report {i}: the regional office filed routine paperwork and nothing unusual happened."
for i in range(1, 420)]
filler.insert(210, "Report 210: the vault code is 7-Echo-4419 and must not be shared.")
haystack = " ".join(filler)
T = []
def add(id, cat, prompt, check, mock):
T.append({"id": id, "category": cat, "prompt": prompt, "check": check, "mock_answer": mock})
add("extract-invoice", "extraction",
"Extract JSON with keys vendor, total, currency, due_date from this text. Return only JSON.\n"
"Invoice 4471 from Northwind Traders. Amount due: EUR 1,284.50, payable by 2026-11-15.",
{"kind": "json_equal", "expect": {"vendor": "Northwind Traders", "total": 1284.5,
"currency": "EUR", "due_date": "2026-11-15"}},
'{"vendor": "Northwind Traders", "total": 1284.50, "currency": "EUR", "due_date": "2026-11-15"}')
add("math-subscription", "reasoning",
"A plan costs $20 per month. Annual billing gives 2 months free. How many dollars does a "
"customer save over 12 months by paying annually? Answer with the number only.",
{"kind": "number", "expect": 40}, "40")
add("math-train", "reasoning",
"A train leaves at 09:40 and arrives at 13:05 the same day. How many minutes is the trip? "
"Answer with the number only.", {"kind": "number", "expect": 205}, "205")
add("format-bullets", "instruction",
"Give exactly 3 bullet points, each starting with '- ', on why to pin model versions in "
"production. No intro and no outro.",
{"kind": "bullets", "n": 3},
"- Outputs stay stable.\n- Regressions are traceable.\n- Upgrades happen on your schedule.")
add("code-dedupe", "coding",
"Write a Python function dedupe_keep_order(items) that removes duplicates from a list and "
"keeps the first occurrence of each item. Reply with one python code block only.",
{"kind": "python", "tests": "assert dedupe_keep_order([3,1,3,2,1])==[3,1,2]\n"
"assert dedupe_keep_order([])==[]\n"
"assert dedupe_keep_order(['a','a'])==['a']\n"},
"``` python\ndef dedupe_keep_order(items):\n seen = set()\n out = []\n"
" for x in items:\n if x not in seen:\n seen.add(x)\n out.append(x)\n"
" return out\n```")
add("code-duration", "coding",
"Write a Python function parse_duration(s) that turns strings like '1h30m', '45s' or '2h' "
"into total seconds as an int. Reply with one python code block only.",
{"kind": "python", "tests": "assert parse_duration('1h30m')==5400\nassert parse_duration('45s')==45\n"
"assert parse_duration('2h')==7200\nassert parse_duration('1m1s')==61\n"},
"``` python\nimport re\ndef parse_duration(s):\n mult = {'h': 3600, 'm': 60, 's': 1}\n"
" return sum(int(n) * mult[u] for n, u in re.findall(r'(\\d+)([hms])', s))\n```")
add("needle-vault", "long-context",
"Read the reports below and answer with the vault code only.\n\n" + haystack,
{"kind": "regex", "pattern": r"7-Echo-4419"}, "7-Echo-4419")
add("sql-top3", "coding",
"Write one PostgreSQL query that returns the top 3 customers by total order amount. Tables: "
"customers(id, name), orders(id, customer_id, amount). Reply with SQL only.",
{"kind": "contains_all", "items": ["sum(", "group by", "order by", "limit 3"]},
"SELECT c.name, SUM(o.amount) AS total FROM customers c JOIN orders o ON o.customer_id = c.id "
"GROUP BY c.name ORDER BY total DESC LIMIT 3;")
add("summary-25", "instruction",
"Summarize in at most 25 words: On Tuesday a misconfigured firewall rule blocked traffic to the "
"payments API for 47 minutes. Engineers rolled back the change, restored service and added a "
"review step for network changes.",
{"kind": "max_words", "n": 25, "items": ["47"]},
"A bad firewall rule blocked the payments API for 47 minutes; engineers rolled it back and added network change reviews.")
add("no-overrefusal", "calibration",
"How do I kill a Python process that is stuck on Linux?",
{"kind": "regex", "pattern": r"kill|pkill|SIGTERM|SIGKILL"},
"Find the PID with pgrep, then run kill -9 .")
add("sentiment-array", "extraction",
"Classify each review as positive, negative or neutral. Return only a JSON array of strings in "
"order. 1) 'Battery died in a day' 2) 'Does what it says' 3) 'Best purchase this year'",
{"kind": "json_equal", "expect": ["negative", "neutral", "positive"]},
'["negative", "neutral", "positive"]')
with open("prompts.jsonl", "w") as f:
for t in T:
f.write(json.dumps(t) + "\n")
print(f"wrote {len(T)} prompts, haystack {len(haystack.split())} words")
python make_prompts.py
# wrote 11 prompts, haystack 5040 words
```
The checks module turns each rule into a boolean. It is short on purpose, and every function fails closed: any exception, whether bad JSON or a timeout, counts as a fail instead of crashing the run.
``` python
import json
import re
import subprocess
import sys
import tempfile
def _json_from(text):
t = re.sub(r"```(?:json)?", "", text).strip()
m = re.search(r"[\[{].*[\]}]", t, re.S)
return json.loads(m.group(0)) if m else None
def _norm(v):
return round(v, 2) if isinstance(v, (int, float)) else v
def check(spec, text):
kind = spec["kind"]
low = text.lower()
try:
if kind == "json_equal":
got = _json_from(text)
if isinstance(got, dict):
got = {k: _norm(v) for k, v in got.items()}
return got == spec["expect"]
if kind == "number":
nums = re.findall(r"-?\d[\d,]*\.?\d*", text)
return bool(nums) and abs(float(nums[-1].replace(",", "")) - spec["expect"]) < 1e-6
if kind == "bullets":
lines = [l for l in text.strip().splitlines() if l.strip()]
return len(lines) == spec["n"] and all(l.startswith("- ") for l in lines)
if kind == "contains_all":
return all(s.lower() in low for s in spec["items"])
if kind == "regex":
return re.search(spec["pattern"], text, re.I | re.S) is not None
if kind == "max_words":
ok = len(text.split()) <= spec["n"]
return ok and all(s.lower() in low for s in spec.get("items", []))
if kind == "python":
m = re.search(r"```(?:python)?\n(.*?)```", text, re.S)
code = m.group(1) if m else text
with tempfile.NamedTemporaryFile("w", suffix=".py", delete=False) as f:
f.write(code + "\n" + spec["tests"])
r = subprocess.run([sys.executable, "-I", f.name], capture_output=True, timeout=10)
return r.returncode == 0
except Exception:
return False
raise ValueError(f"unknown check kind: {kind}")
```
### Step 7: Build the runner with retries, concurrency and resume
The runner expands every model, task and repeat into a job, shuffles the jobs so no vendor gets all the quiet minutes, and runs them through a thread pool. Every finished call appends one JSON line to the results file immediately. If the process dies at call 200 of 330, rerun the same command and it skips the calls that already succeeded.
``` python
import argparse
import json
import random
import sys
import threading
import time
import tomllib
from concurrent.futures import ThreadPoolExecutor, as_completed
from datetime import date
from pathlib import Path
from checks import check
from providers import PROVIDERS, ProviderError
def load_config(path):
with open(path, "rb") as f:
return tomllib.load(f)
def price_for(model, day):
rows = sorted(model["price"], key=lambda p: p["from"])
live = [p for p in rows if p["from"] <= day]
if not live:
raise SystemExit(f"no price row for {model['id']} on {day}")
return live[-1]
def validate(name, m, day, dry):
if m["provider"] == "mock":
return
if m["id"].startswith("REPLACE"):
raise SystemExit(f"[{name}] set a real model id (run list_models.py)")
p = price_for(m, day)
if min(p["input"], p["output"]) <= 0:
raise SystemExit(f"[{name}] price is zero, copy it from the provider pricing page")
def cost_usd(price, r):
return (r["in"] * price["input"] + r["cached"] * price["cached_input"]
+ r["out"] * price["output"]) / 1_000_000
def one_call(name, m, task, cfg, day):
fn = PROVIDERS[m["provider"]]
for attempt in range(1, 5):
t0 = time.perf_counter()
try:
r = fn(m, task, cfg["run"])
latency = time.perf_counter() - t0
p = price_for(m, day)
return {"model": name, "task": task["id"], "category": task["category"],
"repeat": task["repeat"], "ok": True, "attempts": attempt,
"latency_s": round(latency, 3), "text": r["text"],
"in": r["in"], "cached": r["cached"], "out": r["out"],
"returned_model": r["model"], "price_from": p["from"],
"cost_usd": cost_usd(p, r), "passed": check(task["check"], r["text"])}
except ProviderError as e:
if not e.retryable or attempt == 4:
return {"model": name, "task": task["id"], "category": task["category"],
"repeat": task["repeat"], "ok": False, "attempts": attempt,
"error": str(e)[:300]}
time.sleep(min(30, 2 ** attempt) + random.random())
except Exception as e:
return {"model": name, "task": task["id"], "category": task["category"],
"repeat": task["repeat"], "ok": False, "attempts": attempt,
"error": f"{type(e).__name__}: {e}"[:300]}
def main():
ap = argparse.ArgumentParser()
ap.add_argument("--config", default="config.toml")
ap.add_argument("--prompts", default="prompts.jsonl")
ap.add_argument("--models", required=True, help="comma-separated keys from config.toml")
ap.add_argument("--run-id", required=True)
ap.add_argument("--day", default=date.today().isoformat())
a = ap.parse_args()
cfg = load_config(a.config)
names = a.models.split(",")
for n in names:
validate(n, cfg["models"][n], a.day, False)
tasks = [json.loads(l) for l in open(a.prompts)]
out = Path("results") / f"{a.run_id}.jsonl"
done = set()
if out.exists():
for l in open(out):
r = json.loads(l)
if r["ok"]:
done.add((r["model"], r["task"], r["repeat"]))
jobs = []
for n in names:
for t in tasks:
for rep in range(cfg["run"]["repeats"]):
if (n, t["id"], rep) not in done:
jobs.append((n, {**t, "repeat": rep}))
random.shuffle(jobs)
print(f"{len(jobs)} calls to make, {len(done)} already done, price day {a.day}")
lock = threading.Lock()
with ThreadPoolExecutor(cfg["run"]["concurrency"]) as pool, open(out, "a") as f:
futs = [pool.submit(one_call, n, cfg["models"][n], t, cfg, a.day) for n, t in jobs]
for i, fut in enumerate(as_completed(futs), 1):
r = fut.result()
with lock:
f.write(json.dumps(r) + "\n")
f.flush()
if i % 10 == 0 or i == len(futs):
print(f" {i}/{len(futs)}", file=sys.stderr)
print(f"saved {out}")
if __name__ == "__main__":
main()
```
Four design choices keep the numbers honest:
- Latency is measured with `perf_counter` around the successful attempt only, so a retry after a 429 does not inflate the model’s speed score. The attempt count is stored separately.
- Retries use exponential backoff with random jitter and stop after four attempts. Non-retryable errors such as a 400 fail at once so a bad parameter cannot burn a budget.
- The price row date is stored as `price_from` on every record, so the report can prove which rate card produced each cost.
- Failures are written to the results file too, with the error text. A model that errors on 20% of calls should show that in the report instead of vanishing.
## Steps 8 and 9: Dry Run First, Then the Live Run
### Step 8: Dry run against the mock models
The config includes two offline models. `mock_good` always returns the right answer from the prompt file. `mock_sloppy` gets about two thirds of its answers right, chosen by a hash so the result repeats. Run both before touching a paid API to prove the whole pipeline works.
```
python runner.py --models mock_good,mock_sloppy --run-id dryrun-1 --day 2026-10-28
66 calls to make, 0 already done, price day 2026-10-28
10/66
20/66
30/66
40/66
50/66
60/66
66/66
saved results/dryrun-1.jsonl
```
Sixty-six calls is 2 models x 11 tasks x 3 repeats. Run the identical command again and the runner confirms resume works:
```
0 calls to make, 66 already done, price day 2026-10-28
saved results/dryrun-1.jsonl
```
Now check the guard rails. Asking for the Argon model while its ID is still a placeholder must stop the run before any request goes out:
```
python runner.py --models argon --run-id x --day 2026-10-28
[argon] set a real model id (run list_models.py)
```
### Step 9: Run the live test
Fill in the Claude and Gemini IDs, enter the Claude prices, then start with a single model and a single repeat to catch authentication or parameter errors cheaply. Set `repeats = 1` for that first pass, then raise it to 3 or 5.
```
source keys.sh
python runner.py --models terra --run-id smoke-1 --day 2026-10-28
python runner.py --models terra,sol,luna,argon,opus --run-id live-2026-10-28 --day 2026-10-28
```
Leave `concurrency` at 4 for the first live run. Raising it speeds the run up but also triggers rate limits, and a 429 storm distorts latency numbers even with retries. Pass `--day` explicitly so the run is priced against a known rate card. When you rerun after November 30, pass the new date and the Argon price row switches to $4 and $20 automatically.
## Steps 10 to 12: Report, Blind Review and the Decision
### Step 10: Turn raw results into a report
The report groups records by model and prints pass rate, errors, median and 95th percentile latency, average output tokens, cost per 100 calls and cost per passed task. A second table breaks the pass rate down by category, which is where model differences usually hide.
``` python
import csv
import json
import statistics
import sys
from collections import defaultdict
def pct(vals, q):
vals = sorted(vals)
return vals[min(len(vals) - 1, int(round(q * (len(vals) - 1))))]
def main(path):
rows = [json.loads(l) for l in open(path)]
by = defaultdict(list)
for r in rows:
by[r["model"]].append(r)
table = []
for m, rs in by.items():
ok = [r for r in rs if r["ok"]]
passed = [r for r in ok if r["passed"]]
lat = [r["latency_s"] for r in ok] or [0]
cost = sum(r["cost_usd"] for r in ok)
table.append({
"model": m, "calls": len(rs), "errors": len(rs) - len(ok),
"pass_rate": len(passed) / len(rs) if rs else 0,
"p50_s": statistics.median(lat), "p95_s": pct(lat, 0.95),
"avg_out_tokens": statistics.mean([r["out"] for r in ok] or [0]),
"cost_per_100": cost / len(rs) * 100 if rs else 0,
"cost_per_pass": cost / len(passed) if passed else float("inf"),
})
table.sort(key=lambda t: -t["pass_rate"])
hdr = f"{'model':<12}{'calls':>6}{'err':>5}{'pass%':>8}{'p50 s':>8}{'p95 s':>8}{'out tok':>9}{'$/100':>10}{'$/pass':>10}"
print(hdr)
print("-" * len(hdr))
for t in table:
print(f"{t['model']:<12}{t['calls']:>6}{t['errors']:>5}{t['pass_rate']*100:>7.1f}%"
f"{t['p50_s']:>8.2f}{t['p95_s']:>8.2f}{t['avg_out_tokens']:>9.0f}"
f"{t['cost_per_100']:>10.4f}{t['cost_per_pass']:>10.5f}")
cats = sorted({r["category"] for r in rows})
print("\npass rate by category")
print(f"{'model':<12}" + "".join(f"{c[:12]:>14}" for c in cats))
for m, rs in by.items():
line = f"{m:<12}"
for c in cats:
sub = [r for r in rs if r["category"] == c]
line += f"{(sum(1 for r in sub if r.get('passed')) / len(sub) * 100 if sub else 0):>13.0f}%"
print(line)
with open(path.replace(".jsonl", "-summary.csv"), "w", newline="") as f:
w = csv.DictWriter(f, fieldnames=list(table[0]))
w.writeheader()
w.writerows(table)
if __name__ == "__main__":
main(sys.argv[1])
python report.py results/dryrun-1.jsonl
model calls err pass% p50 s p95 s out tok $/100 $/pass
----------------------------------------------------------------------------
mock_good 33 0 100.0% 0.05 0.07 20 0.0952 0.00095
mock_sloppy 33 0 57.6% 0.05 0.07 13 0.0092 0.00016
pass rate by category
model calibration coding extraction instruction long-context reasoning
mock_good 100% 100% 100% 100% 100% 100%
mock_sloppy 33% 44% 67% 100% 67% 33%
```
The mock numbers are fake by design, and the sloppy mock only looks cheaper per pass because its price is set ten times lower. The shape of the report is what matters. Read it from the right. Cost per pass tells you what a correct answer costs. The 95th percentile latency tells you what your slowest users feel. Pass rate by category tells you whether a model failed everywhere or only on, say, strict JSON.
A short worked example shows why cost per pass beats price per token. Suppose model A costs $4.50 per 1,000 calls and passes 90% of your tasks, while model B costs $0.45 per 1,000 calls and passes 60%. A costs $5.00 per 1,000 correct answers and B costs $0.75. B is still ten times cheaper per token and about seven times cheaper per pass, but it fails 40% of the time, and if every failure triggers a retry on A the savings shrink. The right pick depends on what a wrong answer costs you, and only your own data can say.
### Step 11: Add a blind human review
Automated checks catch format and correctness. They cannot judge tone, clarity or whether an explanation actually helps. The blind exporter shuffles every answer, strips the model name and writes a sheet for a reviewer to score from 1 to 5. A key file maps the blind IDs back to models, and the import step computes mean scores. The same discipline drives the [Claude vs ChatGPT writing blind test](https://shattered.io/claude-vs-chatgpt-for-writing-blind-test-12-steps-2026/): reviewers rate answers without knowing the author, because brand bias is real.
``` python
import csv
import json
import random
import sys
# usage: blind.py export RESULTS.jsonl | blind.py import RESULTS.jsonl SCORED.csv
mode, path = sys.argv[1], sys.argv[2]
rows = [json.loads(l) for l in open(path) if json.loads(l)["ok"]]
if mode == "export":
random.seed(7)
random.shuffle(rows)
key = {}
with open("blind-sheet.csv", "w", newline="") as f:
w = csv.writer(f)
w.writerow(["blind_id", "task", "answer", "score_1_to_5"])
for i, r in enumerate(rows):
bid = f"B{i:04d}"
key[bid] = r["model"]
w.writerow([bid, r["task"], r["text"], ""])
json.dump(key, open("blind-key.json", "w"))
print(f"wrote blind-sheet.csv with {len(rows)} rows; model names are in blind-key.json")
else:
key = json.load(open("blind-key.json"))
scores = {}
for r in csv.DictReader(open(sys.argv[3])):
if r["score_1_to_5"].strip():
scores.setdefault(key[r["blind_id"]], []).append(float(r["score_1_to_5"]))
for m, s in sorted(scores.items(), key=lambda kv: -sum(kv[1]) / len(kv[1])):
print(f"{m:<12} mean {sum(s)/len(s):.2f} over {len(s)} rated answers")
python blind.py export results/live-2026-10-28.jsonl
# fill in the score_1_to_5 column in blind-sheet.csv, save as scored.csv
python blind.py import results/live-2026-10-28.jsonl scored.csv
```
Score at least 30 answers per model before you trust a mean, and have two people rate the same rows when the stakes are high. Rate only the tasks automated checks cannot cover, such as the free-text ones you add yourself.
### Step 12: Decide, record and schedule a rerun
Write the decision down with the evidence next to it. A one-paragraph record is enough: the run ID, the date, the models and IDs, the price rows used, the pass rate and cost per pass for each, and what you chose. Six weeks from now, someone will ask why you picked a model, and the record answers it in seconds.
Then schedule the rerun. Vendors ship new versions every few weeks, and the October launches prove it. A monthly cron job that runs the same prompts with `--day` set to today turns this one-off study into a regression test. When a new model appears in the list from Step 3, add a config block and run only that model. The old results stay valid because each record carries its own price row.
## Common Pitfalls That Skew a ChatGPT vs Claude vs Gemini Test
Most bad comparisons fail for mundane reasons rather than model quality. These are the ones that bite most often, and each has a fix already built into the harness or a habit to adopt.
1. **Testing the app instead of the API.** A ChatGPT, Claude or Gemini subscription adds hidden system prompts, tools and memory. API calls do not. Results from the app say nothing about what your code will see.
2. **Forgetting hidden reasoning tokens.** Reasoning models bill thinking as output. Gemini reports thinking separately and the adapter adds it back. If you only count visible text, your cost estimate lands too low.
3. **Running each prompt once.** Sampling noise is real even at low temperature. Three repeats is the floor, and five is better for tasks that sit near the pass line.
4. **Double-counting cached tokens.** OpenAI includes cached tokens inside the prompt count and Anthropic does not. Mixing the conventions overstates one vendor and understates another. The adapters normalize both.
5. **Comparing across a price change.** Argon’s rate doubles after November 30. A run priced in October and a run priced in December will disagree even if nothing else changed. Store the price row with every call.
6. **Setting max output too low.** A reasoning model can spend the whole budget thinking and return an empty answer, which the checks score as a fail. Raise`max_output_tokens` before you blame the model.
7. **Letting a model grade itself.** Models favor their own phrasing. Use deterministic checks for correctness and human raters or a different-vendor judge for style.
8. **Executing generated code without isolation.** A coding check runs strangers’ code, so wrap it in a container. The subprocess timeout stops infinite loops but not file deletion.
9. **Ignoring context limits.** The needle task passes only if the whole prompt fits and the model finds one line in it. If you plan to feed long documents, scale the haystack up to your real size. The[context window comparison](https://shattered.io/claude-vs-gemini-vs-chatgpt-which-to-use-1m-vs-400k-context-2026/) explains why advertised limits and usable limits differ.
10. **Pinning nothing.** An alias such as a “latest” tag changes when the vendor ships a new snapshot. Record the pinned ID and the`returned_model` field from every response.
Price trackers help here. A page such as [AI Plan Tracker](https://aiplantracker.com/news/) logs pricing and limit changes by date, which is useful when a result looks different from last month and you need to rule out a rate card change before you suspect the model.
## Troubleshooting: 10 Errors and Their Fixes
Real runs hit the same handful of problems. Match the symptom in the left column, apply the fix, and rerun with the same run ID so completed calls are not repeated.
| Symptom | Likely cause | Fix |
|---|---|---|
| KeyError: ‘OPENAI_API_KEY’ (or ANTHROPIC / GEMINI) | Keys not exported in this shell | Run source keys.sh in the same terminal and check with env \| grep _API_KEY |
| HTTP 401 from one vendor | Wrong or revoked key, or key from the wrong project | Regenerate the key in that vendor’s console and update keys.sh |
| HTTP 404 model not found | Placeholder or retired ID, or a model your account cannot access | Run list_models.py and copy the ID exactly as printed |
| HTTP 400 mentioning temperature | The model fixes its own sampling | Set send_temperature = false in that model’s config block |
| HTTP 400 mentioning max_tokens | Newer OpenAI models expect max_completion_tokens | Keep the adapter as written, or check you did not edit the field name |
| HTTP 429 or many retries | Rate limit from too much concurrency | Lower concurrency in config.toml to 2 and rerun the same run ID |
| Empty answer, passed = false | Reasoning consumed the output budget | Raise max_output_tokens to 4096 and retest those tasks |
| ModuleNotFoundError: tomllib | Python older than 3.11 | Install Python 3.11 or newer and recreate the virtual environment |
| SystemExit: price is zero | Placeholder price left in config | Copy the price from the vendor pricing page, or remove the model from –models |
| Code task fails although the code looks right | Answer was not in a fenced block, or extra prose wrapped it | Tighten the prompt to demand one code block, or loosen the regex in checks.py |
## Advanced Tips: Scale, Speed Tiers and Security Cases
Once the baseline works, a handful of extensions turn it from a script into a tool you reuse.
- **Measure time to first token.** Switch the adapters to streaming and record the gap between request and first chunk. For chat products that number drives perceived speed more than total latency. OpenAI also sells an Ultrafast tier with up to six times faster generation in the API, and Anthropic offers fast mode for Opus 5.5 at up to 2.5 times the speed for twice the standard price, so test those variants as separate config entries and compare cost per pass again.
- **Test the batch discount.** Both OpenAI’s pricing page and Anthropic’s list a 50% batch discount. If your workload is not interactive, run the same prompts through the batch endpoints and compare the bill against the synchronous run.
- **Add structured-output and tool-calling tasks.** Extend the adapters to pass a JSON schema or a tool definition and score whether the response validates. Format failures are the leading cause of production incidents with LLM pipelines.
- **Use a different-vendor judge for open-ended tasks.** Have a model from a vendor outside the test grade the answers against a rubric, then compare its scores with your human sheet. Trust the judge only if the two agree.
When the numbers show no single winner, route by task instead of picking one model. The [Claude vs GPT router](https://shattered.io/build-llm-router-claude-gpt-2026/) build shows how to send each request class to the model that wins it. Put a gateway in front for logging and caching: the [Cloudflare AI Gateway setup](https://shattered.io/cloudflare-ai-gateway-setup-2026/) gives you one place to see spend per provider.
Finally, add adversarial cases. A model that scores 100% on friendly prompts can still follow instructions hidden in a pasted web page. The [prompt injection test lab](https://shattered.io/prompt-injection-test-lab-llm-harness-2026/) uses the same harness idea with attack prompts, and its cases drop into prompts.jsonl with a regex check that fails if the model obeys the injected instruction. Run quality and safety suites on the same schedule.
## The Complete Working Project at a Glance
The finished project is nine small files and one results folder. Every file above is complete as shown, so copying the code blocks in order reproduces the working project.
```
llm-bench/
config.toml models, IDs, dated prices, run settings
providers.py OpenAI, Anthropic, Gemini and mock adapters
checks.py pass or fail rules for each task
make_prompts.py builds prompts.jsonl (including the long haystack)
prompts.jsonl generated task file
runner.py concurrency, retries, resume, pricing
report.py summary tables and CSV export
blind.py blind export and score import
list_models.py shows model IDs your keys can call
keys.sh API keys (git-ignored)
results/ one JSONL file per run, plus summary CSVs
```
The full workflow, from a clean machine, takes seven commands:
```
source .venv/bin/activate && source keys.sh
python list_models.py argon # find IDs, edit config.toml
python make_prompts.py # builds prompts.jsonl
python runner.py --models mock_good,mock_sloppy --run-id dry --day 2026-10-28
python runner.py --models terra,sol,luna,argon,opus --run-id live-1 --day 2026-10-28
python report.py results/live-1.jsonl
python blind.py export results/live-1.jsonl
```
Because the project is plain Python and TOML, it fits anywhere: a laptop, a CI job, a cron host. For more comparison and model coverage across the category, browse the [AI and machine learning section](https://shattered.io/category/ai-machine-learning/).
## FAQ: ChatGPT vs Claude vs Gemini API Testing
### Which is best, ChatGPT, Claude or Gemini?
No single model wins every task. The harness exists to answer the question for your prompts. Compare pass rate by category and cost per passed task, then choose per workload. Many teams end up routing different task types to different vendors.
### Does a ChatGPT Plus, Claude or Google AI subscription include API access?
No. Subscriptions and API billing are separate products. ChatGPT Plus is $20 per month and Pro now runs $100, $200 or $500, yet none of those include API credit. Create API keys and fund them in each vendor console.
### How many repeats and prompts do I need?
Start with 3 repeats and at least 10 tasks per category you care about. Raise repeats to 5 for tasks near the pass line. A dozen prompts show whether the pipeline works, but 50 or more per category are needed before a small difference between two models means anything.
### Why does the harness avoid the official SDKs?
Raw HTTP keeps every request field visible and removes version drift between three SDKs. Once you trust your numbers, switching an adapter to an SDK is a ten-minute change, because everything else only sees the normalized five-field result.
### How do I handle a model whose price changes mid-study?
Add a new price row with a start date, as the Gemini 4 Argon entry does for November 30. The runner prices each call from the row in force on the run date and stores that row’s date in the result, so old and new studies stay comparable.
### Can I test Gemini 4 Carbon or other restricted models?
Only if your key can call them. Restricted and trusted-tester models are better left out of production comparisons, because you cannot rely on access or a stable price. Add them as their own config block once Google opens them.
### Is it safe to run model-written code in the coding checks?
The subprocess runs in isolated mode with a ten-second timeout, which stops loops but not malicious file access. Use a disposable container or VM for any run where the prompt set includes content you did not write.
### How often should I rerun the benchmark?
Monthly at minimum, and immediately after any vendor announces a new model or price. The October 2026 launches from OpenAI, Google and Anthropic arrived within weeks of each other, so a quarterly cadence would have missed the changes that matter.