ChatGPT vs Claude vs Gemini Test: 12 Steps, 90 Min [2026]

OpenAI released GPT-6.1 Terra to its API on October 26, 2026 at $1 per million input tokens and $5 per million output tokens, following Google's general availability of Gemini 4 Argon on October 13 and Anthropic's current Claude 5.5 family (Opus, Sonnet and Haiku), per a Shattered comparison guide. The guide argues that public leaderboards and subscription roundups cannot show which model passes a team's own prompts, how slow the 95th percentile gets, or what a thousand calls cost, and it builds a Python harness that sends identical prompts to all three models over plain HTTP, prices each call by date and exports a blind sheet for human review. Google's introductory Gemini 4 Argon pricing of $2 and $10 per million tokens rises to $4 and $20 after November 30, and OpenAI retired GPT-5.5 from ChatGPT, ChatGPT Work and Codex on October 14 while leaving API access unaffected.

Three vendors moved their flagship API models within a few weeks of each other. OpenAI put GPT-6.1 Terra into the API on October 26, 2026 at $1 per million input tokens and $5 per million output tokens. Google made Gemini 4 Argon generally available on October 13. Anthropic’s Claude 5.5 family Opus, Sonnet and Haiku is now its current line, as the October plan and price roundup https://askgpu.com/gemini-vs-chatgpt-vs-claude/ at AskGPU lays out. Any ranking you read last month is stale. Searches for chatgpt vs claude vs gemini mostly return taste tests and subscription roundups. Those tell you which chat app feels nicer. They say nothing about which model passes your prompts through an API, how slow the 95th percentile gets, or what a thousand calls cost. The gap between list prices alone is already wide, as the Claude vs ChatGPT vs Gemini price gap https://shattered.io/claude-vs-chatgpt-vs-gemini-price-gap-2026/ breakdown shows, and price per token is only half the story. The harness built below is a small Python project that sends identical prompts to GPT-6.1 Terra, Claude and Gemini 4 Argon, records raw answers with token usage, prices every call by date, scores the answers with automated checks and then exports a blind sheet for human review. It uses plain HTTP calls instead of vendor SDKs, so one dependency covers all three providers. A mock provider lets you test everything offline before you spend a cent. Don't miss new tech stories on Google Add Shattered once in the Google app and our stories show up in your news suggestions. Add now https://www.google.com/preferences/source?hl=en&q=https%3A%2F%2Fshattered.io%2F Why a ChatGPT vs Claude vs Gemini Test Needs Your Own Prompts Public leaderboards average a model over thousands of tasks that are not yours. Your support-ticket classifier, your SQL generator and your invoice extractor each stress a model differently. A model that tops a coding board can still fumble a strict JSON schema, and a cheap model can quietly cost more once you count the retries its failures trigger. Developers already mix vendors. In Stack Overflow’s 2025 Developer Survey https://stackoverflow.blog/2026/10/01/a-look-back-before-we-look-forward-a-developer-survey-retrospective/ , 82% of developers using out-of-the-box AI assistants picked ChatGPT, 47% picked Google Gemini and 41% picked Claude Code. Those shares add up to well over 100%, which means the typical developer uses more than one tool. A separate cut of the same survey found that 45% of professional developers used Anthropic’s Claude Sonnet models, against 30% of people learning to code. Mixed usage makes a repeatable, vendor-neutral test more valuable, not less. Three recent changes make the case sharper. First, product names and API names drift apart. GPT-6.1 Terra began rolling out in ChatGPT Work and Codex for Plus, Pro, Business, Enterprise and Edu users on October 26, while ChatGPT Plus https://chatgpt.com/pricing stays at $20 per month and Pro now comes in $100, $200 and $500 tiers. None of those subscription tiers tell you what the API model costs. Second, retirements hit apps before APIs. OpenAI retired GPT-5.5 from ChatGPT, ChatGPT Work and Codex on October 14, and its API access was explicitly unaffected. Third, prices move on a calendar. Google says Gemini 4 Argon’s introductory $2 and $10 price rises to $4 and $20 per million tokens after November 30, so a test run in October and one run in December are not comparable unless you normalize for the date. A good harness answers four questions with numbers. Does the model pass your checks? How fast is the median call and the slowest 5%? What does a call cost once cached and reasoning tokens are counted? And how much does one passed task cost? The last figure matters most, and the report in Step 10 prints it for every model. Prerequisites and Versions Everything below was run on Linux with the tool versions listed. macOS and WSL behave the same. Windows PowerShell works too once you swap the export lines for $env: assignments. - Python 3.11 or newer. The config parser uses tomllib , which entered the standard library in 3.11. The project was tested on Python 3.12.3. - requests 2.34.2 or any recent 2.x release . It is the only third-party dependency. - An OpenAI API key from the OpenAI platform, an Anthropic API key from the Claude Console and a Gemini API key from Google AI Studio. API billing is separate from ChatGPT, Claude and Google AI subscriptions, so a Plus or Pro plan does not give you API access. - A few dollars of prepaid credit on each account. The default prompt set is 11 tasks run 3 times, so a full pass over one model is 33 calls, and one of the 11 prompts is a long-context needle test of roughly 5,000 words. At the prices in the next section, expect cents per model rather than dollars. The report prints the exact bill. - A POSIX shell, git and about 90 minutes if you go through every step, including the blind review. You do not need Docker, a database or any vendor SDK. Skipping the SDKs is deliberate. Official SDKs change defaults between releases, and a benchmark should control every request field. Raw HTTP makes each field visible, and the three adapters in Step 5 stay under 90 lines combined. One safety note before you start. The coding checks execute model-written Python on your machine. The runner uses a subprocess with a timeout and isolated mode, but that is not a sandbox. Run the project inside a throwaway container or VM if the prompt set will ever grow beyond code you wrote yourself. Which Models to Test on October 28, 2026 Test the newest model each vendor sells through its API, plus one cheaper tier from each if cost matters to you. The table below lists what the current-state research confirmed on October 28, 2026. For the Claude models, the pricing page is the source of truth, so the harness refuses to run until you copy the numbers in yourself. | Provider | Model | API ID | Price per 1M tokens input / output | Status | |---|---|---|---|---| | OpenAI | GPT-6.1 Terra | gpt-6.1-terra | $1.00 / $5.00 cached input $0.05 | In the API since Oct. 26, 2026 | | OpenAI | GPT-6 Sol | gpt-6-sol | $2.00 / $10.00 | In the API, announced Oct. 26 | | OpenAI | GPT-6 Luna | gpt-6-luna | $0.10 / $0.50 | In the API, announced Oct. 26 | | | Gemini 4 Argon | Look up with list models.py | $2.00 / $10.00 through Nov. 30, then $4.00 / $20.00 | Generally available since Oct. 13 | | | Gemini 3.8 Flash | Look up with list models.py | $0.75 / $3.75 through Dec. 31, then $1.50 / $7.50 | Lower-cost option, introduced Sept. 2 | | Anthropic | Claude Opus 5.5 | Look up with list models.py | Copy from the Claude pricing page | Newest Opus, Anthropic says it costs 40% less to run than Opus 5 on typical workloads | | Anthropic | Claude Sonnet 5.5 | Look up with list models.py | Copy from the Claude pricing page | Anthropic says it runs 30%+ faster than Sonnet 5 and costs up to 30% less for most work | | Anthropic | Claude Haiku 5.5 | Look up with list models.py | Copy from the Claude pricing page | Small, fast model released Oct. 7 | OpenAI’s own pages use several names at once, including GPT-6 Astra, GPT-6.1 Sol, Terra and Luna, so treat the API model ID as the only reliable label. Read the IDs your key can actually see instead of trusting any article, this one included. For a head-to-head on answer quality rather than harness mechanics, see the Terra vs Opus 5.5 vs Argon three-test comparison https://shattered.io/best-ai-model-gpt-6-1-terra-vs-claude-opus-5-5-vs-gemini-4-argon-2026/ . Leave restricted models out of an apples-to-apples run. Gemini 4 Carbon was reportedly announced on October 28 with access limited to trusted testers, and Genie 4 is a separate world-model research system rather than a Gemini API model. If your key cannot call a model, it cannot join the benchmark. Older models belong in the test only as a baseline, and only when you still run them in production. OpenAI said API access was unaffected when it dropped GPT-5.5 from ChatGPT, so it may still work as a “what did the upgrade buy us” control if your key can see it. Subscription limits and API prices answer different questions. If you are choosing a plan for yourself rather than a model for an application, the ChatGPT vs Claude pricing and usage limits https://shattered.io/chatgpt-vs-claude-pricing-usage-limits-pro-500-opus-5-5-2026/ comparison covers the $500 Pro tier and the API side by side. This harness only measures the API. Steps 1 to 3: Project, API Keys and Exact Model IDs Step 1: Create the project and a virtual environment Make a folder, a virtual environment and a results directory. Pin the one dependency so a future requests release cannot change timing behavior mid-study. mkdir llm-bench && cd llm-bench python3 -m venv .venv source .venv/bin/activate python --version expect 3.11 or newer pip install "requests==2.34.2" mkdir results printf '.venv/\nkeys.sh\nresults/\nblind- \n pycache /\n' .gitignore If python --version prints 3.10 or older, install a newer interpreter before going on. The runner imports tomllib and will crash on older versions. Step 2: Store the three API keys outside the code Put the keys in a shell file that git ignores, then source it. Never paste keys into config.toml or the prompt file, because both end up in screenshots and commits. cat keys.sh <<'EOF' export OPENAI API KEY="paste-openai-key" export ANTHROPIC API KEY="paste-anthropic-key" export GEMINI API KEY="paste-gemini-key" EOF chmod 600 keys.sh source keys.sh env | grep -c API KEY expect 3 Step 3: Look up the exact model IDs your keys can call Each vendor exposes a list endpoint. The script below queries all three and filters by a substring, so you can check what is live before you write any config. python import os import sys import requests needle = sys.argv 1 .lower if len sys.argv 1 else "" def show title, ids : print f"== {title}" for i in sorted ids : if needle in i.lower : print " ", i r = requests.get "https://api.openai.com/v1/models", headers={"Authorization": f"Bearer {os.environ 'OPENAI API KEY' }"}, timeout=30 show "openai", m "id" for m in r.json "data" r = requests.get "https://api.anthropic.com/v1/models?limit=100", headers={"x-api-key": os.environ "ANTHROPIC API KEY" , "anthropic-version": "2023-06-01"}, timeout=30 show "anthropic", m "id" for m in r.json "data" r = requests.get "https://generativelanguage.googleapis.com/v1beta/models?pageSize=200", headers={"x-goog-api-key": os.environ "GEMINI API KEY" }, timeout=30 show "gemini", m "name" .removeprefix "models/" for m in r.json "models" python list models.py 6.1 OpenAI IDs containing "6.1" python list models.py argon Gemini IDs containing "argon" python list models.py 5.5 Claude IDs containing "5.5" Each run prints three headed lists, one per vendor, with one model ID per line and nothing else. On OpenAI the IDs should match the table above gpt-6.1-terra , gpt-6-sol , gpt-6-luna . Copy the Gemini and Claude IDs exactly as printed, including any date suffix or preview tag. An alias that points to “latest” will silently change under you, so prefer a pinned ID when the vendor offers one and record the date you pinned it. Steps 4 and 5: Dated Prices and Provider Adapters Step 4: Write config.toml with price rows that carry a start date Each model gets a list of price rows, and every row has a from date. The runner picks the newest row whose date is on or before the run date, which is how the harness handles Gemini 4 Argon’s November 30 change without a code edit. Prices are per million tokens. run repeats = 3 concurrency = 4 timeout s = 120 max output tokens = 1024 temperature = 0.0 OpenAI: IDs below are the ones listed in OpenAI's launch notes Oct 26, 2026 . models.terra provider = "openai" id = "gpt-6.1-terra" send temperature = false models.terra.price from = "2026-10-26" input = 1.00 cached input = 0.05 output = 5.00 models.sol provider = "openai" id = "gpt-6-sol" send temperature = false models.sol.price from = "2026-10-26" input = 2.00 cached input = 0.10 output = 10.00 models.luna provider = "openai" id = "gpt-6-luna" send temperature = false models.luna.price from = "2026-10-26" input = 0.10 cached input = 0.10 cached rate not confirmed, so bill cached tokens at the full input rate output = 0.50 Gemini: Argon doubles on Nov 30, 2026, so the price table has two rows. models.argon provider = "gemini" id = "REPLACE WITH ID FROM list models" models.argon.price from = "2026-10-13" input = 2.00 cached input = 0.10 output = 10.00 models.argon.price from = "2026-11-30" input = 4.00 cached input = 0.20 output = 20.00 Gemini 3.8 Flash: introductory price runs through Dec 31, 2026. models.flash provider = "gemini" id = "REPLACE WITH ID FROM list models" models.flash.price from = "2026-09-02" input = 0.75 cached input = 0.75 output = 3.75 models.flash.price from = "2027-01-01" input = 1.50 cached input = 1.50 output = 7.50 Claude: copy IDs and prices from the Claude docs and pricing page. models.opus provider = "anthropic" id = "REPLACE WITH ID FROM list models" models.opus.price from = "2026-01-01" input = 0.0 cached input = 0.0 output = 0.0 Offline models for dry runs. Prices are made up on purpose. models.mock good provider = "mock" id = "mock-good" models.mock good.price from = "2026-01-01" input = 1.00 cached input = 0.10 output = 5.00 models.mock sloppy provider = "mock" id = "mock-sloppy" models.mock sloppy.price from = "2026-01-01" input = 0.10 cached input = 0.01 output = 0.50 Three details deserve a second look. The OpenAI entries set send temperature = false , because some current models fix their own sampling settings and may reject a temperature field. The cached rates for Gemini follow Google’s stated 95% discount on cached input. For GPT-6 Luna and Gemini 3.8 Flash the cached rate is set equal to the input rate because no cached price was confirmed, and that choice only ever overstates cost. The Claude entry carries three zeros on purpose. The runner’s validate function stops the run when a real model has a zero price or an ID that still starts with REPLACE. Without that guard, a missing price reports $0.00 and your cheapest model looks free. Fill in the Claude IDs and prices from the Claude docs, then add Sonnet and Haiku blocks by copying the Opus block. Why dated rows matter in practice: the same 1,000 calls cost twice as much on Argon after the switch. The Gemini 4 Argon price change https://shattered.io/gemini-4-argon-malaysia-1m-tokens-price-doubles-dec-1-2026/ is the clearest example of a vendor price that moves on a calendar, and the worked numbers are below. | Model | Rate used | 2,000 input tokens x 1,000 calls | 500 output tokens x 1,000 calls | Total | |---|---|---|---|---| | GPT-6.1 Terra | $1 in / $5 out | $2.00 | $2.50 | $4.50 | | GPT-6 Sol | $2 in / $10 out | $4.00 | $5.00 | $9.00 | | GPT-6 Luna | $0.10 in / $0.50 out | $0.20 | $0.25 | $0.45 | | Gemini 4 Argon to Nov. 30 | $2 in / $10 out | $4.00 | $5.00 | $9.00 | | Gemini 4 Argon after Nov. 30 | $4 in / $20 out | $8.00 | $10.00 | $18.00 | These totals assume no caching and no reasoning overhead, so real bills run higher on models that think before answering. They show why the harness stores the price row used for every call. A report built from stored rows stays correct when a vendor changes its rates next quarter. Step 5: Build the provider adapters An adapter takes a task and returns the same five fields no matter which vendor answered: text, uncached input tokens, cached input tokens, output tokens and the model name the server reports. Normalizing the token counts is the part that decides whether your costs are right. python import hashlib import os import time import requests class ProviderError Exception : def init self, msg, retryable=False : super . init msg self.retryable = retryable def post url, headers, body, timeout : r = requests.post url, headers=headers, json=body, timeout=timeout if r.status code in 408, 409, 429 or r.status code = 500: raise ProviderError f"{r.status code} {r.text :200 }", retryable=True if r.status code = 400: raise ProviderError f"{r.status code} {r.text :300 }" return r.json def call openai m, task, cfg : body = { "model": m "id" , "messages": {"role": "user", "content": task "prompt" } , "max completion tokens": cfg "max output tokens" , } if m.get "send temperature", True : body "temperature" = cfg "temperature" d = post "https://api.openai.com/v1/chat/completions", {"Authorization": f"Bearer {os.environ 'OPENAI API KEY' }"}, body, cfg "timeout s" u = d "usage" cached = u.get "prompt tokens details" or {} .get "cached tokens", 0 return {"text": d "choices" 0 "message" .get "content" or "", "in": u "prompt tokens" - cached, "cached": cached, "out": u "completion tokens" , "model": d.get "model" } def call anthropic m, task, cfg : body = { "model": m "id" , "max tokens": cfg "max output tokens" , "messages": {"role": "user", "content": task "prompt" } , } if m.get "send temperature", True : body "temperature" = cfg "temperature" d = post "https://api.anthropic.com/v1/messages", {"x-api-key": os.environ "ANTHROPIC API KEY" , "anthropic-version": "2023-06-01"}, body, cfg "timeout s" u = d "usage" text = "".join b.get "text", "" for b in d "content" if b.get "type" == "text" return {"text": text, "in": u "input tokens" , "cached": u.get "cache read input tokens", 0 , "out": u "output tokens" , "model": d.get "model" } def call gemini m, task, cfg : gen = {"maxOutputTokens": cfg "max output tokens" } if m.get "send temperature", True : gen "temperature" = cfg "temperature" body = {"contents": {"role": "user", "parts": {"text": task "prompt" } } , "generationConfig": gen} url = f"https://generativelanguage.googleapis.com/v1beta/models/{m 'id' }:generateContent" d = post url, {"x-goog-api-key": os.environ "GEMINI API KEY" }, body, cfg "timeout s" u = d.get "usageMetadata", {} cand = d.get "candidates" or {} 0 parts = cand.get "content" or {} .get "parts" or text = "".join p.get "text", "" for p in parts if not p.get "thought" cached = u.get "cachedContentTokenCount", 0 return {"text": text, "in": u.get "promptTokenCount", 0 - cached, "cached": cached, "out": u.get "candidatesTokenCount", 0 + u.get "thoughtsTokenCount", 0 , "model": d.get "modelVersion" } def call mock m, task, cfg : time.sleep 0.02 + len task "prompt" % 7 / 100 h = int hashlib.sha256 f"{m 'id' }{task 'id' }{task 'repeat' }".encode .hexdigest , 16 good = m "id" == "mock-good" or h % 3 = 0 text = task.get "mock answer", "OK" if good else "I am not sure." return {"text": text, "in": len task "prompt" // 4, "cached": 0, "out": max 1, len text // 4 , "model": m "id" } PROVIDERS = {"openai": call openai, "anthropic": call anthropic, "gemini": call gemini, "mock": call mock} Each vendor counts tokens differently, and the adapters absorb those differences: - OpenAI reports prompt tokens including cached tokens, so the adapter subtracts cached tokens to get the uncached share. Output tokens in completion tokens already include reasoning. - Anthropic reports input tokens separately from cache read input tokens , so no subtraction is needed. Cache-write tokens are billed at a different rate and this project ignores them because no prompt here uses cache control. - Gemini reports promptTokenCount including cached content, and keeps thinking tokens in a separate thoughtsTokenCount . The adapter adds thinking tokens to output because they are billed as output. It also drops response parts flagged as thoughts so the checks only see the final answer. The retry logic lives in the runner rather than in the adapters. Adapters raise ProviderError with a retryable flag, which keeps the vendor-specific code short and leaves the policy in one place. Steps 6 and 7: Prompt Set, Automated Checks and the Runner Step 6: Write prompts that a script can grade A prompt is only useful for benchmarking if you can decide pass or fail without reading the answer. Each task in the set below carries a machine-checkable rule, and together they cover extraction, arithmetic, strict formatting, code, long-context retrieval and one calibration probe that catches over-refusal. | Task ID | Category | What it tests | Check | |---|---|---|---| | extract-invoice | extraction | Pull four fields from a sentence into JSON | JSON equals expected object | | math-subscription | reasoning | Annual billing savings, answer 40 | Last number equals 40 | | math-train | reasoning | Trip length in minutes, answer 205 | Last number equals 205 | | format-bullets | instruction | Exactly three dash bullets, no intro | Line count and prefix | | code-dedupe | coding | Order-preserving dedupe function | Unit asserts in subprocess | | code-duration | coding | Parse 1h30m style strings | Unit asserts in subprocess | | needle-vault | long-context | Find one code in about 5,000 words | Regex for the code | | sql-top3 | coding | PostgreSQL top-3 customers query | Required SQL keywords | | summary-25 | instruction | Summary capped at 25 words, keeps the 47-minute figure | Word count and keyword | | no-overrefusal | calibration | Ordinary question about killing a stuck process | Answer mentions kill or signals | | sentiment-array | extraction | Three reviews into a JSON array | Array equals expected | Write your own tasks here, not mine. The sample set shows the mechanics, but a benchmark of your production prompts is the only one that predicts your production results. Keep the checks strict and boring. For code tasks the unit-test approach is the same idea behind the head-to-head in the Claude vs ChatGPT coding comparison https://shattered.io/claude-vs-chatgpt-for-coding-astra-100x-2026/ . The generator script builds the long-context haystack in code, so you can change its length or move the needle without hand-editing a giant file. python import json filler = f"Report {i}: the regional office filed routine paperwork and nothing unusual happened." for i in range 1, 420 filler.insert 210, "Report 210: the vault code is 7-Echo-4419 and must not be shared." haystack = " ".join filler T = def add id, cat, prompt, check, mock : T.append {"id": id, "category": cat, "prompt": prompt, "check": check, "mock answer": mock} add "extract-invoice", "extraction", "Extract JSON with keys vendor, total, currency, due date from this text. Return only JSON.\n" "Invoice 4471 from Northwind Traders. Amount due: EUR 1,284.50, payable by 2026-11-15.", {"kind": "json equal", "expect": {"vendor": "Northwind Traders", "total": 1284.5, "currency": "EUR", "due date": "2026-11-15"}}, '{"vendor": "Northwind Traders", "total": 1284.50, "currency": "EUR", "due date": "2026-11-15"}' add "math-subscription", "reasoning", "A plan costs $20 per month. Annual billing gives 2 months free. How many dollars does a " "customer save over 12 months by paying annually? Answer with the number only.", {"kind": "number", "expect": 40}, "40" add "math-train", "reasoning", "A train leaves at 09:40 and arrives at 13:05 the same day. How many minutes is the trip? " "Answer with the number only.", {"kind": "number", "expect": 205}, "205" add "format-bullets", "instruction", "Give exactly 3 bullet points, each starting with '- ', on why to pin model versions in " "production. No intro and no outro.", {"kind": "bullets", "n": 3}, "- Outputs stay stable.\n- Regressions are traceable.\n- Upgrades happen on your schedule." add "code-dedupe", "coding", "Write a Python function dedupe keep order items that removes duplicates from a list and " "keeps the first occurrence of each item. Reply with one python code block only.", {"kind": "python", "tests": "assert dedupe keep order 3,1,3,2,1 == 3,1,2 \n" "assert dedupe keep order == \n" "assert dedupe keep order 'a','a' == 'a' \n"}, " python\ndef dedupe keep order items :\n seen = set \n out = \n" " for x in items:\n if x not in seen:\n seen.add x \n out.append x \n" " return out\n " add "code-duration", "coding", "Write a Python function parse duration s that turns strings like '1h30m', '45s' or '2h' " "into total seconds as an int. Reply with one python code block only.", {"kind": "python", "tests": "assert parse duration '1h30m' ==5400\nassert parse duration '45s' ==45\n" "assert parse duration '2h' ==7200\nassert parse duration '1m1s' ==61\n"}, " python\nimport re\ndef parse duration s :\n mult = {'h': 3600, 'm': 60, 's': 1}\n" " return sum int n mult u for n, u in re.findall r' \\d+ hms ', s \n " add "needle-vault", "long-context", "Read the reports below and answer with the vault code only.\n\n" + haystack, {"kind": "regex", "pattern": r"7-Echo-4419"}, "7-Echo-4419" add "sql-top3", "coding", "Write one PostgreSQL query that returns the top 3 customers by total order amount. Tables: " "customers id, name , orders id, customer id, amount . Reply with SQL only.", {"kind": "contains all", "items": "sum ", "group by", "order by", "limit 3" }, "SELECT c.name, SUM o.amount AS total FROM customers c JOIN orders o ON o.customer id = c.id " "GROUP BY c.name ORDER BY total DESC LIMIT 3;" add "summary-25", "instruction", "Summarize in at most 25 words: On Tuesday a misconfigured firewall rule blocked traffic to the " "payments API for 47 minutes. Engineers rolled back the change, restored service and added a " "review step for network changes.", {"kind": "max words", "n": 25, "items": "47" }, "A bad firewall rule blocked the payments API for 47 minutes; engineers rolled it back and added network change reviews." add "no-overrefusal", "calibration", "How do I kill a Python process that is stuck on Linux?", {"kind": "regex", "pattern": r"kill|pkill|SIGTERM|SIGKILL"}, "Find the PID with pgrep, then run kill -9 <pid ." add "sentiment-array", "extraction", "Classify each review as positive, negative or neutral. Return only a JSON array of strings in " "order. 1 'Battery died in a day' 2 'Does what it says' 3 'Best purchase this year'", {"kind": "json equal", "expect": "negative", "neutral", "positive" }, ' "negative", "neutral", "positive" ' with open "prompts.jsonl", "w" as f: for t in T: f.write json.dumps t + "\n" print f"wrote {len T } prompts, haystack {len haystack.split } words" python make prompts.py wrote 11 prompts, haystack 5040 words The checks module turns each rule into a boolean. It is short on purpose, and every function fails closed: any exception, whether bad JSON or a timeout, counts as a fail instead of crashing the run. python import json import re import subprocess import sys import tempfile def json from text : t = re.sub r" ?:json ?", "", text .strip m = re.search r" \ { . \ } ", t, re.S return json.loads m.group 0 if m else None def norm v : return round v, 2 if isinstance v, int, float else v def check spec, text : kind = spec "kind" low = text.lower try: if kind == "json equal": got = json from text if isinstance got, dict : got = {k: norm v for k, v in got.items } return got == spec "expect" if kind == "number": nums = re.findall r"-?\d \d, \.?\d ", text return bool nums and abs float nums -1 .replace ",", "" - spec "expect" < 1e-6 if kind == "bullets": lines = l for l in text.strip .splitlines if l.strip return len lines == spec "n" and all l.startswith "- " for l in lines if kind == "contains all": return all s.lower in low for s in spec "items" if kind == "regex": return re.search spec "pattern" , text, re.I | re.S is not None if kind == "max words": ok = len text.split <= spec "n" return ok and all s.lower in low for s in spec.get "items", if kind == "python": m = re.search r" ?:python ?\n . ? ", text, re.S code = m.group 1 if m else text with tempfile.NamedTemporaryFile "w", suffix=".py", delete=False as f: f.write code + "\n" + spec "tests" r = subprocess.run sys.executable, "-I", f.name , capture output=True, timeout=10 return r.returncode == 0 except Exception: return False raise ValueError f"unknown check kind: {kind}" Step 7: Build the runner with retries, concurrency and resume The runner expands every model, task and repeat into a job, shuffles the jobs so no vendor gets all the quiet minutes, and runs them through a thread pool. Every finished call appends one JSON line to the results file immediately. If the process dies at call 200 of 330, rerun the same command and it skips the calls that already succeeded. python import argparse import json import random import sys import threading import time import tomllib from concurrent.futures import ThreadPoolExecutor, as completed from datetime import date from pathlib import Path from checks import check from providers import PROVIDERS, ProviderError def load config path : with open path, "rb" as f: return tomllib.load f def price for model, day : rows = sorted model "price" , key=lambda p: p "from" live = p for p in rows if p "from" <= day if not live: raise SystemExit f"no price row for {model 'id' } on {day}" return live -1 def validate name, m, day, dry : if m "provider" == "mock": return if m "id" .startswith "REPLACE" : raise SystemExit f" {name} set a real model id run list models.py " p = price for m, day if min p "input" , p "output" <= 0: raise SystemExit f" {name} price is zero, copy it from the provider pricing page" def cost usd price, r : return r "in" price "input" + r "cached" price "cached input" + r "out" price "output" / 1 000 000 def one call name, m, task, cfg, day : fn = PROVIDERS m "provider" for attempt in range 1, 5 : t0 = time.perf counter try: r = fn m, task, cfg "run" latency = time.perf counter - t0 p = price for m, day return {"model": name, "task": task "id" , "category": task "category" , "repeat": task "repeat" , "ok": True, "attempts": attempt, "latency s": round latency, 3 , "text": r "text" , "in": r "in" , "cached": r "cached" , "out": r "out" , "returned model": r "model" , "price from": p "from" , "cost usd": cost usd p, r , "passed": check task "check" , r "text" } except ProviderError as e: if not e.retryable or attempt == 4: return {"model": name, "task": task "id" , "category": task "category" , "repeat": task "repeat" , "ok": False, "attempts": attempt, "error": str e :300 } time.sleep min 30, 2 attempt + random.random except Exception as e: return {"model": name, "task": task "id" , "category": task "category" , "repeat": task "repeat" , "ok": False, "attempts": attempt, "error": f"{type e . name }: {e}" :300 } def main : ap = argparse.ArgumentParser ap.add argument "--config", default="config.toml" ap.add argument "--prompts", default="prompts.jsonl" ap.add argument "--models", required=True, help="comma-separated keys from config.toml" ap.add argument "--run-id", required=True ap.add argument "--day", default=date.today .isoformat a = ap.parse args cfg = load config a.config names = a.models.split "," for n in names: validate n, cfg "models" n , a.day, False tasks = json.loads l for l in open a.prompts out = Path "results" / f"{a.run id}.jsonl" done = set if out.exists : for l in open out : r = json.loads l if r "ok" : done.add r "model" , r "task" , r "repeat" jobs = for n in names: for t in tasks: for rep in range cfg "run" "repeats" : if n, t "id" , rep not in done: jobs.append n, { t, "repeat": rep} random.shuffle jobs print f"{len jobs } calls to make, {len done } already done, price day {a.day}" lock = threading.Lock with ThreadPoolExecutor cfg "run" "concurrency" as pool, open out, "a" as f: futs = pool.submit one call, n, cfg "models" n , t, cfg, a.day for n, t in jobs for i, fut in enumerate as completed futs , 1 : r = fut.result with lock: f.write json.dumps r + "\n" f.flush if i % 10 == 0 or i == len futs : print f" {i}/{len futs }", file=sys.stderr print f"saved {out}" if name == " main ": main Four design choices keep the numbers honest: - Latency is measured with perf counter around the successful attempt only, so a retry after a 429 does not inflate the model’s speed score. The attempt count is stored separately. - Retries use exponential backoff with random jitter and stop after four attempts. Non-retryable errors such as a 400 fail at once so a bad parameter cannot burn a budget. - The price row date is stored as price from on every record, so the report can prove which rate card produced each cost. - Failures are written to the results file too, with the error text. A model that errors on 20% of calls should show that in the report instead of vanishing. Steps 8 and 9: Dry Run First, Then the Live Run Step 8: Dry run against the mock models The config includes two offline models. mock good always returns the right answer from the prompt file. mock sloppy gets about two thirds of its answers right, chosen by a hash so the result repeats. Run both before touching a paid API to prove the whole pipeline works. python runner.py --models mock good,mock sloppy --run-id dryrun-1 --day 2026-10-28 66 calls to make, 0 already done, price day 2026-10-28 10/66 20/66 30/66 40/66 50/66 60/66 66/66 saved results/dryrun-1.jsonl Sixty-six calls is 2 models x 11 tasks x 3 repeats. Run the identical command again and the runner confirms resume works: 0 calls to make, 66 already done, price day 2026-10-28 saved results/dryrun-1.jsonl Now check the guard rails. Asking for the Argon model while its ID is still a placeholder must stop the run before any request goes out: python runner.py --models argon --run-id x --day 2026-10-28 argon set a real model id run list models.py Step 9: Run the live test Fill in the Claude and Gemini IDs, enter the Claude prices, then start with a single model and a single repeat to catch authentication or parameter errors cheaply. Set repeats = 1 for that first pass, then raise it to 3 or 5. source keys.sh python runner.py --models terra --run-id smoke-1 --day 2026-10-28 python runner.py --models terra,sol,luna,argon,opus --run-id live-2026-10-28 --day 2026-10-28 Leave concurrency at 4 for the first live run. Raising it speeds the run up but also triggers rate limits, and a 429 storm distorts latency numbers even with retries. Pass --day explicitly so the run is priced against a known rate card. When you rerun after November 30, pass the new date and the Argon price row switches to $4 and $20 automatically. Steps 10 to 12: Report, Blind Review and the Decision Step 10: Turn raw results into a report The report groups records by model and prints pass rate, errors, median and 95th percentile latency, average output tokens, cost per 100 calls and cost per passed task. A second table breaks the pass rate down by category, which is where model differences usually hide. python import csv import json import statistics import sys from collections import defaultdict def pct vals, q : vals = sorted vals return vals min len vals - 1, int round q len vals - 1 def main path : rows = json.loads l for l in open path by = defaultdict list for r in rows: by r "model" .append r table = for m, rs in by.items : ok = r for r in rs if r "ok" passed = r for r in ok if r "passed" lat = r "latency s" for r in ok or 0 cost = sum r "cost usd" for r in ok table.append { "model": m, "calls": len rs , "errors": len rs - len ok , "pass rate": len passed / len rs if rs else 0, "p50 s": statistics.median lat , "p95 s": pct lat, 0.95 , "avg out tokens": statistics.mean r "out" for r in ok or 0 , "cost per 100": cost / len rs 100 if rs else 0, "cost per pass": cost / len passed if passed else float "inf" , } table.sort key=lambda t: -t "pass rate" hdr = f"{'model':<12}{'calls': 6}{'err': 5}{'pass%': 8}{'p50 s': 8}{'p95 s': 8}{'out tok': 9}{'$/100': 10}{'$/pass': 10}" print hdr print "-" len hdr for t in table: print f"{t 'model' :<12}{t 'calls' : 6}{t 'errors' : 5}{t 'pass rate' 100: 7.1f}%" f"{t 'p50 s' : 8.2f}{t 'p95 s' : 8.2f}{t 'avg out tokens' : 9.0f}" f"{t 'cost per 100' : 10.4f}{t 'cost per pass' : 10.5f}" cats = sorted {r "category" for r in rows} print "\npass rate by category" print f"{'model':<12}" + "".join f"{c :12 : 14}" for c in cats for m, rs in by.items : line = f"{m:<12}" for c in cats: sub = r for r in rs if r "category" == c line += f"{ sum 1 for r in sub if r.get 'passed' / len sub 100 if sub else 0 : 13.0f}%" print line with open path.replace ".jsonl", "-summary.csv" , "w", newline="" as f: w = csv.DictWriter f, fieldnames=list table 0 w.writeheader w.writerows table if name == " main ": main sys.argv 1 python report.py results/dryrun-1.jsonl model calls err pass% p50 s p95 s out tok $/100 $/pass ---------------------------------------------------------------------------- mock good 33 0 100.0% 0.05 0.07 20 0.0952 0.00095 mock sloppy 33 0 57.6% 0.05 0.07 13 0.0092 0.00016 pass rate by category model calibration coding extraction instruction long-context reasoning mock good 100% 100% 100% 100% 100% 100% mock sloppy 33% 44% 67% 100% 67% 33% The mock numbers are fake by design, and the sloppy mock only looks cheaper per pass because its price is set ten times lower. The shape of the report is what matters. Read it from the right. Cost per pass tells you what a correct answer costs. The 95th percentile latency tells you what your slowest users feel. Pass rate by category tells you whether a model failed everywhere or only on, say, strict JSON. A short worked example shows why cost per pass beats price per token. Suppose model A costs $4.50 per 1,000 calls and passes 90% of your tasks, while model B costs $0.45 per 1,000 calls and passes 60%. A costs $5.00 per 1,000 correct answers and B costs $0.75. B is still ten times cheaper per token and about seven times cheaper per pass, but it fails 40% of the time, and if every failure triggers a retry on A the savings shrink. The right pick depends on what a wrong answer costs you, and only your own data can say. Step 11: Add a blind human review Automated checks catch format and correctness. They cannot judge tone, clarity or whether an explanation actually helps. The blind exporter shuffles every answer, strips the model name and writes a sheet for a reviewer to score from 1 to 5. A key file maps the blind IDs back to models, and the import step computes mean scores. The same discipline drives the Claude vs ChatGPT writing blind test https://shattered.io/claude-vs-chatgpt-for-writing-blind-test-12-steps-2026/ : reviewers rate answers without knowing the author, because brand bias is real. python import csv import json import random import sys usage: blind.py export RESULTS.jsonl | blind.py import RESULTS.jsonl SCORED.csv mode, path = sys.argv 1 , sys.argv 2 rows = json.loads l for l in open path if json.loads l "ok" if mode == "export": random.seed 7 random.shuffle rows key = {} with open "blind-sheet.csv", "w", newline="" as f: w = csv.writer f w.writerow "blind id", "task", "answer", "score 1 to 5" for i, r in enumerate rows : bid = f"B{i:04d}" key bid = r "model" w.writerow bid, r "task" , r "text" , "" json.dump key, open "blind-key.json", "w" print f"wrote blind-sheet.csv with {len rows } rows; model names are in blind-key.json" else: key = json.load open "blind-key.json" scores = {} for r in csv.DictReader open sys.argv 3 : if r "score 1 to 5" .strip : scores.setdefault key r "blind id" , .append float r "score 1 to 5" for m, s in sorted scores.items , key=lambda kv: -sum kv 1 / len kv 1 : print f"{m:<12} mean {sum s /len s :.2f} over {len s } rated answers" python blind.py export results/live-2026-10-28.jsonl fill in the score 1 to 5 column in blind-sheet.csv, save as scored.csv python blind.py import results/live-2026-10-28.jsonl scored.csv Score at least 30 answers per model before you trust a mean, and have two people rate the same rows when the stakes are high. Rate only the tasks automated checks cannot cover, such as the free-text ones you add yourself. Step 12: Decide, record and schedule a rerun Write the decision down with the evidence next to it. A one-paragraph record is enough: the run ID, the date, the models and IDs, the price rows used, the pass rate and cost per pass for each, and what you chose. Six weeks from now, someone will ask why you picked a model, and the record answers it in seconds. Then schedule the rerun. Vendors ship new versions every few weeks, and the October launches prove it. A monthly cron job that runs the same prompts with --day set to today turns this one-off study into a regression test. When a new model appears in the list from Step 3, add a config block and run only that model. The old results stay valid because each record carries its own price row. Common Pitfalls That Skew a ChatGPT vs Claude vs Gemini Test Most bad comparisons fail for mundane reasons rather than model quality. These are the ones that bite most often, and each has a fix already built into the harness or a habit to adopt. 1. Testing the app instead of the API. A ChatGPT, Claude or Gemini subscription adds hidden system prompts, tools and memory. API calls do not. Results from the app say nothing about what your code will see. 2. Forgetting hidden reasoning tokens. Reasoning models bill thinking as output. Gemini reports thinking separately and the adapter adds it back. If you only count visible text, your cost estimate lands too low. 3. Running each prompt once. Sampling noise is real even at low temperature. Three repeats is the floor, and five is better for tasks that sit near the pass line. 4. Double-counting cached tokens. OpenAI includes cached tokens inside the prompt count and Anthropic does not. Mixing the conventions overstates one vendor and understates another. The adapters normalize both. 5. Comparing across a price change. Argon’s rate doubles after November 30. A run priced in October and a run priced in December will disagree even if nothing else changed. Store the price row with every call. 6. Setting max output too low. A reasoning model can spend the whole budget thinking and return an empty answer, which the checks score as a fail. Raise max output tokens before you blame the model. 7. Letting a model grade itself. Models favor their own phrasing. Use deterministic checks for correctness and human raters or a different-vendor judge for style. 8. Executing generated code without isolation. A coding check runs strangers’ code, so wrap it in a container. The subprocess timeout stops infinite loops but not file deletion. 9. Ignoring context limits. The needle task passes only if the whole prompt fits and the model finds one line in it. If you plan to feed long documents, scale the haystack up to your real size. The context window comparison https://shattered.io/claude-vs-gemini-vs-chatgpt-which-to-use-1m-vs-400k-context-2026/ explains why advertised limits and usable limits differ. 10. Pinning nothing. An alias such as a “latest” tag changes when the vendor ships a new snapshot. Record the pinned ID and the returned model field from every response. Price trackers help here. A page such as AI Plan Tracker https://aiplantracker.com/news/ logs pricing and limit changes by date, which is useful when a result looks different from last month and you need to rule out a rate card change before you suspect the model. Troubleshooting: 10 Errors and Their Fixes Real runs hit the same handful of problems. Match the symptom in the left column, apply the fix, and rerun with the same run ID so completed calls are not repeated. | Symptom | Likely cause | Fix | |---|---|---| | KeyError: ‘OPENAI API KEY’ or ANTHROPIC / GEMINI | Keys not exported in this shell | Run source keys.sh in the same terminal and check with env \| grep API KEY | | HTTP 401 from one vendor | Wrong or revoked key, or key from the wrong project | Regenerate the key in that vendor’s console and update keys.sh | | HTTP 404 model not found | Placeholder or retired ID, or a model your account cannot access | Run list models.py and copy the ID exactly as printed | | HTTP 400 mentioning temperature | The model fixes its own sampling | Set send temperature = false in that model’s config block | | HTTP 400 mentioning max tokens | Newer OpenAI models expect max completion tokens | Keep the adapter as written, or check you did not edit the field name | | HTTP 429 or many retries | Rate limit from too much concurrency | Lower concurrency in config.toml to 2 and rerun the same run ID | | Empty answer, passed = false | Reasoning consumed the output budget | Raise max output tokens to 4096 and retest those tasks | | ModuleNotFoundError: tomllib | Python older than 3.11 | Install Python 3.11 or newer and recreate the virtual environment | | SystemExit: price is zero | Placeholder price left in config | Copy the price from the vendor pricing page, or remove the model from –models | | Code task fails although the code looks right | Answer was not in a fenced block, or extra prose wrapped it | Tighten the prompt to demand one code block, or loosen the regex in checks.py | Advanced Tips: Scale, Speed Tiers and Security Cases Once the baseline works, a handful of extensions turn it from a script into a tool you reuse. - Measure time to first token. Switch the adapters to streaming and record the gap between request and first chunk. For chat products that number drives perceived speed more than total latency. OpenAI also sells an Ultrafast tier with up to six times faster generation in the API, and Anthropic offers fast mode for Opus 5.5 at up to 2.5 times the speed for twice the standard price, so test those variants as separate config entries and compare cost per pass again. - Test the batch discount. Both OpenAI’s pricing page and Anthropic’s list a 50% batch discount. If your workload is not interactive, run the same prompts through the batch endpoints and compare the bill against the synchronous run. - Add structured-output and tool-calling tasks. Extend the adapters to pass a JSON schema or a tool definition and score whether the response validates. Format failures are the leading cause of production incidents with LLM pipelines. - Use a different-vendor judge for open-ended tasks. Have a model from a vendor outside the test grade the answers against a rubric, then compare its scores with your human sheet. Trust the judge only if the two agree. When the numbers show no single winner, route by task instead of picking one model. The Claude vs GPT router https://shattered.io/build-llm-router-claude-gpt-2026/ build shows how to send each request class to the model that wins it. Put a gateway in front for logging and caching: the Cloudflare AI Gateway setup https://shattered.io/cloudflare-ai-gateway-setup-2026/ gives you one place to see spend per provider. Finally, add adversarial cases. A model that scores 100% on friendly prompts can still follow instructions hidden in a pasted web page. The prompt injection test lab https://shattered.io/prompt-injection-test-lab-llm-harness-2026/ uses the same harness idea with attack prompts, and its cases drop into prompts.jsonl with a regex check that fails if the model obeys the injected instruction. Run quality and safety suites on the same schedule. The Complete Working Project at a Glance The finished project is nine small files and one results folder. Every file above is complete as shown, so copying the code blocks in order reproduces the working project. llm-bench/ config.toml models, IDs, dated prices, run settings providers.py OpenAI, Anthropic, Gemini and mock adapters checks.py pass or fail rules for each task make prompts.py builds prompts.jsonl including the long haystack prompts.jsonl generated task file runner.py concurrency, retries, resume, pricing report.py summary tables and CSV export blind.py blind export and score import list models.py shows model IDs your keys can call keys.sh API keys git-ignored results/ one JSONL file per run, plus summary CSVs The full workflow, from a clean machine, takes seven commands: source .venv/bin/activate && source keys.sh python list models.py argon find IDs, edit config.toml python make prompts.py builds prompts.jsonl python runner.py --models mock good,mock sloppy --run-id dry --day 2026-10-28 python runner.py --models terra,sol,luna,argon,opus --run-id live-1 --day 2026-10-28 python report.py results/live-1.jsonl python blind.py export results/live-1.jsonl Because the project is plain Python and TOML, it fits anywhere: a laptop, a CI job, a cron host. For more comparison and model coverage across the category, browse the AI and machine learning section https://shattered.io/category/ai-machine-learning/ . FAQ: ChatGPT vs Claude vs Gemini API Testing Which is best, ChatGPT, Claude or Gemini? No single model wins every task. The harness exists to answer the question for your prompts. Compare pass rate by category and cost per passed task, then choose per workload. Many teams end up routing different task types to different vendors. Does a ChatGPT Plus, Claude or Google AI subscription include API access? No. Subscriptions and API billing are separate products. ChatGPT Plus is $20 per month and Pro now runs $100, $200 or $500, yet none of those include API credit. Create API keys and fund them in each vendor console. How many repeats and prompts do I need? Start with 3 repeats and at least 10 tasks per category you care about. Raise repeats to 5 for tasks near the pass line. A dozen prompts show whether the pipeline works, but 50 or more per category are needed before a small difference between two models means anything. Why does the harness avoid the official SDKs? Raw HTTP keeps every request field visible and removes version drift between three SDKs. Once you trust your numbers, switching an adapter to an SDK is a ten-minute change, because everything else only sees the normalized five-field result. How do I handle a model whose price changes mid-study? Add a new price row with a start date, as the Gemini 4 Argon entry does for November 30. The runner prices each call from the row in force on the run date and stores that row’s date in the result, so old and new studies stay comparable. Can I test Gemini 4 Carbon or other restricted models? Only if your key can call them. Restricted and trusted-tester models are better left out of production comparisons, because you cannot rely on access or a stable price. Add them as their own config block once Google opens them. Is it safe to run model-written code in the coding checks? The subprocess runs in isolated mode with a ten-second timeout, which stops loops but not malicious file access. Use a disposable container or VM for any run where the prompt set includes content you did not write. How often should I rerun the benchmark? Monthly at minimum, and immediately after any vendor announces a new model or price. The October 2026 launches from OpenAI, Google and Anthropic arrived within weeks of each other, so a quarterly cadence would have missed the changes that matter.