GBench: DeepMind open-sources evaluation suite with 86 benchmark suites

Google DeepMind released gBench, an open-source benchmarking and evaluation suite for open large language models and vision-language models that bundles 86 canonical academic and domain eval suites across 6 capability pillars. gBench measures serving performance on the hardware it runs on and model capability in three tiers — academic evals, 16 deterministic Golden Set smoke tests, and the GemmaClaw/OpenClaw agentic quality suite with 60 QA scenarios pinned at commit d8bc6989 — and works with any OpenAI-compatible /v1 server via --remote-endpoint, validated on vLLM and Ollama. The suite also adds model discovery from HuggingFace config.json with GPU-tier sizing (1 GPU up to 20B parameters, 2 GPUs up to 80B, 8 GPUs above) and an open-loop stress sweep that certifies throughput limits to ±9% using request cohorts of at least 1099 probes.

gBench is an open-source benchmarking and evaluation suite for open large language models LLMs and vision-language models VLMs . It measures serving performance on the hardware it runs on, and model capability in three tiers: academic eval suites, the Golden Set smoke tests and the agentic quality suite. It works with any OpenAI-compatible /v1 server through --remote-endpoint validated on vLLM and Ollama; on Cloud Run, a service that allows unauthenticated calls , or starts and stops a local vLLM server itself. - OpenAI API compatibility: measure any OpenAI-compatible /v1 endpoint with --remote-endpoint <URL /v1 validated on vLLM and Ollama; on Cloud Run, a service that allows unauthenticated calls; see Remote Endpoint Mode https://github.com/google-deepmind/gbench/blob/main/docs/remote-endpoints.md remote-endpoint-mode---remote-endpoint . - Three capability tiers: 1. Academic and domain evals --evals : 86 canonical Python suites in --evals all plus ui control osworld and swe lancer , which run only when named across 6 core capability pillars General Knowledge & Scientific Reasoning , Mathematics & Proofs , Coding & Algorithmic Design , Long Context & Retrieval , Tool Use & Agentic Workflows , Multimodal Vision & Grounding with the model's reasoning mode on request --eval-thinking , images sent as their original bytes where the dataset stores them no re-encoding , plugin discovery --eval-plugins-dir and custom JSONL datasets --eval-custom-jsonl . 2. Golden Set smoke tests --golden-only : 16 deterministic capability checks code canonical , function call single , structured json , etc. with two-sided presence assertions and python exec unit-test matching, including a labelled chat template subset https://github.com/google-deepmind/gbench/blob/main/docs/golden-set.md chat-template-sensitivity-pairs that separates a broken chat template from a weak model. 3. Agentic quality suite --quality-only : the GemmaClaw / OpenClaw QA suite the QA scenarios of the pinned GemmaClaw commit, 60 at d8bc6989; scenarios that need another model provider are listed apart and not scored testing multi-turn tool calling, session memory and plugin routing, run in gBench's local gbench-quality Docker image built once per GemmaClaw commit . - Academic and domain evals - Model discovery and GPU sizing: model properties read from the HuggingFace config.json parameter count, MoE topology, vision/audio tokens , a recommended GPU tier per model size 1 GPU up to 20B, 2 GPUs up to 80B, 8 GPUs above that is checked against the required --num-gpus a warning when off the tier, a refusal when the model will not fit , and a server startup timeout that scales with model size max 600, min 3600, 300 + params b 10 3 1.3 if MoE . - Repeatability checks: several iterations per configuration with mean, median, standard deviation, P50, P95, P99 and the coefficient of variation, checked against --max-cv-percent default 5.0 . - Open-loop stress sweep --stress-test : one merged Poisson schedule per rate point at increasing rates x2 per step until a point fails; each of --stress-reps repeats gives a bracket pass, fail , and the median repeat's limit is then certified with probes sized for the P99 SLO a cohort of = 1099 requests, about 1400 sent, the same rule on every device to +-9% when that resolution is reached otherwise a certified lower bound with its resolution, a lower bound censored at --stress-max-qps , or no limit when certification refutes the sweep , or reported UNCERTIFIED with the time certifying would need --stress-certify-budget . A point passes when, on the requests planned after the point's ramp-up: the queue is not growing, = 90% finish within the end-to-end time the SLOs imply, P99 TTFT and P99 ITL meet their SLOs, failures stay under 2% and the client kept to its schedule. The queue-growth test reads TTFT; overload that shows only as longer end-to-end times for example KV-cache preemption is left to the completion-fraction and ITL gates. SLOs are per campaign 5000 ms TTFT and 200 ms ITL without --campaign , overridden by --stress-threshold and --itl-slo-ms . Using gBench - Installation https://github.com/google-deepmind/gbench/blob/main/docs/installation.md : requirements, a virtual environment or conda, and the optional installs per use. - CLI argument reference https://github.com/google-deepmind/gbench/blob/main/docs/cli-reference.md : every command-line flag. - Environment variables https://github.com/google-deepmind/gbench/blob/main/docs/environment-variables.md , and every per-suite knob in docs/evals/env-knobs.md https://github.com/google-deepmind/gbench/blob/main/docs/evals/env-knobs.md . - Remote endpoint mode https://github.com/google-deepmind/gbench/blob/main/docs/remote-endpoints.md : benchmarking any OpenAI-compatible /v1 server, plus the preflight, smoke and parallel-suite modes docs/smoke.md https://github.com/google-deepmind/gbench/blob/main/docs/smoke.md . - Results https://github.com/google-deepmind/gbench/blob/main/docs/results.md : output layout, result schemas, provenance and exit codes. - Distributed runs https://github.com/google-deepmind/gbench/blob/main/docs/distributed-gbench.md : splitting a sweep across nodes --shard and merging the results --merge . - Deployment modes https://github.com/google-deepmind/gbench/blob/main/docs/deployment.md : the CLI today; container, Kubernetes and hosted modes on the roadmap. What gBench measures - Performance benchmarks https://github.com/google-deepmind/gbench/blob/main/docs/performance.md : model discovery, GPU tiers and selection, workload campaigns, dataset backends and repeatability. - Academic and custom eval suites https://github.com/google-deepmind/gbench/blob/main/docs/eval-suites.md --evals , with one setup and scoring guide per suite in docs/evals/ https://github.com/google-deepmind/gbench/blob/main/docs/evals , plus toolchains.md https://github.com/google-deepmind/gbench/blob/main/docs/evals/toolchains.md optional dependencies , docker-sandboxes.md https://github.com/google-deepmind/gbench/blob/main/docs/evals/docker-sandboxes.md Docker address pools for the containerized suites and large-sweep-setup.md https://github.com/google-deepmind/gbench/blob/main/docs/evals/large-sweep-setup.md host limits for long sweeps . - Golden Set capability smoke test https://github.com/google-deepmind/gbench/blob/main/docs/golden-set.md --golden-only . - Agentic quality suite https://github.com/google-deepmind/gbench/blob/main/docs/quality.md --quality-only . Background - docs/THIRD PARTY DATASETS.md https://github.com/google-deepmind/gbench/blob/main/docs/THIRD PARTY DATASETS.md : the datasets the suites load, with their licenses, gated and non-commercial terms. Each dataset's license governs your use of it and of the scores computed on it; read it before publishing results. - docs/RESOURCES.md https://github.com/google-deepmind/gbench/blob/main/docs/RESOURCES.md : the upstream source of each suite. docs/GOSPEL.md https://github.com/google-deepmind/gbench/blob/main/docs/GOSPEL.md : what gBench is and who it is for. - Scaffold versioning https://github.com/google-deepmind/gbench/blob/main/docs/scaffold-versioning.md : what each pillar pins, so results stay comparable. Endpoint relay https://github.com/google-deepmind/gbench/blob/main/docs/relay.md : how agent harnesses that cannot be pointed at the endpoint reach the served model. Evals roadmap https://github.com/google-deepmind/gbench/blob/main/docs/evals roadmap.md : planned suites and the engineering playbook. - examples/notebooks/ https://github.com/google-deepmind/gbench/blob/main/examples/notebooks : five runnable notebooks Colab or local Jupyter, against Ollama or a vLLM endpoint; 05 needs Docker, so a local or cloud machine rather than Colab : 01 quickstart https://github.com/google-deepmind/gbench/blob/main/examples/notebooks/01 gbench 101 quickstart.ipynb serving latency and the stress limit , 02 workload campaigns and concurrency https://github.com/google-deepmind/gbench/blob/main/examples/notebooks/02 workloads campaigns and concurrency.ipynb , 03 Golden Set smoke tests https://github.com/google-deepmind/gbench/blob/main/examples/notebooks/03 golden set capability smoke tests.ipynb , 04 academic evals and reasoning mode https://github.com/google-deepmind/gbench/blob/main/examples/notebooks/04 academic evaluations and reasoning.ipynb and 05 agentic quality with GemmaClaw https://github.com/google-deepmind/gbench/blob/main/examples/notebooks/05 agentic quality with gemmaclaw.ipynb . - CONTRIBUTING.md https://github.com/google-deepmind/gbench/blob/main/CONTRIBUTING.md : how to contribute. git clone https://github.com/google-deepmind/gbench.git cd gbench python3 -m venv venv && source venv/bin/activate pip install -e . base package Python 3.11+ : laptops and remote endpoints such as Ollama optional, per use: pip install transformers --remote-endpoint: prompt sizing --tokenizer , Golden Set chat-template cases and the mmlu, wmdp, ruler and longbench v2 code suites pip install vllm local GPU serving, and remote real-dataset workloads --campaign chat-like, --dataset custom/hf ; gBench never installs vllm/torch itself pip install -e ". evals " the eval suites' client, scorer and harness dependencies The installation guide https://github.com/google-deepmind/gbench/blob/main/docs/installation.md covers the requirements, conda, the optional installs per suite and the test environment. 1. Quick local performance smoke test auto-starts local vLLM server gbench --models google/gemma-4-E4B-it --num-gpus 1 --preset quick 2. Run a specific workload performance campaign e.g. agentic long-context gbench --models google/gemma-4-12B-it --num-gpus 1 --campaign agentic 3. Run the Golden Set capability smoke tests on a local vLLM server gbench launches; its tool cases need the model's tool-call parser Gemma 4 example gbench --golden-only --models google/gemma-4-12B-it --num-gpus 1 --tool-call-parser gemma4 --reasoning-parser gemma4 4. Run GemmaClaw agentic quality suite over remote endpoint gbench --quality-only --models google/gemma-4-12B-it --remote-endpoint http://127.0.0.1:8000/v1 --tokenizer google/gemma-4-12B-it 5. Run academic evaluations with Chain-of-Thought reasoning against remote server gbench --evals-only --remote-endpoint http://127.0.0.1:8000/v1 --tokenizer google/gemma-4-E4B-it --evals gpqa mmlu --eval-thinking --max-output-tokens 65536 6. Stage model artifacts from HuggingFace to Google Cloud Storage gbench --models google/gemma-4-12B-it --stage-to-gcs gs://my-gbench-bucket/models/ 7. Dry run: the plan, each server's vllm serve command, port, GPUs and base URL validated gbench --models google/gemma-4-12B-it --num-gpus 1 --preset default --dry-run +-------------------------------------------------------------------------------+ | gbench capability evaluation | +-------------------------------------------------------------------------------+ | Tier 1: academic evals | Tier 2: Golden Set | Tier 3: agentic QA| | --evals | --golden-only | --quality-only | | - bfcl, gpqa, gsm8k, mmlu, | - 16 cases | - GemmaClaw suite | | mmmu pro, mrcr, screenspot | - Two-sided presence | - QA scenarios | | - Reasoning mode --thinking | - Chat template pairs | - Multi-turn QA | | - Vision soft-token budgets | - N/A-aware scoring | - Tools, memory | +---------------------------------+------------------------+--------------------+ | v +-------------------------------------------------------------------------------+ | OpenAI-compatible REST API /v1 | +-------------------------------------------------------------------------------+ | Mode A: local vLLM server <---- OR ---- Mode B: remote endpoint | | Auto start / stop / VRAM poll --remote-endpoint <URL /v1 | +-------------------------------------------------------------------------------+ The full catalog of the 86 suites in 6 pillars, with each suite's protocol and options, custom plugins and JSONL suites: docs/eval-suites.md https://github.com/google-deepmind/gbench/blob/main/docs/eval-suites.md . Each suite has a setup and scoring guide in docs/evals/ https://github.com/google-deepmind/gbench/blob/main/docs/evals . 16 deterministic capability checks code, math, tool calling, structured output, chat templates, factual knowledge, translation, safety, vision and audio with two-sided presence assertions and python exec unit-test matching, including a chat-template subset https://github.com/google-deepmind/gbench/blob/main/docs/golden-set.md chat-template-sensitivity-pairs that separates a broken chat template from a weak model. Details: docs/golden-set.md https://github.com/google-deepmind/gbench/blob/main/docs/golden-set.md . Evaluates multi-turn agent behavior with the GemmaClaw / OpenClaw QA suite: the QA scenarios of the pinned GemmaClaw commit 60 at d8bc6989 ; scenarios that need a model provider other than the model under test and the Gemini API are listed apart and not scored. - Image build : clones GemmaClaw ~/.cache/gbench/gemmaclaw , no working tree and builds a local gbench-quality image from the commit's git archive the TypeScript workspace installed and built inside it , then runs the suite in a container of that image against /v1 . - Build cache : the image is tagged with the commit and a digest of its Dockerfile, helpers and gBench's edits, so it is built once per commit several minutes on the first run . - Configuration : generates a throwaway openclaw.json that disables the third-party plugins the suite does not use, to keep memory use down. - Gemini-only helper models : the QA suite's alternate model google/gemini-3.1-pro-preview , via --alt-model , in place of openai/gpt-5.4 , image-generation model google/gemini-3.1-flash-image-preview , via a recorded edit applied in the image's build context, in place of openai/gpt-image-1 and memory embeddings google/gemini-embedding-001 , in place of openai/text-embedding-3-small run on the Gemini API, and every Gemini call is recorded in gemini calls ; GEMINI API KEYS or GEMINI API KEY is required. No key reaches the model under test: the suite runs in gBench's own Docker container no network, cloud metadata servers unreachable, a read-only checkout, a user without capabilities, no host path writable , and its only way out is a forwarder in the container that hands its model-endpoint and Gemini calls to a relay in gBench over a unix socket; the relay alone holds the keys and forwards only the Gemini API calls the suite makes. Harness errors relay, endpoint, Gemini or container failures are not scored; a model-endpoint 4xx other than 400 and 422 on the suite's own requests for example the 404 of an endpoint that serves the model under another name is one, and a run with no scenario left to score withholds its pass rate null , printed withheld , never 0. Details: docs/quality.md https://github.com/google-deepmind/gbench/blob/main/docs/quality.md . Run specific GemmaClaw QA scenarios by scenario id from a local checkout gbench --quality-only --remote-endpoint http://127.0.0.1:8000/v1 --tokenizer google/gemma-4-E4B-it --scenarios memory-recall mcp-plugin-tools-call --gemmaclaw-path /path/to/gemmaclaw 1. Install development environment in a test environment, see tests/README.md : pip install -e ". dev,evals,omnidocbench " transformers and python -m nltk.downloader punkt punkt tab ; the system tools the tests exercise are bubblewrap, a Java runtime and Docker the Unit Tests workflow installs exactly this 2. Run code style check: ruff check gbench/ 3. Run type checker: mypy gbench/ 4. Run unit test suite: pytest tests/ the Gemma 4 chat-template artifact check is not a unit test; it lives in tools/template checks/ , see its README 5. The pull-request gate is the unit-test suite pytest tests/ , see CONTRIBUTING.md https://github.com/google-deepmind/gbench/blob/main/CONTRIBUTING.md . A change to the Golden Set keeps its two-sided presence assertions; the Golden Set itself measures a model and never gates a gBench merge. Copyright 2026 Google LLC. Licensed under the Apache License, Version 2.0.