GBench: DeepMind open-sources evaluation suite with 86 benchmark suites

gBench is an open-source benchmarking and evaluation suite for open large language models (LLMs) and vision-language models (VLMs). It measures serving performance on the hardware it runs on, and model capability in three tiers: academic eval suites, the Golden Set smoke tests and the agentic quality suite.

It works with any OpenAI-compatible /v1 server through --remote-endpoint (validated on vLLM and Ollama; on Cloud Run, a service that allows unauthenticated calls), or starts and stops a local vLLM server itself.

• OpenAI API compatibility: measure any OpenAI-compatible /v1 endpoint with--remote-endpoint <URL>/v1 (validated on vLLM and Ollama; on Cloud Run, a service that allows unauthenticated calls; seeRemote Endpoint Mode ).
• Three capability tiers:
  1. Academic and domain evals (--evals ): 86 canonical Python suites in--evals all (plusui_control_osworld andswe_lancer , which run only when named) across 6 core capability pillars (General Knowledge & Scientific Reasoning ,Mathematics & Proofs ,Coding & Algorithmic Design ,Long Context & Retrieval ,Tool Use & Agentic Workflows ,Multimodal Vision & Grounding ) with the model's reasoning mode on request (--eval-thinking ), images sent as their original bytes where the dataset stores them (no re-encoding), plugin discovery (--eval-plugins-dir ) and custom JSONL datasets (--eval-custom-jsonl ).
  2. Golden Set smoke tests (--golden-only ): 16 deterministic capability checks (code_canonical ,function_call_single ,structured_json , etc.) with two-sided presence assertions andpython_exec unit-test matching, including a labelledchat template subset that separates a broken chat template from a weak model.
  3. Agentic quality suite (--quality-only ): the GemmaClaw / OpenClaw QA suite (the QA scenarios of the pinned GemmaClaw commit, 60 at d8bc6989; scenarios that need another model provider are listed apart and not scored) testing multi-turn tool calling, session memory and plugin routing, run in gBench's localgbench-quality Docker image (built once per GemmaClaw commit).
• Academic and domain evals (
• Model discovery and GPU sizing: model properties read from the HuggingFace config.json (parameter count, MoE topology, vision/audio tokens), a recommended GPU tier per model size (1 GPU up to 20B, 2 GPUs up to 80B, 8 GPUs above) that is checked against the required--num-gpus (a warning when off the tier, a refusal when the model will not fit), and a server startup timeout that scales with model size (max(600, min(3600, (300 + params_b * 10) * 3 * (1.3 if MoE))) ).
• Repeatability checks: several iterations per configuration with mean, median, standard deviation, P50, P95, P99 and the coefficient of variation, checked against --max-cv-percent (default 5.0).
• Open-loop stress sweep (--stress-test ): one merged Poisson schedule per rate point at increasing rates (x2 per step) until a point fails; each of--stress-reps repeats gives a bracket [pass, fail], and the median repeat's limit is then certified with probes sized for the P99 SLO (a cohort of >= 1099 requests, about 1400 sent, the same rule on every device) to +-9% when that resolution is reached (otherwise a certified lower bound with its resolution, a lower bound censored at--stress-max-qps , or no limit when certification refutes the sweep), or reported UNCERTIFIED with the time certifying would need (--stress-certify-budget ). A point passes when, on the requests planned after the point's ramp-up: the queue is not growing, >= 90% finish within the end-to-end time the SLOs imply, P99 TTFT and P99 ITL meet their SLOs, failures stay under 2% and the client kept to its schedule. The queue-growth test reads TTFT; overload that shows only as longer end-to-end times (for example KV-cache preemption) is left to the completion-fraction and ITL gates. SLOs are per campaign (5000 ms TTFT and 200 ms ITL without--campaign ), overridden by--stress-threshold and--itl-slo-ms .

Using gBench

• Installation : requirements, a virtual environment or conda, and the optional installs per use.
• CLI argument reference : every command-line flag.
• Environment variables , and every per-suite knob indocs/evals/env-knobs.md .
• Remote endpoint mode : benchmarking any OpenAI-compatible/v1 server, plus the preflight, smoke and parallel-suite modes (docs/smoke.md ).
• Results : output layout, result schemas, provenance and exit codes.
• Distributed runs : splitting a sweep across nodes (--shard ) and merging the results (--merge ).
• Deployment modes : the CLI today; container, Kubernetes and hosted modes on the roadmap.

What gBench measures

• Performance benchmarks : model discovery, GPU tiers and selection, workload campaigns, dataset backends and repeatability.
• Academic and custom eval suites (--evals ), with one setup and scoring guide per suite indocs/evals/ , plustoolchains.md (optional dependencies),docker-sandboxes.md (Docker address pools for the containerized suites) andlarge-sweep-setup.md (host limits for long sweeps).
• Golden Set capability smoke test (--golden-only ).
• Agentic quality suite (--quality-only ).

Background

• docs/THIRD_PARTY_DATASETS.md : the datasets the suites load, with their licenses, gated and non-commercial terms. Each dataset's license governs your use of it and of the scores computed on it; read it before publishing results.
• docs/RESOURCES.md : the upstream source of each suite.docs/GOSPEL.md : what gBench is and who it is for.
• Scaffold versioning : what each pillar pins, so results stay comparable.Endpoint relay : how agent harnesses that cannot be pointed at the endpoint reach the served model.Evals roadmap : planned suites and the engineering playbook.
• examples/notebooks/ : five runnable notebooks (Colab or local Jupyter, against Ollama or a vLLM endpoint; 05 needs Docker, so a local or cloud machine rather than Colab):01 quickstart (serving latency and the stress limit),02 workload campaigns and concurrency ,03 Golden Set smoke tests ,04 academic evals and reasoning mode and05 agentic quality with GemmaClaw .
• CONTRIBUTING.md : how to contribute.

git clone https://github.com/google-deepmind/gbench.git
cd gbench
python3 -m venv venv && source venv/bin/activate

pip install -e .                 # base package (Python 3.11+): laptops and remote endpoints such as Ollama
pip install transformers         # --remote-endpoint: prompt sizing (--tokenizer), Golden Set chat-template
pip install vllm                 # local GPU serving, and remote real-dataset workloads (--campaign chat-like,
pip install -e ".[evals]"        # the eval suites' client, scorer and harness dependencies

The installation guide covers the requirements, conda, the optional installs per suite and the test environment.

gbench --models google/gemma-4-E4B-it --num-gpus 1 --preset quick

gbench --models google/gemma-4-12B-it --num-gpus 1 --campaign agentic

gbench --golden-only --models google/gemma-4-12B-it --num-gpus 1 --tool-call-parser gemma4 --reasoning-parser gemma4

gbench --quality-only --models google/gemma-4-12B-it --remote-endpoint http://127.0.0.1:8000/v1 --tokenizer google/gemma-4-12B-it

gbench --evals-only --remote-endpoint http://127.0.0.1:8000/v1 --tokenizer google/gemma-4-E4B-it --evals gpqa mmlu --eval-thinking --max-output-tokens 65536

gbench --models google/gemma-4-12B-it --stage-to-gcs gs://my-gbench-bucket/models/

gbench --models google/gemma-4-12B-it --num-gpus 1 --preset default --dry-run
+-------------------------------------------------------------------------------+
|                         gbench capability evaluation                          |
+-------------------------------------------------------------------------------+
|  Tier 1: academic evals         |  Tier 2: Golden Set    |  Tier 3: agentic QA|
|  (--evals)                      |  (--golden-only)       |  (--quality-only)  |
|  - bfcl, gpqa, gsm8k, mmlu,     |  - 16 cases            |  - GemmaClaw suite |
|    mmmu_pro, mrcr, screenspot   |  - Two-sided presence  |  - QA scenarios    |
|  - Reasoning mode (--thinking)  |  - Chat template pairs |  - Multi-turn QA   |
|  - Vision soft-token budgets    |  - N/A-aware scoring   |  - Tools, memory   |
+---------------------------------+------------------------+--------------------+
                                         |
                                         v
+-------------------------------------------------------------------------------+
|                      OpenAI-compatible REST API (/v1)                         |
+-------------------------------------------------------------------------------+
|  Mode A: local vLLM server  <---- OR ---->  Mode B: remote endpoint           |
|  (Auto start / stop / VRAM poll)            (--remote-endpoint <URL>/v1)      |
+-------------------------------------------------------------------------------+

The full catalog of the 86 suites in 6 pillars, with each suite's protocol and options, custom plugins and JSONL suites: docs/eval-suites.md. Each suite has a setup and scoring guide in docs/evals/.

16 deterministic capability checks (code, math, tool calling, structured output, chat templates, factual knowledge, translation, safety, vision and audio) with two-sided presence assertions and python_exec unit-test matching, including a chat-template subset that separates a broken chat template from a weak model. Details: docs/golden-set.md.

Evaluates multi-turn agent behavior with the GemmaClaw / OpenClaw QA suite: the QA scenarios of the pinned GemmaClaw commit (60 at d8bc6989); scenarios that need a model provider other than the model under test and the Gemini API are listed apart and not scored.

• Image build : clones GemmaClaw (~/.cache/gbench/gemmaclaw , no working tree) and builds a localgbench-quality image from the commit'sgit archive (the TypeScript workspace installed and built inside it), then runs the suite in a container of that image against/v1 .
• Build cache : the image is tagged with the commit and a digest of its Dockerfile, helpers and gBench's edits, so it is built once per commit (several minutes on the first run).
• Configuration : generates a throwawayopenclaw.json that disables the third-party plugins the suite does not use, to keep memory use down.
• Gemini-only helper models : the QA suite's alternate model (google/gemini-3.1-pro-preview , via--alt-model , in place ofopenai/gpt-5.4 ), image-generation model (google/gemini-3.1-flash-image-preview , via a recorded edit applied in the image's build context, in place ofopenai/gpt-image-1 ) and memory embeddings (google/gemini-embedding-001 , in place ofopenai/text-embedding-3-small ) run on the Gemini API, and every Gemini call is recorded ingemini_calls ;GEMINI_API_KEYS orGEMINI_API_KEY is required. No key reaches the model under test: the suite runs in gBench's own Docker container (no network, cloud metadata servers unreachable, a read-only checkout, a user without capabilities, no host path writable), and its only way out is a forwarder in the container that hands its model-endpoint and Gemini calls to a relay in gBench over a unix socket; the relay alone holds the keys and forwards only the Gemini API calls the suite makes. Harness errors (relay, endpoint, Gemini or container failures) are not scored; a model-endpoint 4xx other than 400 and 422 on the suite's own requests (for example the 404 of an endpoint that serves the model under another name) is one, and a run with no scenario left to score withholds its pass rate (null , printedwithheld ), never 0. Details:docs/quality.md .

gbench --quality-only --remote-endpoint http://127.0.0.1:8000/v1 --tokenizer google/gemma-4-E4B-it --scenarios memory-recall mcp-plugin-tools-call --gemmaclaw-path /path/to/gemmaclaw

1. Install development environment (in a test environment, see tests/README.md): pip install -e ".[dev,evals,omnidocbench]" transformers andpython -m nltk.down punkt punkt_tab ; the system tools the tests exercise are bubblewrap, a Java runtime and Docker (the Unit Tests workflow installs exactly this)
2. Run code style check: ruff check gbench/
3. Run type checker: mypy gbench/
4. Run unit test suite: pytest tests/ (the Gemma 4 chat-template artifact check is not a unit test; it lives intools/template_checks/ , see its README)
5. The pull-request gate is the unit-test suite (pytest tests/ , seeCONTRIBUTING.md ). A change to the Golden Set keeps its two-sided presence assertions; the Golden Set itself measures a model and never gates a gBench merge.

Copyright 2026 Google LLC. Licensed under the Apache License, Version 2.0.