{"slug": "gbench-deepmind-open-sources-evaluation-suite-with-86-benchmark-suites", "title": "GBench: DeepMind open-sources evaluation suite with 86 benchmark suites", "summary": "Google DeepMind released gBench, an open-source benchmarking and evaluation suite for open large language models and vision-language models that bundles 86 canonical academic and domain eval suites across 6 capability pillars. gBench measures serving performance on the hardware it runs on and model capability in three tiers — academic evals, 16 deterministic Golden Set smoke tests, and the GemmaClaw/OpenClaw agentic quality suite with 60 QA scenarios pinned at commit d8bc6989 — and works with any OpenAI-compatible /v1 server via --remote-endpoint, validated on vLLM and Ollama. The suite also adds model discovery from HuggingFace config.json with GPU-tier sizing (1 GPU up to 20B parameters, 2 GPUs up to 80B, 8 GPUs above) and an open-loop stress sweep that certifies throughput limits to ±9% using request cohorts of at least 1099 probes.", "body_md": "gBench is an open-source benchmarking and evaluation suite for open large language models (LLMs) and vision-language models (VLMs). It measures serving performance on the hardware it runs on, and model capability in three tiers: academic eval suites, the Golden Set smoke tests and the agentic quality suite.\n\nIt works with any OpenAI-compatible `/v1` server through `--remote-endpoint` (validated on vLLM and Ollama; on Cloud Run, a service that allows unauthenticated calls), or starts and stops a local vLLM server itself.\n\n- OpenAI API compatibility: measure any OpenAI-compatible `/v1` endpoint with`--remote-endpoint <URL>/v1` (validated on vLLM and Ollama; on Cloud Run, a service that allows unauthenticated calls; see[Remote Endpoint Mode](https://github.com/google-deepmind/gbench/blob/main/docs/remote-endpoints.md#remote-endpoint-mode---remote-endpoint) ).\n- Three capability tiers:\n  1. Academic and domain evals (`--evals` ): 86 canonical Python suites in`--evals all` (plus`ui_control_osworld` and`swe_lancer` , which run only when named) across 6 core capability pillars (`General Knowledge & Scientific Reasoning` ,`Mathematics & Proofs` ,`Coding & Algorithmic Design` ,`Long Context & Retrieval` ,`Tool Use & Agentic Workflows` ,`Multimodal Vision & Grounding` ) with the model's reasoning mode on request (`--eval-thinking` ), images sent as their original bytes where the dataset stores them (no re-encoding), plugin discovery (`--eval-plugins-dir` ) and custom JSONL datasets (`--eval-custom-jsonl` ).\n  2. Golden Set smoke tests (`--golden-only` ): 16 deterministic capability checks (`code_canonical` ,`function_call_single` ,`structured_json` , etc.) with two-sided presence assertions and`python_exec` unit-test matching, including a labelled[chat template subset](https://github.com/google-deepmind/gbench/blob/main/docs/golden-set.md#chat-template-sensitivity-pairs) that separates a broken chat template from a weak model.\n  3. Agentic quality suite (`--quality-only` ): the GemmaClaw / OpenClaw QA suite (the QA scenarios of the pinned GemmaClaw commit, 60 at d8bc6989; scenarios that need another model provider are listed apart and not scored) testing multi-turn tool calling, session memory and plugin routing, run in gBench's local`gbench-quality` Docker image (built once per GemmaClaw commit).\n- Academic and domain evals (\n- Model discovery and GPU sizing: model properties read from the HuggingFace `config.json` (parameter count, MoE topology, vision/audio tokens), a recommended GPU tier per model size (1 GPU up to 20B, 2 GPUs up to 80B, 8 GPUs above) that is checked against the required`--num-gpus` (a warning when off the tier, a refusal when the model will not fit), and a server startup timeout that scales with model size (`max(600, min(3600, (300 + params_b * 10) * 3 * (1.3 if MoE)))` ).\n- Repeatability checks: several iterations per configuration with mean, median, standard deviation, P50, P95, P99 and the coefficient of variation, checked against `--max-cv-percent` (default 5.0).\n- Open-loop stress sweep (`--stress-test` ): one merged Poisson schedule per rate point at increasing rates (x2 per step) until a point fails; each of`--stress-reps` repeats gives a bracket [pass, fail], and the median repeat's limit is then certified with probes sized for the P99 SLO (a cohort of >= 1099 requests, about 1400 sent, the same rule on every device) to +-9% when that resolution is reached (otherwise a certified lower bound with its resolution, a lower bound censored at`--stress-max-qps` , or no limit when certification refutes the sweep), or reported UNCERTIFIED with the time certifying would need (`--stress-certify-budget` ). A point passes when, on the requests planned after the point's ramp-up: the queue is not growing, >= 90% finish within the end-to-end time the SLOs imply, P99 TTFT and P99 ITL meet their SLOs, failures stay under 2% and the client kept to its schedule. The queue-growth test reads TTFT; overload that shows only as longer end-to-end times (for example KV-cache preemption) is left to the completion-fraction and ITL gates. SLOs are per campaign (5000 ms TTFT and 200 ms ITL without`--campaign` ), overridden by`--stress-threshold` and`--itl-slo-ms` .\n\n**Using gBench**\n\n- [Installation](https://github.com/google-deepmind/gbench/blob/main/docs/installation.md) : requirements, a virtual environment or conda, and the optional installs per use.\n- [CLI argument reference](https://github.com/google-deepmind/gbench/blob/main/docs/cli-reference.md) : every command-line flag.\n- [Environment variables](https://github.com/google-deepmind/gbench/blob/main/docs/environment-variables.md) , and every per-suite knob in[docs/evals/env-knobs.md](https://github.com/google-deepmind/gbench/blob/main/docs/evals/env-knobs.md) .\n- [Remote endpoint mode](https://github.com/google-deepmind/gbench/blob/main/docs/remote-endpoints.md) : benchmarking any OpenAI-compatible`/v1` server, plus the preflight, smoke and parallel-suite modes ([docs/smoke.md](https://github.com/google-deepmind/gbench/blob/main/docs/smoke.md) ).\n- [Results](https://github.com/google-deepmind/gbench/blob/main/docs/results.md) : output layout, result schemas, provenance and exit codes.\n- [Distributed runs](https://github.com/google-deepmind/gbench/blob/main/docs/distributed-gbench.md) : splitting a sweep across nodes (`--shard` ) and merging the results (`--merge` ).\n- [Deployment modes](https://github.com/google-deepmind/gbench/blob/main/docs/deployment.md) : the CLI today; container, Kubernetes and hosted modes on the roadmap.\n\n**What gBench measures**\n\n- [Performance benchmarks](https://github.com/google-deepmind/gbench/blob/main/docs/performance.md) : model discovery, GPU tiers and selection, workload campaigns, dataset backends and repeatability.\n- [Academic and custom eval suites](https://github.com/google-deepmind/gbench/blob/main/docs/eval-suites.md) (`--evals` ), with one setup and scoring guide per suite in[docs/evals/](https://github.com/google-deepmind/gbench/blob/main/docs/evals) , plus[toolchains.md](https://github.com/google-deepmind/gbench/blob/main/docs/evals/toolchains.md) (optional dependencies),[docker-sandboxes.md](https://github.com/google-deepmind/gbench/blob/main/docs/evals/docker-sandboxes.md) (Docker address pools for the containerized suites) and[large-sweep-setup.md](https://github.com/google-deepmind/gbench/blob/main/docs/evals/large-sweep-setup.md) (host limits for long sweeps).\n- [Golden Set capability smoke test](https://github.com/google-deepmind/gbench/blob/main/docs/golden-set.md) (`--golden-only` ).\n- [Agentic quality suite](https://github.com/google-deepmind/gbench/blob/main/docs/quality.md) (`--quality-only` ).\n\n**Background**\n\n- [docs/THIRD_PARTY_DATASETS.md](https://github.com/google-deepmind/gbench/blob/main/docs/THIRD_PARTY_DATASETS.md) : the datasets the suites load, with their licenses, gated and non-commercial terms. Each dataset's license governs your use of it and of the scores computed on it; read it before publishing results.\n- [docs/RESOURCES.md](https://github.com/google-deepmind/gbench/blob/main/docs/RESOURCES.md) : the upstream source of each suite.[docs/GOSPEL.md](https://github.com/google-deepmind/gbench/blob/main/docs/GOSPEL.md) : what gBench is and who it is for.\n- [Scaffold versioning](https://github.com/google-deepmind/gbench/blob/main/docs/scaffold-versioning.md) : what each pillar pins, so results stay comparable.[Endpoint relay](https://github.com/google-deepmind/gbench/blob/main/docs/relay.md) : how agent harnesses that cannot be pointed at the endpoint reach the served model.[Evals roadmap](https://github.com/google-deepmind/gbench/blob/main/docs/evals_roadmap.md) : planned suites and the engineering playbook.\n- [examples/notebooks/](https://github.com/google-deepmind/gbench/blob/main/examples/notebooks) : five runnable notebooks (Colab or local Jupyter, against Ollama or a vLLM endpoint; 05 needs Docker, so a local or cloud machine rather than Colab):[01 quickstart](https://github.com/google-deepmind/gbench/blob/main/examples/notebooks/01_gbench_101_quickstart.ipynb) (serving latency and the stress limit),[02 workload campaigns and concurrency](https://github.com/google-deepmind/gbench/blob/main/examples/notebooks/02_workloads_campaigns_and_concurrency.ipynb) ,[03 Golden Set smoke tests](https://github.com/google-deepmind/gbench/blob/main/examples/notebooks/03_golden_set_capability_smoke_tests.ipynb) ,[04 academic evals and reasoning mode](https://github.com/google-deepmind/gbench/blob/main/examples/notebooks/04_academic_evaluations_and_reasoning.ipynb) and[05 agentic quality with GemmaClaw](https://github.com/google-deepmind/gbench/blob/main/examples/notebooks/05_agentic_quality_with_gemmaclaw.ipynb) .\n- [CONTRIBUTING.md](https://github.com/google-deepmind/gbench/blob/main/CONTRIBUTING.md) : how to contribute.\n\n```\ngit clone https://github.com/google-deepmind/gbench.git\ncd gbench\npython3 -m venv venv && source venv/bin/activate\n\npip install -e .                 # base package (Python 3.11+): laptops and remote endpoints such as Ollama\n# optional, per use:\npip install transformers         # --remote-endpoint: prompt sizing (--tokenizer), Golden Set chat-template\n                                 # cases and the mmlu, wmdp, ruler and longbench_v2_code suites\npip install vllm                 # local GPU serving, and remote real-dataset workloads (--campaign chat-like,\n                                 # --dataset custom/hf); gBench never installs vllm/torch itself\npip install -e \".[evals]\"        # the eval suites' client, scorer and harness dependencies\n```\n\nThe [installation guide](https://github.com/google-deepmind/gbench/blob/main/docs/installation.md) covers the requirements, conda, the optional installs per suite and the test environment.\n\n```\n# 1. Quick local performance smoke test (auto-starts local vLLM server)\ngbench --models google/gemma-4-E4B-it --num-gpus 1 --preset quick\n\n# 2. Run a specific workload performance campaign (e.g. agentic long-context)\ngbench --models google/gemma-4-12B-it --num-gpus 1 --campaign agentic\n\n# 3. Run the Golden Set capability smoke tests on a local vLLM server\n#    gbench launches; its tool cases need the model's tool-call parser (Gemma 4 example)\ngbench --golden-only --models google/gemma-4-12B-it --num-gpus 1 --tool-call-parser gemma4 --reasoning-parser gemma4\n\n# 4. Run GemmaClaw agentic quality suite over remote endpoint\ngbench --quality-only --models google/gemma-4-12B-it --remote-endpoint http://127.0.0.1:8000/v1 --tokenizer google/gemma-4-12B-it\n\n# 5. Run academic evaluations with Chain-of-Thought reasoning against remote server\ngbench --evals-only --remote-endpoint http://127.0.0.1:8000/v1 --tokenizer google/gemma-4-E4B-it --evals gpqa mmlu --eval-thinking --max-output-tokens 65536\n\n# 6. Stage model artifacts from HuggingFace to Google Cloud Storage\ngbench --models google/gemma-4-12B-it --stage-to-gcs gs://my-gbench-bucket/models/\n\n# 7. Dry run: the plan, each server's vllm serve command, port, GPUs and base URL (validated)\ngbench --models google/gemma-4-12B-it --num-gpus 1 --preset default --dry-run\n+-------------------------------------------------------------------------------+\n|                         gbench capability evaluation                          |\n+-------------------------------------------------------------------------------+\n|  Tier 1: academic evals         |  Tier 2: Golden Set    |  Tier 3: agentic QA|\n|  (--evals)                      |  (--golden-only)       |  (--quality-only)  |\n|  - bfcl, gpqa, gsm8k, mmlu,     |  - 16 cases            |  - GemmaClaw suite |\n|    mmmu_pro, mrcr, screenspot   |  - Two-sided presence  |  - QA scenarios    |\n|  - Reasoning mode (--thinking)  |  - Chat template pairs |  - Multi-turn QA   |\n|  - Vision soft-token budgets    |  - N/A-aware scoring   |  - Tools, memory   |\n+---------------------------------+------------------------+--------------------+\n                                         |\n                                         v\n+-------------------------------------------------------------------------------+\n|                      OpenAI-compatible REST API (/v1)                         |\n+-------------------------------------------------------------------------------+\n|  Mode A: local vLLM server  <---- OR ---->  Mode B: remote endpoint           |\n|  (Auto start / stop / VRAM poll)            (--remote-endpoint <URL>/v1)      |\n+-------------------------------------------------------------------------------+\n```\n\nThe full catalog of the 86 suites in 6 pillars, with each suite's protocol and options, custom plugins and JSONL suites: [docs/eval-suites.md](https://github.com/google-deepmind/gbench/blob/main/docs/eval-suites.md). Each suite has a setup and scoring guide in [docs/evals/](https://github.com/google-deepmind/gbench/blob/main/docs/evals).\n\n16 deterministic capability checks (code, math, tool calling, structured output, chat templates, factual knowledge, translation, safety, vision and audio) with two-sided presence assertions and `python_exec` unit-test matching, including a [chat-template subset](https://github.com/google-deepmind/gbench/blob/main/docs/golden-set.md#chat-template-sensitivity-pairs) that separates a broken chat template from a weak model. Details: [docs/golden-set.md](https://github.com/google-deepmind/gbench/blob/main/docs/golden-set.md).\n\nEvaluates multi-turn agent behavior with the GemmaClaw / OpenClaw QA suite: the QA scenarios of the pinned GemmaClaw commit (60 at d8bc6989); scenarios that need a model provider other than the model under test and the Gemini API are listed apart and not scored.\n\n- **Image build** : clones GemmaClaw (`~/.cache/gbench/gemmaclaw` , no working tree) and builds a local`gbench-quality` image from the commit's`git archive` (the TypeScript workspace installed and built inside it), then runs the suite in a container of that image against`/v1` .\n- **Build cache** : the image is tagged with the commit and a digest of its Dockerfile, helpers and gBench's edits, so it is built once per commit (several minutes on the first run).\n- **Configuration** : generates a throwaway`openclaw.json` that disables the third-party plugins the suite does not use, to keep memory use down.\n- **Gemini-only helper models** : the QA suite's alternate model (`google/gemini-3.1-pro-preview` , via`--alt-model` , in place of`openai/gpt-5.4` ), image-generation model (`google/gemini-3.1-flash-image-preview` , via a recorded edit applied in the image's build context, in place of`openai/gpt-image-1` ) and memory embeddings (`google/gemini-embedding-001` , in place of`openai/text-embedding-3-small` ) run on the Gemini API, and every Gemini call is recorded in`gemini_calls` ;`GEMINI_API_KEYS` or`GEMINI_API_KEY` is required. No key reaches the model under test: the suite runs in gBench's own Docker container (no network, cloud metadata servers unreachable, a read-only checkout, a user without capabilities, no host path writable), and its only way out is a forwarder in the container that hands its model-endpoint and Gemini calls to a relay in gBench over a unix socket; the relay alone holds the keys and forwards only the Gemini API calls the suite makes. Harness errors (relay, endpoint, Gemini or container failures) are not scored; a model-endpoint 4xx other than 400 and 422 on the suite's own requests (for example the 404 of an endpoint that serves the model under another name) is one, and a run with no scenario left to score withholds its pass rate (`null` , printed`withheld` ), never 0. Details:[docs/quality.md](https://github.com/google-deepmind/gbench/blob/main/docs/quality.md) .\n\n```\n# Run specific GemmaClaw QA scenarios (by scenario id) from a local checkout\ngbench --quality-only --remote-endpoint http://127.0.0.1:8000/v1 --tokenizer google/gemma-4-E4B-it --scenarios memory-recall mcp-plugin-tools-call --gemmaclaw-path /path/to/gemmaclaw\n```\n\n1. Install development environment (in a test environment, see tests/README.md): `pip install -e \".[dev,evals,omnidocbench]\" transformers` and`python -m nltk.downloader punkt punkt_tab` ; the system tools the tests exercise are bubblewrap, a Java runtime and Docker (the Unit Tests workflow installs exactly this)\n2. Run code style check: `ruff check gbench/`\n3. Run type checker: `mypy gbench/`\n4. Run unit test suite: `pytest tests/` (the Gemma 4 chat-template artifact check is not a unit test; it lives in`tools/template_checks/` , see its README)\n5. The pull-request gate is the unit-test suite (`pytest tests/` , see[CONTRIBUTING.md](https://github.com/google-deepmind/gbench/blob/main/CONTRIBUTING.md) ). A change to the Golden Set keeps its two-sided presence assertions; the Golden Set itself measures a model and never gates a gBench merge.\n\nCopyright 2026 Google LLC. Licensed under the Apache License, Version 2.0.", "url": "https://wpnews.pro/news/gbench-deepmind-open-sources-evaluation-suite-with-86-benchmark-suites", "canonical_source": "https://github.com/google-deepmind/gbench", "published_at": "2026-10-31 13:43:29+00:00", "updated_at": "2026-10-31 14:18:41.603462+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-research", "ai-infrastructure", "ai-tools"], "entities": ["Google DeepMind", "gBench", "vLLM", "Ollama", "HuggingFace", "GemmaClaw", "OpenClaw", "Cloud Run"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/gbench-deepmind-open-sources-evaluation-suite-with-86-benchmark-suites", "markdown": "https://wpnews.pro/news/gbench-deepmind-open-sources-evaluation-suite-with-86-benchmark-suites.md", "text": "https://wpnews.pro/news/gbench-deepmind-open-sources-evaluation-suite-with-86-benchmark-suites.txt", "jsonld": "https://wpnews.pro/news/gbench-deepmind-open-sources-evaluation-suite-with-86-benchmark-suites.jsonld"}}