Oct 23, 2026 Β· 9 min read
TL;DR #
This is the fifth model in the ThinkingCap series. We reduced how much Google's gemma-4-26B-A4B-it thinks, without compromising the quality of its answers.
-
Up to 69.1% reduction in thinking tokens, on average 38.9% measured across twelve benchmarks, for0.15pp of accuracy
-
The first ThinkingCap on a Google model, and the smallest accuracy cost of any ThinkingCap so far
-
Looping drops from 3.6% of the original model's responses to 0.8% of ours
-
Same sampling settings (thinking on, temperature 1.0, top_p 0.95), same serving stack (vLLM), drop-in replacement ThinkingCap PRO: want even better efficiency? The PRO version of this model uses up to 87.6% fewer thinking tokens, 62% on average. Faster answers and lower inference costs at scale. Contact us at enterprise@bottlecapai.com.
Our fifth ThinkingCap #
We have published four token-efficient models so far: ThinkingCap-Qwen3.6-27B, ThinkingCap-Qwen3.8-27B, ThinkingCap-NVIDIA-Nemotron-3.5-Lightning-30B-A3B and ThinkingCap: Qwen3.8-Flash-Next.
gemma-4-26B-A4B is Google's sparse mixture of experts β 26B parameters with about 4B active per token. It is the third model family we have applied this work to, after Qwen and NVIDIA. On this one we put extra effort into keeping the accuracy intact: thinking comes down 38.9% on average at a cost of only 0.15pp of accuracy, which is smaller than the run-to-run spread on most of the suite.
Video comparison of the original model and the ThinkingCap model, prompted with the same input. Both models run on the same hardware and use the same inference settings.
The training objective #
We start with gemma-4-26B-A4B-it. The goal is to spend fewer tokens getting to the answer, and change nothing else: preserve the knowledge, reasoning ability, answer quality, instruction following and safety behaviour.
Results #
Accuracy per task family, gemma-4-26B-A4B-it against the ThinkingCap model Accuracy barely moves in any family, and in two it moves up. Instruction following and agentic gains 0.65pp and maths gains 0.39pp; code is flat at 0.02pp; knowledge and reasoning costs 0.51pp and long-context and multimodal 0.92pp. Averaged over all twelve benchmarks the change is β0.15pp.
Mean thinking tokens per task family, gemma-4-26B-A4B-it against the ThinkingCap model
The thinking is where the change is, and every family gets shorter. Long-context and multimodal reduces the most, 54.4%, then knowledge and reasoning at 44.4%, maths at 36.1% and code at 27.6%; instruction following and agentic reduces the least at 17.2%. Averaged over the twelve benchmarks it is 38.9%.
| Benchmark | Accuracy (base) | Accuracy (ours) | Ξ accuracy | Mean thinking tokens | Ξ mean thinking | Trunc. % | Looping % |
|---|---|---|---|---|---|---|---|
| Math | |||||||
aime26 |
86.20% Β±0.86 | 86.72% Β±0.98 | +0.52pp | 14766 β 8963 | -39.3% | 5.2 β 0.5% | 5.5 β 0.9% |
hmmt_nov25 |
81.51% Β±1.06 | 80.89% Β±1.19 | -0.62pp | 16462 β 10542 | -36.0% | 9.0 β 1.2% | 8.6 β 1.2% |
hmmt_feb26 |
72.92% Β±1.08 | 74.20% Β±1.16 | +1.28pp | 22005 β 14717 | -33.1% | 18.6 β 4.6% | 17.0 β 4.5% |
| Code | |||||||
livecodebench |
76.98% Β±0.56 | 76.96% Β±0.67 | -0.02pp | 11560 β 8375 | -27.6% | 6.8 β 1.3% | 2.2 β 0.3% |
| Knowledge & reasoning | |||||||
gpqa_diamond |
79.92% Β±0.57 | 78.96% Β±0.67 | -0.96pp | 7699 β 4284 | -44.4% | 0.3 β 0.1% | 0.3 β 0.1% |
mmlu_pro |
84.55% Β±0.29 | 84.64% Β±0.16 | +0.09pp | 3907 β 1989 | -49.1% | 0.0 β 0.0% | 0.1 β 0.0% |
mmmlu |
84.09% Β±0.26 | 83.43% Β±0.27 | -0.66pp | 1212 β 731 | -39.7% | 0.0 β 0.0% | 0.0 β 0.0% |
| Long-context & multimodal | |||||||
aa_lcr |
64.62% Β±2.49 | 60.62% Β±2.89 | -4.00pp | 6519 β 2655 | -59.3% | 4.0 β 0.0% | 3.6 β 0.4% |
longbench_v2 |
59.43% Β±1.14 | 60.88% Β±1.29 | +1.45pp | 9486 β 2934 | -69.1% | 7.7 β 0.5% | 5.9 β 1.2% |
realworldqa |
70.83% Β±0.56 | 70.61% Β±0.56 | -0.21pp | 819 β 533 | -34.9% | 0.0 β 0.0% | 0.0 β 0.0% |
| Instruction following & agentic | |||||||
ifbench |
76.83% Β±0.80 | 78.67% Β±0.73 | +1.83pp | 6470 β 5585 | -13.7% | 0.4 β 0.4% | 0.5 β 0.5% |
tau2 |
64.25% Β±0.97 | 63.73% Β±1.26 | -0.53pp | 4374 β 3466 | -20.8% | 0.0 β 0.0% | 0.0 β 0.0% |
| Average (12 benchmarks) | 75.18% | 75.03% | -0.15pp | 8773 β 5398 | -38.9% | 4.35 β 0.72% | 3.64 β 0.76% |
Every row is the original BF16 against the ThinkingCap model, run on the same harness with the same settings on the same questions. Accuracy is the mean over seeds, Β± the 95% confidence interval from the spread between them. Every cell in the last row is an average across the twelve benchmarks, each counting once β so Ξ mean thinking there is the mean of the twelve reductions, not the ratio of the two token figures beside it, which is 38.5%. The same distinction applies to the family bars above, which plot each family's mean tokens. On ΟΒ²-bench the thinking tokens are per task, summed across the rollout's turns, rather than per
response. Column definitions are in Evaluation methodology. Every one of the twelve gets shorter. The deepest cuts are LongBench v2 (β69.1%), AA-LCR (β59.3%) and MMLU-Pro (β49.1%); the shallowest IFBench (β13.7%) and ΟΒ²-bench (β20.8%). The two shallow rows are the two whose work is least about thinking at length: following a list of format constraints, and applying a fixed customer-service policy through the right tool calls.
On accuracy, most of the results are within noise: on eight of the twelve the gap is smaller than the run-to-run spread, and five of the twelve come out ahead of the original model. The four that move further than their intervals allow are IFBench (+1.83pp, in our favour), GPQA-Diamond
(β0.96pp), MMMLU (β0.66pp) and AA-LCR (β4.00pp). AA-LCR is the one real cost in the suite. Looping improves across the suite. The original model loops on 3.64% of its responses, against 0.76% for ours β and on the three benchmarks where it loops most, AIME 2026, HMMT Nov 25 and HMMT Feb 26, the rate falls from 5.5%, 8.6% and 17.0% to 0.9%, 1.2% and 4.5%.
Accuracy against the thinking-token budget, both models
Read left to right, this is the question a deployment actually asks: for a given token budget per question, how much accuracy do you get? ThinkingCap's curve sits above the original model's through the whole middle of the range β the same accuracy for a smaller budget β and the two converge once the budget is wide enough that neither model is constrained by it.
Evaluation methodology #
Reasoning models are noisy to evaluate. At the recommended temperature of 1.0 both the content and the length of a response move run to run, so a single seed tells you very little. Everything below is multiple seeds per benchmark, with accuracy reported as the mean and a 95% confidence interval across seeds.
Both sides are the same run. The original model and ours were evaluated by the same harness, with identical sampling and serving settings. The original model was measured by us, not quoted
from published figures. The configuration:
- Harness β our internal evaluation suite, running both models on vLLM.
- Context β generation capped at 65,536 tokens per response on every benchmark, and per turn on ΟΒ²-bench.
- Sampling β thinking on; temperature 1.0, top_p 0.95, top_k 64. Ours inherits Google's recommended preset unchanged, so no part of the comparison is a decoding change.
- Accuracy β the mean across seeds with a 95% t-interval, which says how far the answer moves when only the sampling seed changes.
- Seeds β 64 on AIME 2026, HMMT Nov 2025 and HMMT Feb 2026; 32 on GPQA-Diamond and LiveCodeBench; 16 on RealWorldQA, IFBench and ΟΒ²-bench; 8 on AA-LCR and LongBench v2; 4 on MMLU-Pro and MMMLU, whose question counts are large enough that question sampling is not the limiting noise.
- Question sets β AIME 2026 (30 problems), HMMT Nov 2025 (30), HMMT Feb 2026 (33), LiveCodeBench (175), GPQA-Diamond (198), MMLU-Pro (12,032), MMMLU (9,996), AA-LCR (100), LongBench v2 (370), RealWorldQA (765), IFBench (300) and ΟΒ²-bench (278 tasks across airline, retail and telecom).
- Scoring β exact match on the extracted answer, except LiveCodeBench, where the generated code is run against the problem's tests; IFBench, scored by its programmatic verifiers; and ΟΒ²-bench and AA-LCR, which are judged. ΟΒ²-bench also runs against a simulated user and is reported as the unweighted mean over its three domains.
What the columns mean
- Accuracy (base) /Accuracy (ours) β as above;Ξ accuracy is the difference between them, in percentage points.
- Ξ mean thinking β the headline: the change in thinking tokens, totalled across the benchmark and divided by its question count, so a long question weighs more than a short one.
- Mean thinking tokens are the tokens before the reasoning closes, averaged over every question and seed of the benchmark, written
base β ours. - Truncated % β the share of responses whose thinking never closed, which score zero.Looping % β the share caught by a compression-based loop detector, which flags a response whose text compresses far beyond what ordinary prose does.
Get the model #
You can find the model here: bottlecapai/ThinkingCap-gemma-4-26B-A4B-it, with GGUF builds published alongside it.
Contact us #
Have any questions, or a workload you want this run against?
Send us a mail at hey@bottlecapai.com ThinkingCap PRO: want even better efficiency? The PRO version of this model uses up to 87.6% fewer thinking tokens, 62% on average. Faster answers and lower inference costs at scale. Contact us at enterprise@bottlecapai.com.