TL;DR:VIDRAFT topped "The First Gemma Challenge" leaderboard with a verified 510.58 tokens-per-second (TPS) score ongoogle/gemma-4-E4B-it
using a single NVIDIA A10G GPU — while a rival submission posted a faster raw number but failed the quality gate. This post breaks down the public configuration choices that made it possible, and what engineers can steal for their own inference tuning work.
"The First Gemma Challenge" was a constrained inference-speed competition with two hard rules: one fixed GPU (NVIDIA A10G), one fixed model (google/gemma-4-E4B-it
). Participants could not swap in stronger hardware or a lighter model. The only lever available was software-level optimization.
The scoring metric was TPS (Tokens Per Second), but with a simultaneous Perplexity (PPL) budget — submissions whose PPL exceeded approximately 2.42 were disqualified regardless of speed. The organizers also ran a blind re-evaluation against a held-out prompt set that participants never saw, which meant any configuration over-fitted to self-reported benchmarks would get caught.
VIDRAFT's winning submission — configuration name vidraft-fw188-ctk49-n64-patchbridge-v1
— posted:
A competing entry recorded 535.91 TPS, but its PPL landed at approximately 2.44, breaching the quality threshold. That's why the lower raw number was recognized as the verified SOTA.
The public manifest.json
for the winning configuration reveals three conceptual optimization pillars:
SLIDING_WINDOW=188
)
The KV-cache memory bandwidth is the dominant bottleneck during autoregressive generation. Restricting the attention window to only the most recent tokens reduces this pressure and increases throughput — but shrink the window too far and you lose context, causing PPL to spike. The value 188 is notably not a round number, which strongly suggests it was determined empirically rather than chosen from a default. The team overrode the model's text_config.sliding_window
via HF_OVERRIDES
and enabled Flash Attention sliding (FA_SLIDING=1
) to match.
CENTROID_TOP_K=49
) This parameter sits closer to the kernel level and affects both throughput and PPL simultaneously. According to the source analysis, values like 44, 48, and 49 were tested sequentially — the goal being to find the highest value that still kept PPL within budget. Bigger is not automatically better; it's a Pareto search under the quality constraint.
The configuration uses: WARMUP_BRIDGE=1
, WARMUP_NUM_PROMPTS=64
, WARMUP_MAX_TOKENS=1
, WARMUP_SEED=42
. This fires 64 single-token dummy prompts before the timed benchmark begins, so that CUDA graph capture and JIT compilation costs are absorbed before the clock starts. The source article notes this warm-up was worth approximately 15 TPS — a significant margin in a competition decided by tens of TPS.
Equally important: PRECACHE_BENCH=0
is explicitly set, disabling a flag that would have inflated the self-reported TPS. The team chose to measure what the blind evaluator would actually see.
SPECULATIVE_CONFIG
, with num_speculative_tokens=7
and method=mtp
— a draft-then-verify approach that increases tokens generated per forward passMAX_MODEL_LEN=4096
GPU_MEMORY_UTILIZATION=0.90
MAX_NUM_BATCHED_TOKENS=512
MAX_NUM_SEQS=1
| Submission | TPS | PPL | Blind eval |
|---|---|---|---|
VIDRAFT (vidraft-fw188-ctk49-n64-patchbridge-v1 ) |
|||
| 510.58 | |||
| 2.3930 | |||
| ✅ Passed | |||
| Competing entry | 535.91 | ~2.44 | ❌ Failed (PPL > 2.42) |
The key takeaway: raw throughput rank and validated rank diverged because the quality gate was enforced on a held-out prompt distribution, not the participants' own test sets.
The model used in the competition is publicly available from Google on Hugging Face:
huggingface-cli download google/gemma-4-E4B-it
The specific VIDRAFT configuration (vidraft-fw188-ctk49-n64-patchbridge-v1
) and any VIDRAFT-specific tooling are not confirmed as publicly released at the time of writing. Check VIDRAFT's Hugging Face organization and their GitHub for updates. If access channels are announced, they will appear there first.
Q: Why does the PPL threshold matter more than raw TPS in a "speed" competition?
A: Because TPS without a quality floor is trivially gamed — you can degrade output until the model produces garbage very quickly. The PPL ceiling plus blind re-evaluation together enforce that the speed number reflects real, deployable inference quality.
Q: Can I apply these same techniques to other models or GPUs?
A: The concepts — quality-gated parameter search, warm-up separation, sliding window tuning, speculative decoding — are general inference engineering practices. The specific numeric values (SLIDING_WINDOW=188
, CENTROID_TOP_K=49
, etc.) were tuned for google/gemma-4-E4B-it
on a single A10G and should be treated as starting points, not copy-paste targets, for different hardware or model configurations.
Q: What is speculative decoding ( method=mtp) doing here?
Originally reported by note (일본) (2026-08-15) — source article.