{"slug": "local-ai-775-tok-s-diffusiongemma-bf16-on-nvidia-rtx-6000-pro", "title": "Local AI: 775 tok/s, DiffusionGemma (BF16) on Nvidia RTX 6000 Pro", "summary": "A developer achieved 775 tokens per second running the full BF16 DiffusionGemma model on an Nvidia RTX 6000 Pro using a Red Hat fork of vLLM, demonstrating extremely fast local AI inference at short contexts. However, performance degrades sharply with longer inputs, with a time-to-first-token of 22 seconds at 100,000 tokens. The result highlights the trade-off between speed and context length for locally deployed large language models.", "body_md": "Inanely Fast Local AI: 775 token per second! 🤯 I was able to run the new DiffusionGemma (full BF16 model) by @googlegemma on vLLM (fork by Red Hat) on Nvidia RTX 6000 Pro. It's blazing fast at short contexts, but gets slow very quickly. At 100k, TTFT is 22s!\n■ Leave a comment setup and command to run the model.", "url": "https://wpnews.pro/news/local-ai-775-tok-s-diffusiongemma-bf16-on-nvidia-rtx-6000-pro", "canonical_source": "https://twitter.com/OrganicGPT/status/2064883777499795716", "published_at": "2026-06-11 21:33:42+00:00", "updated_at": "2026-06-11 21:49:53.736151+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "large-language-models", "generative-ai", "ai-infrastructure"], "entities": ["DiffusionGemma", "Google", "vLLM", "Red Hat", "Nvidia", "RTX 6000 Pro"], "alternates": {"html": "https://wpnews.pro/news/local-ai-775-tok-s-diffusiongemma-bf16-on-nvidia-rtx-6000-pro", "markdown": "https://wpnews.pro/news/local-ai-775-tok-s-diffusiongemma-bf16-on-nvidia-rtx-6000-pro.md", "text": "https://wpnews.pro/news/local-ai-775-tok-s-diffusiongemma-bf16-on-nvidia-rtx-6000-pro.txt", "jsonld": "https://wpnews.pro/news/local-ai-775-tok-s-diffusiongemma-bf16-on-nvidia-rtx-6000-pro.jsonld"}}