That research has returned in the best possible way: as a new open weight (Apache 2 licensed) Gemma model, google/diffusiongemma-26B-A4B-it.
NVIDIA are currently hosting the model for free on their NIM cloud API. I used that API to generate this pelican, which took 4.4s (according to time uv run generate.py
) to return 2,409 tokens - so at least 500 tokens/second.
Via [Hacker News](https://news.ycombinator.com/item?id=48478471)
Tags: [google](https://simonwillison.net/tags/google), [ai](https://simonwillison.net/tags/ai), [generative-ai](https://simonwillison.net/tags/generative-ai), [llms](https://simonwillison.net/tags/llms), [nvidia](https://simonwillison.net/tags/nvidia), [pelican-riding-a-bicycle](https://simonwillison.net/tags/pelican-riding-a-bicycle), [gemma](https://simonwillison.net/tags/gemma), [llm-release](https://simonwillison.net/tags/llm-release), [llm-performance](https://simonwillison.net/tags/llm-performance)
source & further reading
simonwillison.net — original article
OpenAI WebRTC Audio Session, now with document context
Quoting Andrew Singleton
Claude Fable is relentlessly proactive