{"slug": "deepseek-ultrafast", "title": "DeepSeek Ultrafast", "summary": "Baseten released DeepSeek Ultrafast, an inference engine for DeepSeek-V4.1-Flash that reaches 2,060 tokens/s decode on 8 x B300 (TP8) and 1,740 tokens/s on 4 x GB300 (TP4), while prefilling 1,000,000 tokens in 11.7 s (85,000 tokens/s) and 13.2 s (76,000 tokens/s) respectively. The engine supports 1M context length and multi-server KV cache sharing over RDMA, with Redis tracking which replica owns each conversation's cache, which Baseten says removes the need for prefix-aware routing and enables better load balancing. DeepSeek-V4.1-Flash-Fast is available on Baseten Model APIs, and the engine builds for sm_100a (B200) and sm_103a (B300/GB300) GPUs with the CUDA 13 toolkit.", "body_md": "An ultra-fast inference engine for **DeepSeek-V4.1-Flash** for `sm_100a` and `sm_103a`. It supports 1M context length and multi-server KV cache sharing.\n\n|  | 4 x GB300 (TP4) | 8 x B300 (TP8) | \n|---|---|---|\n| **Decode** (coding) | **1,740 tokens/s** | **2,060 tokens/s** | \n| **Prefill** | **1,000,000 tokens in 13.2 s** (76,000 tokens/s) | **1,000,000 tokens in 11.7 s** (85,000 tokens/s) | \n\nDecode is measured on *\"implement the tetris game. output HTML code directly - no explanation.\"* Instructions for reproducing the results are at the bottom of this page.\n\n*Streaming a complete Tetris game, with live decode speed, then playing it. Shown at real-time speed.*\n\nDeepSeek-V4.1-Flash-Fast is available on [Baseten Model APIs](https://www.baseten.co/products/model-apis/). More performance improvements and open source releases are coming.\n\n- 4 GB300 (`sm_103a` ), 4 or 8 B300 (`sm_103a` ) or 4 or 8 B200 (`sm_100a` ) GPUs\n- The CUDA 13 toolkit\n- [DeepSeek-V4.1-Flash](https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash) in`models/`\n- A Redis instance for multi-server KV cache coordination\n\n```\nsudo apt-get install -y build-essential pkg-config cmake clang libclang-dev \\\n    libibverbs-dev librdmacm-dev libopencv-dev redis-server curl ca-certificates\ncurl -sSf https://sh.rustup.rs | sh -s -- -y --default-toolchain none && . \"$HOME/.cargo/env\"\nrustup toolchain install\nDSUF_SM=sm_103a cargo build --release -p deepseek-ultrafast-engine --bin run -p deepseek-ultrafast-server --bin serve\n```\n\nThe server keeps each conversation's KV cache between turns. Redis records which replica owns each conversation's cache; locally it only needs to be running. The server finds the machine's RDMA ports itself:\n\n```\nredis-server --bind 127.0.0.1 --save '' --appendonly no --daemonize yes\n\nDSUF_REDIS_URL=redis://127.0.0.1:6379 target/release/serve --ckpt models --gpus 0,1,2,3 --port 8000\n```\n\nOn an 8-GPU B300 machine, use all eight (`--gpus 0,1,2,3,4,5,6,7`; build with `DSUF_SM=sm_103a`).\n\nIt is ready when `curl localhost:8000/readyz` returns 200, about 20 seconds after start.\n\nUse your favorite OpenAI-compatible client. For example:\n\n```\nDEEPSEEK_BASE_URL=http://localhost:8000 DEEPSEEK_API_KEY=none \\\n    npx @deepseek-ai/dsh headless \"summarize this repo\"\n```\n\nEvery server keeps recent conversations' KV caches in host memory. When a conversation's next turn lands on a different server, that server fetches the cache directly from the server that holds it, host memory to host memory over RDMA, instead of recomputing the prompt. Redis records which server owns each conversation's cache and announces ownership changes.\n\nThanks to DeepSeek's extreme KV cache compression, the transfers are very fast and can be done on every request.\n\nThis setup removes the need for prefix-aware routing and enables better load balancing.\n\nSee [deploy/kubernetes.yaml](https://github.com/basetenlabs/deepseek-ultrafast/blob/main/deploy/kubernetes.yaml) for an example multi-server deployment.\n\nStream the Tetris answer from the server and report the decode speed:\n\n``` python\npython3 - <<'EOF'\nimport json, time, urllib.request\n\nbody = {\n    \"model\": \"deepseek-v4.1-flash\",\n    \"messages\": [{\"role\": \"user\", \"content\": \"implement the tetris game. output HTML code directly - no explanation.\"}],\n    \"thinking\": {\"type\": \"disabled\"},\n    \"stream\": True,\n    \"stream_options\": {\"include_usage\": True},\n}\nreq = urllib.request.Request(\"http://localhost:8000/v1/chat/completions\", json.dumps(body).encode(),\n                             {\"Content-Type\": \"application/json\"})\nstart, first, tokens = time.time(), None, 0\nwith urllib.request.urlopen(req) as stream:\n    for line in stream:\n        if not line.startswith(b\"data: {\"):\n            continue\n        chunk = json.loads(line[6:])\n        if chunk.get(\"usage\"):\n            tokens = chunk[\"usage\"][\"completion_tokens\"]\n        for choice in chunk[\"choices\"]:\n            if text := choice[\"delta\"].get(\"content\"):\n                first = first or time.time()\n                print(text, end=\"\", flush=True)\nend = time.time()\nprint(f\"\\n\\n{tokens} tokens at {tokens / (end - first):,.0f} tokens/s (first token after {first - start:.2f} s)\")\nEOF\n```\n\nIt ends with a line like `5142 tokens at 1,717 tokens/s (first token after 0.04 s)`.\n\nWe see 1,700-1,770 tokens/s on 4xGB300 and 1,980-2,170 tokens/s on 8xB300 (`--gpus 0,1,2,3,4,5,6,7`).\n\n- [deepseek-recipe](https://github.com/deepseek-ai/deepseek-recipe) : prompt rendering and response parsing\n- [tokenizers](https://github.com/huggingface/tokenizers) : tokenization\n- [cuDNN](https://developer.nvidia.com/cudnn) and[cuBLAS](https://developer.nvidia.com/cublas) : the vision encoder\n- [OpenCV](https://opencv.org) : image decoding\n- [Velo](https://github.com/ai-dynamo/velo) : KV transfer between replicas over RDMA\n\nThe kernels were optimized by coding agents that could read the source of\n[FlashInfer](https://github.com/flashinfer-ai/flashinfer) and [SGLang](https://github.com/sgl-project/sglang)\nwhile they worked.", "url": "https://wpnews.pro/news/deepseek-ultrafast", "canonical_source": "https://github.com/basetenlabs/deepseek-ultrafast", "published_at": "2026-11-13 17:39:03+00:00", "updated_at": "2026-11-13 17:50:02.690299+00:00", "lang": "en", "topics": ["ai-infrastructure", "large-language-models", "ai-tools", "mlops"], "entities": ["Baseten", "DeepSeek", "DeepSeek-V4.1-Flash", "DeepSeek Ultrafast", "Nvidia B300", "Nvidia GB300", "Redis", "CUDA 13"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/deepseek-ultrafast", "markdown": "https://wpnews.pro/news/deepseek-ultrafast.md", "text": "https://wpnews.pro/news/deepseek-ultrafast.txt", "jsonld": "https://wpnews.pro/news/deepseek-ultrafast.jsonld"}}