DeepSeek Ultrafast

Baseten released DeepSeek Ultrafast, an inference engine for DeepSeek-V4.1-Flash that reaches 2,060 tokens/s decode on 8 x B300 (TP8) and 1,740 tokens/s on 4 x GB300 (TP4), while prefilling 1,000,000 tokens in 11.7 s (85,000 tokens/s) and 13.2 s (76,000 tokens/s) respectively. The engine supports 1M context length and multi-server KV cache sharing over RDMA, with Redis tracking which replica owns each conversation's cache, which Baseten says removes the need for prefix-aware routing and enables better load balancing. DeepSeek-V4.1-Flash-Fast is available on Baseten Model APIs, and the engine builds for sm_100a (B200) and sm_103a (B300/GB300) GPUs with the CUDA 13 toolkit.

An ultra-fast inference engine for DeepSeek-V4.1-Flash for sm 100a and sm 103a . It supports 1M context length and multi-server KV cache sharing. | | 4 x GB300 TP4 | 8 x B300 TP8 | |---|---|---| | Decode coding | 1,740 tokens/s | 2,060 tokens/s | | Prefill | 1,000,000 tokens in 13.2 s 76,000 tokens/s | 1,000,000 tokens in 11.7 s 85,000 tokens/s | Decode is measured on "implement the tetris game. output HTML code directly - no explanation." Instructions for reproducing the results are at the bottom of this page. Streaming a complete Tetris game, with live decode speed, then playing it. Shown at real-time speed. DeepSeek-V4.1-Flash-Fast is available on Baseten Model APIs https://www.baseten.co/products/model-apis/ . More performance improvements and open source releases are coming. - 4 GB300 sm 103a , 4 or 8 B300 sm 103a or 4 or 8 B200 sm 100a GPUs - The CUDA 13 toolkit - DeepSeek-V4.1-Flash https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash in models/ - A Redis instance for multi-server KV cache coordination sudo apt-get install -y build-essential pkg-config cmake clang libclang-dev \ libibverbs-dev librdmacm-dev libopencv-dev redis-server curl ca-certificates curl -sSf https://sh.rustup.rs | sh -s -- -y --default-toolchain none && . "$HOME/.cargo/env" rustup toolchain install DSUF SM=sm 103a cargo build --release -p deepseek-ultrafast-engine --bin run -p deepseek-ultrafast-server --bin serve The server keeps each conversation's KV cache between turns. Redis records which replica owns each conversation's cache; locally it only needs to be running. The server finds the machine's RDMA ports itself: redis-server --bind 127.0.0.1 --save '' --appendonly no --daemonize yes DSUF REDIS URL=redis://127.0.0.1:6379 target/release/serve --ckpt models --gpus 0,1,2,3 --port 8000 On an 8-GPU B300 machine, use all eight --gpus 0,1,2,3,4,5,6,7 ; build with DSUF SM=sm 103a . It is ready when curl localhost:8000/readyz returns 200, about 20 seconds after start. Use your favorite OpenAI-compatible client. For example: DEEPSEEK BASE URL=http://localhost:8000 DEEPSEEK API KEY=none \ npx @deepseek-ai/dsh headless "summarize this repo" Every server keeps recent conversations' KV caches in host memory. When a conversation's next turn lands on a different server, that server fetches the cache directly from the server that holds it, host memory to host memory over RDMA, instead of recomputing the prompt. Redis records which server owns each conversation's cache and announces ownership changes. Thanks to DeepSeek's extreme KV cache compression, the transfers are very fast and can be done on every request. This setup removes the need for prefix-aware routing and enables better load balancing. See deploy/kubernetes.yaml https://github.com/basetenlabs/deepseek-ultrafast/blob/main/deploy/kubernetes.yaml for an example multi-server deployment. Stream the Tetris answer from the server and report the decode speed: python python3 - <<'EOF' import json, time, urllib.request body = { "model": "deepseek-v4.1-flash", "messages": {"role": "user", "content": "implement the tetris game. output HTML code directly - no explanation."} , "thinking": {"type": "disabled"}, "stream": True, "stream options": {"include usage": True}, } req = urllib.request.Request "http://localhost:8000/v1/chat/completions", json.dumps body .encode , {"Content-Type": "application/json"} start, first, tokens = time.time , None, 0 with urllib.request.urlopen req as stream: for line in stream: if not line.startswith b"data: {" : continue chunk = json.loads line 6: if chunk.get "usage" : tokens = chunk "usage" "completion tokens" for choice in chunk "choices" : if text := choice "delta" .get "content" : first = first or time.time print text, end="", flush=True end = time.time print f"\n\n{tokens} tokens at {tokens / end - first :,.0f} tokens/s first token after {first - start:.2f} s " EOF It ends with a line like 5142 tokens at 1,717 tokens/s first token after 0.04 s . We see 1,700-1,770 tokens/s on 4xGB300 and 1,980-2,170 tokens/s on 8xB300 --gpus 0,1,2,3,4,5,6,7 . - deepseek-recipe https://github.com/deepseek-ai/deepseek-recipe : prompt rendering and response parsing - tokenizers https://github.com/huggingface/tokenizers : tokenization - cuDNN https://developer.nvidia.com/cudnn and cuBLAS https://developer.nvidia.com/cublas : the vision encoder - OpenCV https://opencv.org : image decoding - Velo https://github.com/ai-dynamo/velo : KV transfer between replicas over RDMA The kernels were optimized by coding agents that could read the source of FlashInfer https://github.com/flashinfer-ai/flashinfer and SGLang https://github.com/sgl-project/sglang while they worked.