Baseten open-sourced DeepSeek Ultrafast on November 13, 2026: a Rust and CUDA inference engine built specifically for DeepSeek-V4.1-Flash on NVIDIA sm_100a and sm_103a GPUs. The GitHub repository publishes tables claiming 2,060 tokens per second decode on 8×B300, million-token prefill in 11.7 seconds, and multi-server KV cache sharing coordinated through Redis and RDMA.
If you operate frontier open models in production, this is not a minor version bump. It is a from-scratch serving stack in Rust with an OpenAI-compatible /v1/chat/completions surface, aimed at the same checkpoint family we cover in DeepSeek’s full application API suite and NVIDIA’s Snapshot checkpoint restore work. The README is unusually reproducible: build flags, Redis setup, a Python streaming script, and expected log line ranges.
TL;DR #
| Question | Answer |
|---|---|
| Model supported | DeepSeek-V4.1-Flash (local models/ checkpoint) |
| Languages | Rust server + CUDA kernels |
| Claimed 8×B300 decode | 2,060 tok/s on Tetris HTML coding prompt |
| Claimed 4×GB300 decode | 1,740 tok/s |
| 1M-token prefill (8×B300) | 11.7 s (~85,000 tok/s) |
| Multi-server KV | Redis metadata + Velo RDMA host memory transfer |
| API | OpenAI-compatible on port 8000 |
| Hosted option | Baseten Model APIs (per README) |
What Baseten shipped #
The repository landed with three commits and a focused README — no sprawling monorepo. Baseten positions DeepSeek Ultrafast as inference that skips Python in the hot path: cargo build --release produces serve and run binaries after you install CUDA 13 toolchain pieces and IB verbs libraries.
Performance table (from README). Decode numbers use the prompt "implement the tetris game. output HTML code directly - no explanation." with thinking disabled. The README also reports step time and acceptance length from speculative decoding blocks — about 5.6–5.7 tokens per step on Blackwell configs. That matters for agents: long HTML generations are exactly the workloads where constant tokens-per-second dominates wall clock.
Scale-out story. Multi-server mode is the differentiator versus a single-node vLLM pod. When a conversation’s next request hits a different replica, the engine is supposed to pull KV cache host-to-host over RDMA instead of re-prefilling. Redis only stores ownership metadata; payloads never round-trip through object storage. The README includes a Kubernetes sample under deploy/kubernetes.yaml.
Lineage. Acknowledgments say CUDA kernels were optimized with agents that read FlashInfer and SGLang source — a meta twist consistent with compiler-agent inference tuning stories elsewhere this month.
DeepSeek Ultrafast Rust inference engine serving V4.1-Flash on NVIDIA B300 GPUs with Redis KV coordination
How to try the documented quick start #
The README assumes a Linux host with 4 or 8 eligible GPUs. These steps paraphrase the repo; always read the current README before production.
Build
rustup toolchain install
DSUF_SM=sm_103a cargo build --release -p deepseek-ultrafast-engine --bin run -p deepseek-ultrafast-server --bin serve
Use sm_100a on B200 per the environment flag docs.
Single-server serve
redis-server --bind 127.0.0.1 --save '' --appendonly no --daemonize yes
DSUF_REDIS_URL=redis://127.0.0.1:6379 target/release/serve --ckpt models --gpus 0,1,2,3 --port 8000
Wait until curl localhost:8000/readyz returns 200 (~20 seconds after start per README).
Client
DEEPSEEK_BASE_URL=http://localhost:8000 DEEPSEEK_API_KEY=none \
npx @deepseek-ai/dsh headless " summarize this repo "
Reproduce the Tetris throughput script
The README embeds a Python urllib streaming client that prints completion tokens per second. Baseten says 30 runs on 4×GB300 land around 1,700–1,770 tok/s, and 8×B300 around 1,980–2,170 tok/s, with average_iteration_ms and average_accept_length logged per request.
Cost reality check. These numbers require B300 or GB300 class hardware and CUDA 13. They do not translate to a single RTX card. Compare instead to other Blackwell-specific efforts like TensorFold on GB10 or DeepSeek DeepInfer — different models, different goals.
KV cache sharing in plain language #
Most serving stacks either sticky-route conversations to one GPU or eat a full re-prefill when load balancing moves a session. Baseten’s design keeps compressed KV in host DRAM on the replica that last saw the session. Redis publishes which host owns conversation X. The next replica issues an RDMA pull from that host’s memory.
For agent builders running long system prompts + tool transcripts, that pattern cuts repeated prefill cost when you horizontally scale chat workers. It also introduces ops complexity: Redis availability, RDMA fabric health, and strict version compatibility across replicas. The README’s Kubernetes manifest is the starting point for anyone comparing against SGLang consortium production paths.
Multi-server KV cache sharing for DeepSeek V4.1-Flash inference across RDMA-linked GPU nodes
What developers are asking #
Is 2,060 tok/s verified independently? Not by explainx.ai. We did not run 8×B300. The claim lives in Baseten’s README and reproduction script. Treat it like any vendor benchmark: same prompt, same temperature, same hardware SKU — change one variable and throughput moves.
Does this replace vLLM? Only if your entire fleet standardizes on one MoE checkpoint and you control Blackwell images. Teams mixing dozens of models still need general servers; see our vLLM vs SGLang comparison.
How does it relate to DeepSeek’s own APIs? DeepSeek hosts cloud APIs with different SLOs and pricing in our pricing post. Ultrafast is for self-hosted V4.1-Flash on your metal or Baseten’s paid API tier.
What about security? A Rust server reduces Python supply-chain surface in the daemon, but you still trust weight files, Redis, and network exposure on port 8000. Put it behind auth and private networking like any internal inference endpoint.
Comparison anchors #
| Stack | Scope | Best when |
|---|---|---|
| DeepSeek Ultrafast | V4.1-Flash, Rust, Blackwell-tuned | You own B300/GB300 racks and want max tok/s on one model |
| vLLM | Many models, Python ecosystem | Default production serving with broad HF support |
| SGLang | MoE + speculative decoding at scale | You want RadixAttention and consortium integrations |
| llama.cpp | GGUF, edge and laptop | Local dev, not 8×B300 clusters |
Baseten’s engine is closest in spirit to purpose-built repos like Titanic Flash on Spark clusters — chase throughput on a flagship open weight model by owning the whole stack.
Reading Baseten’s throughput table like an operator #
Decode tokens per second is a single-number summary of a pipeline that includes draft acceptance, KV bandwidth, weight layout, and whether the Tetris prompt is mostly HTML boilerplate. Baseten’s README is honest about the prompt: it is a coding emit task with thinking disabled — representative for agents that dump UI, not for multi-tool ReAct loops with retrieval.
When you compare against NVIDIA’s published vLLM numbers for V4.1-Flash, line up four knobs:
| Knob | Why it matters |
|---|---|
| Tensor parallel width | 4× vs 8× changes all-reduce overhead |
| Speculative acceptance length | Higher accept length inflates “tokens per step” |
| Precision | FP8/BF16 mix changes memory bandwidth |
| Prompt class | Code vs prose vs 128K context rewrites the curve |
If you self-host, log average_iteration_ms and average_accept_length from the server’s request_end line — Baseten cites 2.69–2.74 ms step times on 8×B300. Without those fields in your logs, you cannot reproduce their table.
Production checklist before you expose port 8000 #
- Pin weights — checksum
models/and restrict write access; supply-chain attacks on checkpoints are a real incident class. - Isolate Redis — bind to localhost or a VPC security group; conversation metadata is still sensitive.
- Network policy — expose the server only inside your mesh; pair withBedrock-style per-principal budgets if you meter internal tenants.
- Capacity plan — 2,060 tok/s on eight B300s does not mean 2,060 × concurrent users; batching and KV transfer latency steal headroom.
- Fallback path — keep vLLM or SGLang warm for models Ultrafast does not support so agents do not hard-fail.
Teams running coding agents should also read Claude Code observability changes: when you swap inference backends, re-baseline latency and cost per merged PR.
Failure modes we expect (even if the README is silent) #
- Redis loss — ownership metadata desyncs; replicas may re-prefill or return stale KV unless the client retries with session affinity.
- RDMA misconfiguration — silent fallback to slow paths can crater multi-replica wins; verify NIC flags listed in the README’s
libibverbsdependencies. - Checkpoint mismatch — wrong
DSUF_SMbuild vs GPU SKU fails at load with opaque CUDA errors; treatrust-toolchain.tomlas part of the release artifact. - License/API abuse — hosting Ultrafast as a public API without DeepSeek license compliance is on you, not Baseten’s repo.
Document incidents the way we recommend for white-house incident disclosure expectations: if your agent hurts a downstream system while backed by Ultrafast, your org owns notification.
What to watch next #
The README promises more performance improvements and open-source releases. Worth tracking: whether DeepSeek-V4.1-Flash-Fast on Baseten APIs matches self-hosted numbers, whether multi-replica KV survives real agent traffic with tool loops, and whether kernels merge upstream concepts into SGLang or stay a fork.
If you benchmark, publish hardware SKU, driver, prompt hash, and power cap. The AI inference discourse drowns in single-number tweets; structured receipts help everyone.
Researchers shipping cyber evals should also note Vals AI’s DeepSeek cyber ranges: faster inference changes red-team economics, but it does not automatically make open-weight models safer to expose on the public internet.
Hosted vs self-hosted: If you lack eight B300s, Baseten’s Model APIs tier may be the realistic way to touch Ultrafast speeds; treat the GitHub repo as reference firmware for what “fast” means on Blackwell, not as a laptop install guide. Budget a full day to reproduce their Tetris script before you promise leadership a production cutover date.
Related reading #
Benchmarks and hardware requirements reflect Baseten’s README as of November 13, 2026.