# Baseten DeepSeek Ultrafast: Rust Engine Hits 2,060 tok/s on V4.1-Flash

> Source: <https://explainx.ai/blog/baseten-deepseek-ultrafast-2060-tps-rust-inference-engine-2026?utm_source=rss&utm_medium=feed&utm_campaign=blog>
> Published: 2026-11-13 00:00:00+00:00

**Baseten open-sourced DeepSeek Ultrafast on November 13, 2026**: a Rust and CUDA inference engine built specifically for **DeepSeek-V4.1-Flash** on NVIDIA **sm_100a** and **sm_103a** GPUs. The [GitHub repository](https://github.com/basetenlabs/deepseek-ultrafast?ref=explainx) publishes tables claiming **2,060 tokens per second** decode on **8×B300**, **million-token prefill in 11.7 seconds**, and **multi-server KV cache sharing** coordinated through Redis and RDMA.

If you operate frontier open models in production, this is not a minor version bump. It is a **from-scratch serving stack** in Rust with an OpenAI-compatible `/v1/chat/completions` surface, aimed at the same checkpoint family we cover in [DeepSeek’s full application API suite](https://explainx.ai/blog/deepseek-full-suite-application-apis-web-search-file-url-asr-tts-pricing-2026) and [NVIDIA’s Snapshot checkpoint restore work](https://explainx.ai/blog/nvidia-snapshot-gpu-checkpoint-restore-kubernetes-dsx-2026). The README is unusually reproducible: build flags, Redis setup, a Python streaming script, and expected log line ranges.

## TL;DR

| Question | Answer | 
|---|---|
| Model supported | DeepSeek-V4.1-Flash (local `models/` checkpoint) | 
| Languages | Rust server + CUDA kernels | 
| Claimed 8×B300 decode | **2,060 tok/s** on Tetris HTML coding prompt | 
| Claimed 4×GB300 decode | **1,740 tok/s** | 
| 1M-token prefill (8×B300) | **11.7 s** (~85,000 tok/s) | 
| Multi-server KV | Redis metadata + **Velo** RDMA host memory transfer | 
| API | OpenAI-compatible on port **8000** | 
| Hosted option | **Baseten Model APIs** (per README) | 

## What Baseten shipped

The repository landed with three commits and a focused README — no sprawling monorepo. Baseten positions **DeepSeek Ultrafast** as inference that skips Python in the hot path: `cargo build --release` produces `serve` and `run` binaries after you install CUDA 13 toolchain pieces and IB verbs libraries.

**Performance table (from README).** Decode numbers use the prompt *"implement the tetris game. output HTML code directly - no explanation."* with thinking disabled. The README also reports **step time** and **acceptance length** from speculative decoding blocks — about **5.6–5.7 tokens per step** on Blackwell configs. That matters for agents: long HTML generations are exactly the workloads where constant tokens-per-second dominates wall clock.

**Scale-out story.** Multi-server mode is the differentiator versus a single-node vLLM pod. When a conversation’s next request hits a different replica, the engine is supposed to **pull KV cache host-to-host over RDMA** instead of re-prefilling. Redis only stores ownership metadata; payloads never round-trip through object storage. The README includes a **Kubernetes sample** under `deploy/kubernetes.yaml`.

**Lineage.** Acknowledgments say CUDA kernels were optimized with agents that read **[FlashInfer](https://github.com/flashinfer-ai/flashinfer?ref=explainx)** and **SGLang** source — a meta twist consistent with [compiler-agent inference tuning stories](https://explainx.ai/blog/together-compiler-agents-optimize-inference-1111-tps-minimax-1205-tps-deepseek-claimed-vs-verified-2026) elsewhere this month.

DeepSeek Ultrafast Rust inference engine serving V4.1-Flash on NVIDIA B300 GPUs with Redis KV coordination

## How to try the documented quick start

The README assumes a Linux host with **4 or 8** eligible GPUs. These steps paraphrase the repo; always read the current README before production.

### Build

```
rustup toolchain install
DSUF_SM=sm_103a cargo build --release -p deepseek-ultrafast-engine --bin run -p deepseek-ultrafast-server --bin serve
```

Use `sm_100a` on B200 per the environment flag docs.

### Single-server serve

```
redis-server --bind 127.0.0.1 --save '' --appendonly no --daemonize yes
DSUF_REDIS_URL=redis://127.0.0.1:6379 target/release/serve --ckpt models --gpus 0,1,2,3 --port 8000
```

Wait until `curl localhost:8000/readyz` returns **200** (~20 seconds after start per README).

### Client

```
DEEPSEEK_BASE_URL=http://localhost:8000 DEEPSEEK_API_KEY=none \
npx @deepseek-ai/dsh headless " summarize this repo "
```

### Reproduce the Tetris throughput script

The README embeds a Python `urllib` streaming client that prints completion tokens per second. Baseten says **30 runs** on 4×GB300 land around **1,700–1,770 tok/s**, and 8×B300 around **1,980–2,170 tok/s**, with `average_iteration_ms` and `average_accept_length` logged per request.

**Cost reality check.** These numbers require **B300 or GB300 class** hardware and CUDA 13. They do not translate to a single RTX card. Compare instead to other Blackwell-specific efforts like [TensorFold on GB10](https://explainx.ai/blog/tensorfold-zig-local-llm-inference-metal-cuda-api-2026) or [DeepSeek DeepInfer](https://explainx.ai/blog/deepseek-deepinfer-open-source-inference-engine-v4-1-flash-155-trillion-tokens-claimed-vs-verified-2026) — different models, different goals.

## KV cache sharing in plain language

Most serving stacks either **sticky-route** conversations to one GPU or **eat a full re-prefill** when load balancing moves a session. Baseten’s design keeps **compressed KV in host DRAM** on the replica that last saw the session. Redis publishes which host owns conversation `X`. The next replica issues an **RDMA pull** from that host’s memory.

For agent builders running **long system prompts + tool transcripts**, that pattern cuts repeated prefill cost when you horizontally scale chat workers. It also introduces **ops complexity**: Redis availability, RDMA fabric health, and strict version compatibility across replicas. The README’s Kubernetes manifest is the starting point for anyone comparing against [SGLang consortium production paths](https://explainx.ai/blog/sglang-consortium-lmsys-inference-summit-november-2026).

Multi-server KV cache sharing for DeepSeek V4.1-Flash inference across RDMA-linked GPU nodes

## What developers are asking

**Is 2,060 tok/s verified independently?** Not by explainx.ai. We did not run 8×B300. The claim lives in Baseten’s README and reproduction script. Treat it like any vendor benchmark: same prompt, same temperature, same hardware SKU — change one variable and throughput moves.

**Does this replace vLLM?** Only if your entire fleet standardizes on **one MoE checkpoint** and you control Blackwell images. Teams mixing dozens of models still need general servers; see our [vLLM vs SGLang comparison](https://explainx.ai/blog/vllm-vs-sglang-inference-server-comparison-2026).

**How does it relate to DeepSeek’s own APIs?** DeepSeek hosts cloud APIs with different SLOs and pricing in [our pricing post](https://explainx.ai/blog/deepseek-full-suite-application-apis-web-search-file-url-asr-tts-pricing-2026). Ultrafast is for **self-hosted** V4.1-Flash on your metal or Baseten’s paid API tier.

**What about security?** A Rust server reduces Python supply-chain surface in the daemon, but you still trust **weight files**, **Redis**, and **network exposure** on port 8000. Put it behind auth and private networking like any internal inference endpoint.

## Comparison anchors

| Stack | Scope | Best when | 
|---|---|---|
| **DeepSeek Ultrafast** | V4.1-Flash, Rust, Blackwell-tuned | You own B300/GB300 racks and want max tok/s on one model | 
| **vLLM** | Many models, Python ecosystem | Default production serving with broad HF support | 
| **SGLang** | MoE + speculative decoding at scale | You want RadixAttention and consortium integrations | 
| **llama.cpp** | GGUF, edge and laptop | Local dev, not 8×B300 clusters | 

Baseten’s engine is closest in spirit to **purpose-built** repos like [Titanic Flash on Spark clusters](https://explainx.ai/blog/glm-5-3-two-dgx-stations-vllm-tensor-parallel-753b-nvfp4-607-tokens-per-second-guide-2026) — chase throughput on a flagship open weight model by owning the whole stack.

## Reading Baseten’s throughput table like an operator

Decode tokens per second is a **single-number summary** of a pipeline that includes draft acceptance, KV bandwidth, weight layout, and whether the Tetris prompt is mostly HTML boilerplate. Baseten’s README is honest about the prompt: it is a **coding emit** task with thinking disabled — representative for agents that dump UI, not for multi-tool ReAct loops with retrieval.

When you compare against [NVIDIA’s published vLLM numbers for V4.1-Flash](https://explainx.ai/blog/nvidia-snapshot-gpu-checkpoint-restore-kubernetes-dsx-2026), line up four knobs:

| Knob | Why it matters | 
|---|---|
| Tensor parallel width | 4× vs 8× changes all-reduce overhead | 
| Speculative acceptance length | Higher accept length inflates “tokens per step” | 
| Precision | FP8/BF16 mix changes memory bandwidth | 
| Prompt class | Code vs prose vs 128K context rewrites the curve | 

If you self-host, log **`average_iteration_ms`** and **` average_accept_length`** from the server’s `request_end` line — Baseten cites **2.69–2.74 ms** step times on 8×B300. Without those fields in your logs, you cannot reproduce their table.

## Production checklist before you expose port 8000

1. **Pin weights** — checksum`models/` and restrict write access; supply-chain attacks on checkpoints are a real incident class.
2. **Isolate Redis** — bind to localhost or a VPC security group; conversation metadata is still sensitive.
3. **Network policy** — expose the server only inside your mesh; pair with[Bedrock-style per-principal budgets](https://explainx.ai/blog/track-and-cap-amazon-bedrock-spend-per-iam-principal-cur-2-0-aws-budgets-deny-policy-guide-2026) if you meter internal tenants.
4. **Capacity plan** — 2,060 tok/s on eight B300s does not mean 2,060 × concurrent users; batching and KV transfer latency steal headroom.
5. **Fallback path** — keep vLLM or SGLang warm for models Ultrafast does not support so agents do not hard-fail.

Teams running coding agents should also read [Claude Code observability changes](https://explainx.ai/blog/claude-code-2-1-323-removes-subagentstatusline-agent-panel-observability-2026): when you swap inference backends, re-baseline latency and cost per merged PR.

## Failure modes we expect (even if the README is silent)

- **Redis loss** — ownership metadata desyncs; replicas may re-prefill or return stale KV unless the client retries with session affinity.
- **RDMA misconfiguration** — silent fallback to slow paths can crater multi-replica wins; verify NIC flags listed in the README’s`libibverbs` dependencies.
- **Checkpoint mismatch** — wrong`DSUF_SM` build vs GPU SKU fails at load with opaque CUDA errors; treat`rust-toolchain.toml` as part of the release artifact.
- **License/API abuse** — hosting Ultrafast as a public API without DeepSeek license compliance is on you, not Baseten’s repo.

Document incidents the way we recommend for [white-house incident disclosure expectations](https://explainx.ai/blog/white-house-mandates-ai-incident-disclosure-super-intelligence-force-anthropic-2026): if your agent hurts a downstream system while backed by Ultrafast, your org owns notification.

## What to watch next

The README promises **more performance improvements and open-source releases**. Worth tracking: whether **DeepSeek-V4.1-Flash-Fast** on Baseten APIs matches self-hosted numbers, whether **multi-replica KV** survives real agent traffic with tool loops, and whether kernels merge upstream concepts into SGLang or stay a fork.

If you benchmark, publish **hardware SKU, driver, prompt hash, and power cap**. The AI inference discourse drowns in single-number tweets; structured receipts help everyone.

Researchers shipping cyber evals should also note [Vals AI’s DeepSeek cyber ranges](https://explainx.ai/blog/vals-ai-cyberoffense-gpt-6-sol-daybreak-deepseek-v4-1-flash-open-weight-cyber-range-2026): faster inference changes red-team economics, but it does not automatically make open-weight models safer to expose on the public internet.

**Hosted vs self-hosted:** If you lack eight B300s, Baseten’s **Model APIs** tier may be the realistic way to touch Ultrafast speeds; treat the GitHub repo as reference firmware for what “fast” means on Blackwell, not as a laptop install guide. Budget a full day to reproduce their Tetris script before you promise leadership a production cutover date.

## Related reading

*Benchmarks and hardware requirements reflect Baseten’s README as of November 13, 2026.*