{"slug": "baseten-deepseek-ultrafast-rust-engine-hits-2060-tok-s-on-v4-1-flash", "title": "Baseten DeepSeek Ultrafast: Rust Engine Hits 2,060 tok/s on V4.1-Flash", "summary": "Baseten open-sourced DeepSeek Ultrafast on November 13, 2026, a Rust and CUDA inference engine built for DeepSeek-V4.1-Flash on NVIDIA sm_100a and sm_103a GPUs, with the GitHub repository claiming 2,060 tokens per second decode on 8×B300 and a 1M-token prefill in 11.7 seconds. The engine serves an OpenAI-compatible /v1/chat/completions API on port 8000 and coordinates multi-server KV cache sharing through Redis metadata plus RDMA host-to-host transfer, avoiding re-prefill when a request lands on a different replica. Baseten's README reports 1,740 tok/s decode on 4×GB300 and about 5.6–5.7 accepted tokens per speculative decoding step, and includes build flags, a Kubernetes sample and a Python streaming client for reproduction.", "body_md": "**Baseten open-sourced DeepSeek Ultrafast on November 13, 2026**: a Rust and CUDA inference engine built specifically for **DeepSeek-V4.1-Flash** on NVIDIA **sm_100a** and **sm_103a** GPUs. The [GitHub repository](https://github.com/basetenlabs/deepseek-ultrafast?ref=explainx) publishes tables claiming **2,060 tokens per second** decode on **8×B300**, **million-token prefill in 11.7 seconds**, and **multi-server KV cache sharing** coordinated through Redis and RDMA.\n\nIf you operate frontier open models in production, this is not a minor version bump. It is a **from-scratch serving stack** in Rust with an OpenAI-compatible `/v1/chat/completions` surface, aimed at the same checkpoint family we cover in [DeepSeek’s full application API suite](https://explainx.ai/blog/deepseek-full-suite-application-apis-web-search-file-url-asr-tts-pricing-2026) and [NVIDIA’s Snapshot checkpoint restore work](https://explainx.ai/blog/nvidia-snapshot-gpu-checkpoint-restore-kubernetes-dsx-2026). The README is unusually reproducible: build flags, Redis setup, a Python streaming script, and expected log line ranges.\n\n## TL;DR\n\n| Question | Answer | \n|---|---|\n| Model supported | DeepSeek-V4.1-Flash (local `models/` checkpoint) | \n| Languages | Rust server + CUDA kernels | \n| Claimed 8×B300 decode | **2,060 tok/s** on Tetris HTML coding prompt | \n| Claimed 4×GB300 decode | **1,740 tok/s** | \n| 1M-token prefill (8×B300) | **11.7 s** (~85,000 tok/s) | \n| Multi-server KV | Redis metadata + **Velo** RDMA host memory transfer | \n| API | OpenAI-compatible on port **8000** | \n| Hosted option | **Baseten Model APIs** (per README) | \n\n## What Baseten shipped\n\nThe repository landed with three commits and a focused README — no sprawling monorepo. Baseten positions **DeepSeek Ultrafast** as inference that skips Python in the hot path: `cargo build --release` produces `serve` and `run` binaries after you install CUDA 13 toolchain pieces and IB verbs libraries.\n\n**Performance table (from README).** Decode numbers use the prompt *\"implement the tetris game. output HTML code directly - no explanation.\"* with thinking disabled. The README also reports **step time** and **acceptance length** from speculative decoding blocks — about **5.6–5.7 tokens per step** on Blackwell configs. That matters for agents: long HTML generations are exactly the workloads where constant tokens-per-second dominates wall clock.\n\n**Scale-out story.** Multi-server mode is the differentiator versus a single-node vLLM pod. When a conversation’s next request hits a different replica, the engine is supposed to **pull KV cache host-to-host over RDMA** instead of re-prefilling. Redis only stores ownership metadata; payloads never round-trip through object storage. The README includes a **Kubernetes sample** under `deploy/kubernetes.yaml`.\n\n**Lineage.** Acknowledgments say CUDA kernels were optimized with agents that read **[FlashInfer](https://github.com/flashinfer-ai/flashinfer?ref=explainx)** and **SGLang** source — a meta twist consistent with [compiler-agent inference tuning stories](https://explainx.ai/blog/together-compiler-agents-optimize-inference-1111-tps-minimax-1205-tps-deepseek-claimed-vs-verified-2026) elsewhere this month.\n\nDeepSeek Ultrafast Rust inference engine serving V4.1-Flash on NVIDIA B300 GPUs with Redis KV coordination\n\n## How to try the documented quick start\n\nThe README assumes a Linux host with **4 or 8** eligible GPUs. These steps paraphrase the repo; always read the current README before production.\n\n### Build\n\n```\nrustup toolchain install\nDSUF_SM=sm_103a cargo build --release -p deepseek-ultrafast-engine --bin run -p deepseek-ultrafast-server --bin serve\n```\n\nUse `sm_100a` on B200 per the environment flag docs.\n\n### Single-server serve\n\n```\nredis-server --bind 127.0.0.1 --save '' --appendonly no --daemonize yes\nDSUF_REDIS_URL=redis://127.0.0.1:6379 target/release/serve --ckpt models --gpus 0,1,2,3 --port 8000\n```\n\nWait until `curl localhost:8000/readyz` returns **200** (~20 seconds after start per README).\n\n### Client\n\n```\nDEEPSEEK_BASE_URL=http://localhost:8000 DEEPSEEK_API_KEY=none \\\nnpx @deepseek-ai/dsh headless \" summarize this repo \"\n```\n\n### Reproduce the Tetris throughput script\n\nThe README embeds a Python `urllib` streaming client that prints completion tokens per second. Baseten says **30 runs** on 4×GB300 land around **1,700–1,770 tok/s**, and 8×B300 around **1,980–2,170 tok/s**, with `average_iteration_ms` and `average_accept_length` logged per request.\n\n**Cost reality check.** These numbers require **B300 or GB300 class** hardware and CUDA 13. They do not translate to a single RTX card. Compare instead to other Blackwell-specific efforts like [TensorFold on GB10](https://explainx.ai/blog/tensorfold-zig-local-llm-inference-metal-cuda-api-2026) or [DeepSeek DeepInfer](https://explainx.ai/blog/deepseek-deepinfer-open-source-inference-engine-v4-1-flash-155-trillion-tokens-claimed-vs-verified-2026) — different models, different goals.\n\n## KV cache sharing in plain language\n\nMost serving stacks either **sticky-route** conversations to one GPU or **eat a full re-prefill** when load balancing moves a session. Baseten’s design keeps **compressed KV in host DRAM** on the replica that last saw the session. Redis publishes which host owns conversation `X`. The next replica issues an **RDMA pull** from that host’s memory.\n\nFor agent builders running **long system prompts + tool transcripts**, that pattern cuts repeated prefill cost when you horizontally scale chat workers. It also introduces **ops complexity**: Redis availability, RDMA fabric health, and strict version compatibility across replicas. The README’s Kubernetes manifest is the starting point for anyone comparing against [SGLang consortium production paths](https://explainx.ai/blog/sglang-consortium-lmsys-inference-summit-november-2026).\n\nMulti-server KV cache sharing for DeepSeek V4.1-Flash inference across RDMA-linked GPU nodes\n\n## What developers are asking\n\n**Is 2,060 tok/s verified independently?** Not by explainx.ai. We did not run 8×B300. The claim lives in Baseten’s README and reproduction script. Treat it like any vendor benchmark: same prompt, same temperature, same hardware SKU — change one variable and throughput moves.\n\n**Does this replace vLLM?** Only if your entire fleet standardizes on **one MoE checkpoint** and you control Blackwell images. Teams mixing dozens of models still need general servers; see our [vLLM vs SGLang comparison](https://explainx.ai/blog/vllm-vs-sglang-inference-server-comparison-2026).\n\n**How does it relate to DeepSeek’s own APIs?** DeepSeek hosts cloud APIs with different SLOs and pricing in [our pricing post](https://explainx.ai/blog/deepseek-full-suite-application-apis-web-search-file-url-asr-tts-pricing-2026). Ultrafast is for **self-hosted** V4.1-Flash on your metal or Baseten’s paid API tier.\n\n**What about security?** A Rust server reduces Python supply-chain surface in the daemon, but you still trust **weight files**, **Redis**, and **network exposure** on port 8000. Put it behind auth and private networking like any internal inference endpoint.\n\n## Comparison anchors\n\n| Stack | Scope | Best when | \n|---|---|---|\n| **DeepSeek Ultrafast** | V4.1-Flash, Rust, Blackwell-tuned | You own B300/GB300 racks and want max tok/s on one model | \n| **vLLM** | Many models, Python ecosystem | Default production serving with broad HF support | \n| **SGLang** | MoE + speculative decoding at scale | You want RadixAttention and consortium integrations | \n| **llama.cpp** | GGUF, edge and laptop | Local dev, not 8×B300 clusters | \n\nBaseten’s engine is closest in spirit to **purpose-built** repos like [Titanic Flash on Spark clusters](https://explainx.ai/blog/glm-5-3-two-dgx-stations-vllm-tensor-parallel-753b-nvfp4-607-tokens-per-second-guide-2026) — chase throughput on a flagship open weight model by owning the whole stack.\n\n## Reading Baseten’s throughput table like an operator\n\nDecode tokens per second is a **single-number summary** of a pipeline that includes draft acceptance, KV bandwidth, weight layout, and whether the Tetris prompt is mostly HTML boilerplate. Baseten’s README is honest about the prompt: it is a **coding emit** task with thinking disabled — representative for agents that dump UI, not for multi-tool ReAct loops with retrieval.\n\nWhen you compare against [NVIDIA’s published vLLM numbers for V4.1-Flash](https://explainx.ai/blog/nvidia-snapshot-gpu-checkpoint-restore-kubernetes-dsx-2026), line up four knobs:\n\n| Knob | Why it matters | \n|---|---|\n| Tensor parallel width | 4× vs 8× changes all-reduce overhead | \n| Speculative acceptance length | Higher accept length inflates “tokens per step” | \n| Precision | FP8/BF16 mix changes memory bandwidth | \n| Prompt class | Code vs prose vs 128K context rewrites the curve | \n\nIf you self-host, log **`average_iteration_ms`** and **` average_accept_length`** from the server’s `request_end` line — Baseten cites **2.69–2.74 ms** step times on 8×B300. Without those fields in your logs, you cannot reproduce their table.\n\n## Production checklist before you expose port 8000\n\n1. **Pin weights** — checksum`models/` and restrict write access; supply-chain attacks on checkpoints are a real incident class.\n2. **Isolate Redis** — bind to localhost or a VPC security group; conversation metadata is still sensitive.\n3. **Network policy** — expose the server only inside your mesh; pair with[Bedrock-style per-principal budgets](https://explainx.ai/blog/track-and-cap-amazon-bedrock-spend-per-iam-principal-cur-2-0-aws-budgets-deny-policy-guide-2026) if you meter internal tenants.\n4. **Capacity plan** — 2,060 tok/s on eight B300s does not mean 2,060 × concurrent users; batching and KV transfer latency steal headroom.\n5. **Fallback path** — keep vLLM or SGLang warm for models Ultrafast does not support so agents do not hard-fail.\n\nTeams running coding agents should also read [Claude Code observability changes](https://explainx.ai/blog/claude-code-2-1-323-removes-subagentstatusline-agent-panel-observability-2026): when you swap inference backends, re-baseline latency and cost per merged PR.\n\n## Failure modes we expect (even if the README is silent)\n\n- **Redis loss** — ownership metadata desyncs; replicas may re-prefill or return stale KV unless the client retries with session affinity.\n- **RDMA misconfiguration** — silent fallback to slow paths can crater multi-replica wins; verify NIC flags listed in the README’s`libibverbs` dependencies.\n- **Checkpoint mismatch** — wrong`DSUF_SM` build vs GPU SKU fails at load with opaque CUDA errors; treat`rust-toolchain.toml` as part of the release artifact.\n- **License/API abuse** — hosting Ultrafast as a public API without DeepSeek license compliance is on you, not Baseten’s repo.\n\nDocument incidents the way we recommend for [white-house incident disclosure expectations](https://explainx.ai/blog/white-house-mandates-ai-incident-disclosure-super-intelligence-force-anthropic-2026): if your agent hurts a downstream system while backed by Ultrafast, your org owns notification.\n\n## What to watch next\n\nThe README promises **more performance improvements and open-source releases**. Worth tracking: whether **DeepSeek-V4.1-Flash-Fast** on Baseten APIs matches self-hosted numbers, whether **multi-replica KV** survives real agent traffic with tool loops, and whether kernels merge upstream concepts into SGLang or stay a fork.\n\nIf you benchmark, publish **hardware SKU, driver, prompt hash, and power cap**. The AI inference discourse drowns in single-number tweets; structured receipts help everyone.\n\nResearchers shipping cyber evals should also note [Vals AI’s DeepSeek cyber ranges](https://explainx.ai/blog/vals-ai-cyberoffense-gpt-6-sol-daybreak-deepseek-v4-1-flash-open-weight-cyber-range-2026): faster inference changes red-team economics, but it does not automatically make open-weight models safer to expose on the public internet.\n\n**Hosted vs self-hosted:** If you lack eight B300s, Baseten’s **Model APIs** tier may be the realistic way to touch Ultrafast speeds; treat the GitHub repo as reference firmware for what “fast” means on Blackwell, not as a laptop install guide. Budget a full day to reproduce their Tetris script before you promise leadership a production cutover date.\n\n## Related reading\n\n*Benchmarks and hardware requirements reflect Baseten’s README as of November 13, 2026.*", "url": "https://wpnews.pro/news/baseten-deepseek-ultrafast-rust-engine-hits-2060-tok-s-on-v4-1-flash", "canonical_source": "https://explainx.ai/blog/baseten-deepseek-ultrafast-2060-tps-rust-inference-engine-2026?utm_source=rss&utm_medium=feed&utm_campaign=blog", "published_at": "2026-11-13 00:00:00+00:00", "updated_at": "2026-11-13 23:22:53.049433+00:00", "lang": "en", "topics": ["ai-infrastructure", "large-language-models", "ai-tools", "mlops", "developer-tools"], "entities": ["Baseten", "DeepSeek Ultrafast", "DeepSeek-V4.1-Flash", "NVIDIA", "B300", "GB300", "Redis", "FlashInfer"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/baseten-deepseek-ultrafast-rust-engine-hits-2060-tok-s-on-v4-1-flash", "markdown": "https://wpnews.pro/news/baseten-deepseek-ultrafast-rust-engine-hits-2060-tok-s-on-v4-1-flash.md", "text": "https://wpnews.pro/news/baseten-deepseek-ultrafast-rust-engine-hits-2060-tok-s-on-v4-1-flash.txt", "jsonld": "https://wpnews.pro/news/baseten-deepseek-ultrafast-rust-engine-hits-2060-tok-s-on-v4-1-flash.jsonld"}}