Serving Claude Code from DeepInfer on 4x B200: the cache, the tail and the memory ceiling InferenceKey benchmarked DeepSeek's open-source DeepInfer engine on four NVIDIA B200 GPUs serving Anthropic's Claude Code, reporting a 2.55-second median time to first token across 100 synthetic agents with 450,000-token conversations and zero failed requests. The team's prefix cache lifted token read throughput from about 200K to 1.8 million tokens per second, with 96% of tokens served from cache, but p95 TTFT hit 37 seconds as warm turns queued behind cold ones, and a 200-agent run at 500K tokens pushed median TTFT past two minutes. The writeup also documents a silent cache-miss bug in their first adapter, fixed in version 0.3.2, and advises sizing the GPU tier for concurrently active sessions rather than total sessions. If you serve a coding agent from your own GPUs, the model and the engine are now the easy part. DeepSeek trained DeepSeek V4.1 Flash, wrote DeepInfer and released it under Apache 2.0. Anthropic's Claude Code needs only a few environment variables to talk to another endpoint. What decides whether the machine pays off sits between them: a cache, a queue and a memory budget. We ran this on one server with four NVIDIA B200 cards in the inferencekey cloud. We gave 100 synthetic agents about 450,000 tokens the word pieces a model reads of conversation each. The median wait for the first word of an answer was 2.55 seconds, with zero failed requests. Numbers, method and limits are in the full DeepInfer on 4x B200 research post https://inferencekey.com/research/deepinfer-4xb200-one-machine-claude-code . A coding agent resends the whole conversation on every step, and the model must read all of it before replying. A prefix cache keeps the result of that reading for the part already seen, so only the new tail is read. In our runs 96% of the tokens the model read came from the cache. DeepInfer https://github.com/deepseek-ai/DeepInfer leaves the cache for you to build, as its own page says. With no cache, our four cards got through about 200K tokens per second, starting each conversation from zero. The inferencekey cache lifted that to about 1.8 million on the same cards: nine times the read rate. It has two tiers: about 57 million tokens in GPU memory roughly 120 of our conversations and 617 million in host RAM for quiet sessions. Our first adapter, running the engine as four ranks one process per card , served zero cached tokens. It told the cache to drop each entry when the final token came out, yet the engine only closes a sequence on the step after that. Every request was read in full, silently. A small quality gate caught it first. Hide one fact in a long document, ask for it with an empty cache, then ask again with the document cached, and compare. Temperature 0 makes the model pick its most likely word every time, so the two answers should match. The fixed backend 0.3.2 served repeats 95 to 99% from the cache, and five documents of up to 200,000 tokens returned the same correct fact both ways. Run a check like this first and log the cache hit rate per request. A silent miss looks like a slow GPU. Time to first token TTFT is how long an agent waits before the answer starts streaming. The 100-agent run had a 2.55 s median TTFT and still failed our bar: 95% of turns starting within 15 s. That 95th percentile, p95, sat at 37 seconds. Warm turns, whose conversation was already cached, queued behind seven cold ones: brand-new 450K conversations read from zero. Two untested fixes: let cached turns jump the queue, or limit how many fresh sessions start together. Chart p95 TTFT beside the median, split into warm and cold turns. At 500K tokens per conversation, we had 200 agents ask at once. Their conversations overflowed GPU memory and the median TTFT went past two minutes. No request failed, yet nobody could have worked like that. Host RAM only rescues idle sessions. With agents this busy, the cards hold about 120 such conversations at once. Size the GPU tier for sessions active at once, not for the ones that exist. The per-step results for 128K, 500K and 1M conversations https://inferencekey.com/research/deepinfer-4xb200-one-machine-claude-code show what limits each size: compute for new prompts at 128K, GPU memory at 500K. Claude Code asks for a different model name per role, so map every role to the one model you serve: export ANTHROPIC BASE URL=https://api.inferencekey.com/endpoint/