cd /news/ai-infrastructure/evolving-infra-an-inference-framewor… · home › topics › ai-infrastructure › article
[ARTICLE · art-172852] src=github.com ↗ pub= topic=ai-infrastructure verified=true sentiment=↑ positive

Evolving Infra: An inference framework that adapt to dynamic agent workload

MIT HAN Lab's Evolving Infra, an LLM serving system that adapts to agent workloads purely online with no offline profiling, cut mean end-to-end latency by up to 4.1× and P95 time to first token by up to 14× versus NVIDIA Dynamo while reaching up to 2.7× Dynamo's throughput, tested serving coding, math and computer-use agents with GLM-5.2 and Kimi K2.6 on up to 256 GB300 GPUs. The framework adapts through three designs: decoupling compute allocation from KV cache management so a node switches between prefill and decode in seconds without a restart, a cost model that predicts prefill and decode load including sub-agents and context management, and an agent-aware distributed KV manager that evicts KV no agent branch needs while keeping KV agents will reuse. Evolving Infra ships as runtime image docker.io/weili0234/evolving-infra:v0.1.0 for x86_64 (H200, B300) and aarch64 (GB300), with setup and benchmarking driven by coding agents such as Claude Code and Codex following per-task skills.

key takeaway

Evolving Infra's online adaptation — decoupled prefill/decode switching in seconds, a cost model covering sub-agents, and an agent-aware KV manager — delivered up to 4.1× lower mean end-to-end latency, 14× lower P95 time to first token and 2.7× throughput versus NVIDIA Dynamo on up to 256 GB300 GPUs

read5 min views3 publishedNov 19, 2026
Evolving Infra: An inference framework that adapt to dynamic agent workload
Image: Michielbdejong (auto-discovered)

<sub>Corresponding author: hao.kang.sys@gmail.com</sub>

Tip

This is an agent-native repository. Use your coding agent (Claude Code, Codex, ...) to understand, deploy, benchmark and develop Evolving Infra. It reads AGENTS.md for the rules and the map of the code, and follows a skill for each task.

demo.mp4 #

Evolving Infra is an LLM serving system that adapts itself to the agent workload purely online, with no offline profiling. Serving coding, math and computer-use agents with GLM-5.2 and Kimi K2.6 on up to 256 GB300 GPUs, it reduces mean end-to-end latency by up to 4.1× and P95 time to first token by up to 14× compared with NVIDIA Dynamo, with up to 2.7× its throughput.

<sub>Math (Putnam), coding (Claude Code) and computer-use (OSWorld) agents, three concurrency levels each. Lower is better.</sub>

Evolving Infra adapts online through three designs:

  1. Decouple compute allocation from KV cache management, so online adaptation is cheap: a node switches between prefill and decodein seconds, without a restart , and its KV cache stays in a host-memory pool that every node can read.
  2. Predict the prefill and decode load of agents with acost model that coverssub-agents and context management . The planner uses it to match the prefill/decode split to the workload, and the router to place each request.
  3. Manage KV with anagent-aware distributed KV manager : signals from agent context management drive KV transfer and eviction, so itevicts the KV that no agent branch needs, such as histories replaced by compaction and the context of finished sub-agents, andkeeps the KV that agents will reuse.

The router sends each request to a prefill node and a decode node. The planner moves nodes between the two groups (dashed arrow). KV stays in shared host memory while an agent branch still uses it. Architecture describes the request path.

Evolving Infra is set up and run by your coding agent (Claude Code, Codex, ...): each task below is a skill that the agent follows step by step, checking each step and reporting what it ran. Every component is in the runtime image, built for x86_64 (H200, B300) and aarch64 (GB300): docker.io/weili0234/evolving-infra:v0.1.0.

  1. Deploy the model. Ask your agent: "Deploy<model> on<hosts> with Evolving Infra, followingskills/deploy-stack/SKILL.md ." Give it the model directory, the hosts and how commands reach them (a shell on each, or a scheduler). It inspects the machines, writes a config when the model or GPU has no example, starts the stack and checks that it answers. For a coding-agent harness, have it use the config ofexamples/agents for your model, which adds the tool-call parsers.
  2. Connect the agent. Ask your agent:
  • Pi, Claude Code or Codex : "Run<harness> on our Evolving Infra stack at<router url> , followingskills/connect-harness/SKILL.md ." Each has a launch script; the harness's own settings do not change.
  • Any other harness (OpenCode, Aider, an in-house CLI, ...): "Add a launch script for<harness> , followingskills/add-harness/SKILL.md ." It finds which API the harness sends, writesexamples/agents/<harness>.sh and checks it with one prompt and one tool call.
  • Your own agent's code : "Connect our agent to Evolving Infra, followingskills/connect-agent/SKILL.md ." Its requests then carry agent identities, so the KV cache of its live branches is kept.

Ask your agent: "Reproduce the paper's Claude Code result on our Slurm cluster, following skills/run-benchmark/SKILL.md." Give it the workload (Claude Code, Putnam or OSWorld), the Slurm account and partition, the image, the model directory and a directory every node sees. A run holds nine GPU nodes for about two hours. The workloads and the paper's results are in benchmarks/.

When a run does not start, fails a request or scores below the reference, ask it to follow skills/diagnose-run/SKILL.md.

Directory What it contains
evolving_infra/engine/ Serving engine, derived from SGLang ( decoupled_engine );what we changed
evolving_infra/router/core/ Rust router ( sgl-router )
evolving_infra/router/frontend/ ,evolving_infra/router/indexer/ ,evolving_infra/router/client/ Tokenizer service, KV directory, Python client
evolving_infra/planner/ Planner
evolving_infra/launch/ ,examples/ evolving-infra serve ; a config per hardware and model, and how to run one under Slurm or on Modal
AGENTS.md , skills/ Instructions and skills for coding agents
benchmarks/ Agent trace replay, run checks, reference results
docker/ Runtime image

We welcome contributions, and we recommend making them with a coding agent:

  • Use Claude Opus 5.5, GPT-6 Astra or a stronger model.
  • Point the agent at AGENTS.md : the contribution rules, the repository map and the commands to build and test.
  • Before opening a pull request, have the agent run the check-pr skill. It checks the change and fills the pull request template: claims, the evidence for each, and the root cause of a fix.
  • Attach the agent's session transcript to your pull request, so reviewers can see how the change was made and checked. Scrub it first: python tools/scrub_transcript.py <session file> replaces tokens, e-mail addresses, home directories and the names you give it, and reports any string that still looks like a secret.
  • For a larger change, open an issue first.

Join the InferenceRSI community to discuss the project, on Discord or in the WeChat group:

If you use Evolving Infra, please cite the paper:

@misc{kang2026evolvinginfraagentadaptivellm,
  title={Evolving Infra: Agent-Adaptive LLM Serving at Scale},
  author={Hao Kang and Weili Xu and Ziyang Li and Zheng Du and Hanchen Li and Tian Xia and Song Bian and Tushar Krishna and Huizi Mao and Ion Stoica and Song Han},
  year={2026},
  eprint={2611.12304},
  archivePrefix={arXiv},
  primaryClass={cs.DC},
  url={https://arxiv.org/abs/2611.12304},
}

MIT (LICENSE). The engine (evolving_infra/engine/) and the router core (evolving_infra/router/core/) are derived from SGLang at upstream commit 9a489f8d2f and, with Trie (benchmarks/trie/), stay under the Apache License 2.0 in their own LICENSE files; NOTICE lists every included third-party work.

── more in #ai-infrastructure 4 stories · sorted by recency
── more on @mit han lab 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/evolving-infra-an-in…] indexed:0 read:5min 2026-11-19 · —