{"slug": "evolving-infra-an-inference-framework-that-adapt-to-dynamic-agent-workload", "title": "Evolving Infra: An inference framework that adapt to dynamic agent workload", "summary": "MIT HAN Lab's Evolving Infra, an LLM serving system that adapts to agent workloads purely online with no offline profiling, cut mean end-to-end latency by up to 4.1× and P95 time to first token by up to 14× versus NVIDIA Dynamo while reaching up to 2.7× Dynamo's throughput, tested serving coding, math and computer-use agents with GLM-5.2 and Kimi K2.6 on up to 256 GB300 GPUs. The framework adapts through three designs: decoupling compute allocation from KV cache management so a node switches between prefill and decode in seconds without a restart, a cost model that predicts prefill and decode load including sub-agents and context management, and an agent-aware distributed KV manager that evicts KV no agent branch needs while keeping KV agents will reuse. Evolving Infra ships as runtime image docker.io/weili0234/evolving-infra:v0.1.0 for x86_64 (H200, B300) and aarch64 (GB300), with setup and benchmarking driven by coding agents such as Claude Code and Codex following per-task skills.", "body_md": "<sub>Corresponding author: [hao.kang.sys@gmail.com](mailto:hao.kang.sys@gmail.com)</sub>\n\nTip\n\n**This is an agent-native repository.** Use your coding agent (Claude Code, Codex, ...) to\nunderstand, deploy, benchmark and develop Evolving Infra. It reads [`AGENTS.md`](https://github.com/mit-han-lab/EvolvingInfra/blob/main/AGENTS.md) for\nthe rules and the map of the code, and follows a [skill](https://github.com/mit-han-lab/EvolvingInfra/blob/main/skills/README.md) for each task.\n\n## demo.mp4\n\nEvolving Infra is an LLM serving system that adapts itself to the agent workload **purely online**,\nwith **no offline profiling**. Serving coding, math and computer-use agents with GLM-5.2 and Kimi K2.6\non up to 256 GB300 GPUs, it reduces mean end-to-end latency by up to **4.1×** and P95 time to\nfirst token by up to **14×** compared with [NVIDIA Dynamo](https://github.com/ai-dynamo/dynamo),\nwith up to **2.7×** its throughput.\n\n<sub>Math (Putnam), coding (Claude Code) and computer-use (OSWorld) agents, three concurrency levels each. Lower is better.</sub>\n\nEvolving Infra adapts online through three designs:\n\n1. **Decouple** compute allocation from KV cache management, so online adaptation is cheap: a node\nswitches between prefill and decode**in seconds, without a restart** , and its KV cache stays in\na host-memory pool that every node can read.\n2. **Predict** the prefill and decode load of agents with a**cost model** that covers**sub-agents and context management** . The planner uses it to match the prefill/decode split to\nthe workload, and the router to place each request.\n3. **Manage KV** with an**agent-aware distributed KV manager** : signals from agent context\nmanagement drive KV transfer and eviction, so it**evicts** the KV that no agent branch needs,\nsuch as histories replaced by compaction and the context of finished sub-agents, and**keeps** the KV that agents will reuse.\n\nThe router sends each request to a prefill node and a decode node. The planner moves nodes\nbetween the two groups (dashed arrow). KV stays in shared host memory while an agent branch still\nuses it. [Architecture](https://github.com/mit-han-lab/EvolvingInfra/blob/main/docs/architecture.md) describes the request path.\n\nEvolving Infra is set up and run by your coding agent (Claude Code, Codex, ...): each task below\nis a [skill](https://github.com/mit-han-lab/EvolvingInfra/blob/main/skills/README.md) that the agent follows step by step, checking each step and\nreporting what it ran. Every component is in the runtime image, built for x86_64 (H200, B300)\nand aarch64 (GB300): `docker.io/weili0234/evolving-infra:v0.1.0`.\n\n1. **Deploy the model.** Ask your agent: \"Deploy`<model>` on`<hosts>` with Evolving Infra,\nfollowing`skills/deploy-stack/SKILL.md` .\" Give it the model directory, the hosts and how\ncommands reach them (a shell on each, or a scheduler). It inspects the machines, writes a\nconfig when the model or GPU has no example, starts the stack and checks that it answers. For\na coding-agent harness, have it use the config of[`examples/agents`](https://github.com/mit-han-lab/EvolvingInfra/blob/main/examples/agents/README.md) for your model, which adds the tool-call parsers.\n2. **Connect the agent.** Ask your agent:\n  - **Pi, Claude Code or Codex** : \"Run`<harness>` on our Evolving Infra stack at`<router url>` , following`skills/connect-harness/SKILL.md` .\" Each has a launch script; the\nharness's own settings do not change.\n  - **Any other harness** (OpenCode, Aider, an in-house CLI, ...): \"Add a launch script for`<harness>` , following`skills/add-harness/SKILL.md` .\" It finds which API the harness\nsends, writes`examples/agents/<harness>.sh` and checks it with one prompt and one tool call.\n  - **Your own agent's code** : \"Connect our agent to Evolving Infra, following`skills/connect-agent/SKILL.md` .\" Its requests then carry agent identities, so the KV cache\nof its live branches is kept.\n\nAsk your agent: \"Reproduce the paper's Claude Code result on our Slurm cluster, following\n`skills/run-benchmark/SKILL.md`.\" Give it the workload (Claude Code, Putnam or OSWorld), the\nSlurm account and partition, the image, the model directory and a directory every node sees. A\nrun holds nine GPU nodes for about two hours. The workloads and the paper's results are in\n[benchmarks/](https://github.com/mit-han-lab/EvolvingInfra/blob/main/benchmarks/README.md).\n\nWhen a run does not start, fails a request or scores below the reference, ask it to follow\n`skills/diagnose-run/SKILL.md`.\n\n| Directory | What it contains | \n|---|---|\n| [`evolving_infra/engine/`](https://github.com/mit-han-lab/EvolvingInfra/blob/main/evolving_infra/engine/README.md) | Serving engine, derived from SGLang ( `decoupled_engine` );[what we changed](https://github.com/mit-han-lab/EvolvingInfra/blob/main/evolving_infra/engine/CHANGES.md) | \n| [`evolving_infra/router/core/`](https://github.com/mit-han-lab/EvolvingInfra/blob/main/evolving_infra/router/core/README.md) | Rust router ( `sgl-router` ) | \n| [`evolving_infra/router/frontend/`](https://github.com/mit-han-lab/EvolvingInfra/blob/main/evolving_infra/router/frontend/README.md) ,[`evolving_infra/router/indexer/`](https://github.com/mit-han-lab/EvolvingInfra/blob/main/evolving_infra/router/indexer/README.md) ,[`evolving_infra/router/client/`](https://github.com/mit-han-lab/EvolvingInfra/blob/main/evolving_infra/router/client/__init__.py) | Tokenizer service, KV directory, Python client | \n| [`evolving_infra/planner/`](https://github.com/mit-han-lab/EvolvingInfra/blob/main/evolving_infra/planner/README.md) | Planner | \n| [`evolving_infra/launch/`](https://github.com/mit-han-lab/EvolvingInfra/blob/main/evolving_infra/launch/__main__.py) ,[`examples/`](https://github.com/mit-han-lab/EvolvingInfra/blob/main/examples/README.md) | `evolving-infra serve` ; a config per hardware and model, and how to run one under Slurm or on Modal | \n| [`AGENTS.md`](https://github.com/mit-han-lab/EvolvingInfra/blob/main/AGENTS.md) ,[` skills/`](https://github.com/mit-han-lab/EvolvingInfra/blob/main/skills) | Instructions and skills for coding agents | \n| [`benchmarks/`](https://github.com/mit-han-lab/EvolvingInfra/blob/main/benchmarks/README.md) | Agent trace replay, run checks, reference results | \n| [`docker/`](https://github.com/mit-han-lab/EvolvingInfra/blob/main/docker/README.md) | Runtime image | \n\nWe welcome contributions, and we recommend making them with a coding agent:\n\n- Use Claude Opus 5.5, GPT-6 Astra or a stronger model.\n- Point the agent at [`AGENTS.md`](https://github.com/mit-han-lab/EvolvingInfra/blob/main/AGENTS.md) : the contribution rules, the repository map and\nthe commands to build and test.\n- Before opening a pull request, have the agent run the\n[`check-pr`](https://github.com/mit-han-lab/EvolvingInfra/blob/main/skills/check-pr/SKILL.md) skill. It checks the change and fills\nthe pull request template: claims, the evidence for each, and the root cause of a fix.\n- Attach the agent's session transcript to your pull request, so\nreviewers can see how the change was made and checked. Scrub it first:\n`python tools/scrub_transcript.py <session file>` replaces tokens, e-mail addresses, home\ndirectories and the names you give it, and reports any string that still looks like a secret.\n- For a larger change, open an issue first.\n\nJoin the InferenceRSI community to discuss the project, on [Discord](https://discord.gg/UEffTj99N) or in the WeChat group:\n\nIf you use Evolving Infra, please cite the [paper](https://arxiv.org/abs/2611.12304):\n\n```\n@misc{kang2026evolvinginfraagentadaptivellm,\n  title={Evolving Infra: Agent-Adaptive LLM Serving at Scale},\n  author={Hao Kang and Weili Xu and Ziyang Li and Zheng Du and Hanchen Li and Tian Xia and Song Bian and Tushar Krishna and Huizi Mao and Ion Stoica and Song Han},\n  year={2026},\n  eprint={2611.12304},\n  archivePrefix={arXiv},\n  primaryClass={cs.DC},\n  url={https://arxiv.org/abs/2611.12304},\n}\n```\n\nMIT ([LICENSE](https://github.com/mit-han-lab/EvolvingInfra/blob/main/LICENSE)). The engine (`evolving_infra/engine/`) and the router core\n(`evolving_infra/router/core/`) are derived from SGLang at upstream commit `9a489f8d2f` and, with\nTrie (`benchmarks/trie/`), stay under the Apache License 2.0 in their own LICENSE files;\n[NOTICE](https://github.com/mit-han-lab/EvolvingInfra/blob/main/NOTICE) lists every included third-party work.", "url": "https://wpnews.pro/news/evolving-infra-an-inference-framework-that-adapt-to-dynamic-agent-workload", "canonical_source": "https://github.com/mit-han-lab/EvolvingInfra", "published_at": "2026-11-19 20:56:54+00:00", "updated_at": "2026-11-19 21:15:02.783555+00:00", "lang": "en", "topics": ["ai-infrastructure", "ai-agents", "large-language-models", "mlops", "ai-tools"], "entities": ["MIT HAN Lab", "Evolving Infra", "NVIDIA Dynamo", "GLM-5.2", "Kimi K2.6", "Claude Code", "Codex", "GB300"], "also_reported_by": [], "takeaway": "Evolving Infra's online adaptation — decoupled prefill/decode switching in seconds, a cost model covering sub-agents, and an agent-aware KV manager — delivered up to 4.1× lower mean end-to-end latency, 14× lower P95 time to first token and 2.7× throughput versus NVIDIA Dynamo on up to 256 GB300 GPUs", "recommendation": null, "alternates": {"html": "https://wpnews.pro/news/evolving-infra-an-inference-framework-that-adapt-to-dynamic-agent-workload", "markdown": "https://wpnews.pro/news/evolving-infra-an-inference-framework-that-adapt-to-dynamic-agent-workload.md", "text": "https://wpnews.pro/news/evolving-infra-an-inference-framework-that-adapt-to-dynamic-agent-workload.txt", "jsonld": "https://wpnews.pro/news/evolving-infra-an-inference-framework-that-adapt-to-dynamic-agent-workload.jsonld"}}