Evolving Infra: An inference framework that adapt to dynamic agent workload MIT HAN Lab's Evolving Infra, an LLM serving system that adapts to agent workloads purely online with no offline profiling, cut mean end-to-end latency by up to 4.1× and P95 time to first token by up to 14× versus NVIDIA Dynamo while reaching up to 2.7× Dynamo's throughput, tested serving coding, math and computer-use agents with GLM-5.2 and Kimi K2.6 on up to 256 GB300 GPUs. The framework adapts through three designs: decoupling compute allocation from KV cache management so a node switches between prefill and decode in seconds without a restart, a cost model that predicts prefill and decode load including sub-agents and context management, and an agent-aware distributed KV manager that evicts KV no agent branch needs while keeping KV agents will reuse. Evolving Infra ships as runtime image docker.io/weili0234/evolving-infra:v0.1.0 for x86_64 (H200, B300) and aarch64 (GB300), with setup and benchmarking driven by coding agents such as Claude Code and Codex following per-task skills.