cd /news/ai-infrastructure/keep-the-agent-warm-how-atom-serves-… · home › topics › ai-infrastructure › article
[ARTICLE · art-171090] src=rocm.blogs.amd.com ↗ pub= topic=ai-infrastructure verified=true sentiment=↑ positive

Keep the Agent Warm: How ATOM Serves AgentX Workloads on AMD Instinct MI355X GPUs

AMD reported that its ROCm-native ATOM inference engine serves Kimi-K3 at up to 17.5K total tokens per second per chip with 56 concurrent sessions across eight AMD Instinct MI355X GPUs, and at a P90 interactivity of 152 tokens per second per user with a single session, on SemiAnalysis's InferenceX AgentX benchmark (run of 2026-10-15, chart as of 2026-10-23). The AgentX benchmark replays the cc-traces-weka-062126 corpus of 393 Claude Code sessions and 98,827 model requests, where median input is 142,016 tokens and median output 444 tokens. ATOM addresses the workload with LMCache history reuse across GPU HBM and CPU DRAM, ReplaySSM speculative-state compaction, DCP MLA KV sharding, PrefillDelayer scheduling that coalesces short prefills, and a persistent Mono decode kernel written in FlyDSL.

by read14 min views4 publishedNov 17, 2026
Keep the Agent Warm: How ATOM Serves AgentX Workloads on AMD Instinct MI355X GPUs
Image: Rocm (auto-discovered)

A coding agent makes many model calls in one session, and each one re-sends almost everything before it: one recorded session makes 198 calls, and call #48 sends 195.5K tokens, only 1.3K of them new. Agentic workloads combine long shared histories and short outputs, interrupted by tool calls and parallel sub-agents. Serving them well takes more than raw GPU compute: it means keeping the agent warm, with its history cached and its decode moving when new requests arrive.

In short. On eight AMD Instinct™ MI355X GPUs, ATOM serves Kimi-K3 at up to 17.5K total tok/s per chip with 56 concurrent sessions (input plus output tokens, cached input included) and a P90 interactivity of 152 tok/s per user with a single session on the SemiAnalysis InferenceX AgentX benchmark (run of 2026-10-15, chart as of 2026-10-23; Figure 9).

This post uses Kimi-K3, a hybrid model with multi-head latent attention (MLA) and Kimi Delta Attention (KDA) layers, as the case study. It runs on ATOM, AMD’s ROCm-native inference engine, and is measured with the AgentX benchmark.

We’ll cover three parts:

  • The AgentX workload and its challenges. Two recorded sessions show long shared histories, bursty sub-agent fan-out and idle gaps. Together they pose three serving challenges: keep history reusable, balance new requests against ongoing decode, and cut per-step cost when batches are small.
  • ATOM optimizations at three levels: memory, scheduling and compute. Memory: reuse history across GPU HBM and CPU DRAM (LMCache), compact speculative state (ReplaySSM) and shard MLA KV across GPUs (DCP). Scheduling: hold and coalesce short prefills (PrefillDelayer) so active agents keep generating. Compute: fuse each decode step into persistent kernels (Mono decode, written in FlyDSL).
  • Results and summary. InferenceX results, deployment settings by concurrency, and a summary that maps each challenge to its technique and measured effect.

AgentX Workload: Real Coding Sessions, Branching Requests# #

AgentX. SemiAnalysis provides AgentX within InferenceX to evaluate serving performance under agentic workload patterns.[1]

Trace dataset. AgentX replays the public cc-traces-weka-062126 corpus of Claude Code sessions: 393 sessions and 98,827 model requests.[2] Median input is 142,016 tokens and median output 444 tokens—long histories, short continuations.[3] Replay keeps each request’s size, shared prefixes, dependencies and timing, but sends synthetic tokens instead of the original code and tool output.[2], [4] Token counts come from the trace’s 64-token blocks, not the Kimi-K3 tokenizer.[2]

Workload characteristics. Following the InferenceX workload overview and AgentX methodology,[1], [4] three patterns recur. Figures 1 and 2 show them in two recorded sessions: Figure 1 in tokens, Figure 2 in time. We picked these sessions because each pattern is easy to see in them; they are examples, not corpus averages, and the corpus-wide statistics are given alongside.

  • Growing, reusable history. Every call re-sends the conversation so far. In Figure 1, call #48 sends 195.5K tokens, and only 1.3K of them are new. If the server still holds the state for the rest,prefix-cache reuse limits prefill—the computation that processes input before generation—to those new tokens.[1], [4]
  • Sub-agent fan-out and dependencies. A parent agent forks sub-agents and waits for their results. A sub-agent may inherit a prefix or start without one: Figure 1 shows two of the three sub-agents forked after call #48, and each starts with a 35.0K-token call that re-sends none of the parent’s history. The initial calls from all three sub-agents are almost identical to one another, so only the first is fully cold. In Figure 2, four sub-agents send 18 requests in 63 s, and the parent waits 118 s for the last one, then resumes 6 s later; these times were recorded against the original model API, not in an ATOM replay. The corpus has42,029 sub-agent requests in 1,697 groups .[2] Benchmark concurrency countslive session trees , not requests, so one session can keep several requests in flight.[1]
  • Idle gaps. Tool runs and user think time leave gaps between calls, such as the parent’s 79 s gap in Figure 2. AgentX replays the recorded gaps, within its idle-gap limits, not the activities behind them.[1], [4]

Figure 1. One coding session in tokens: each call re-sends the whole history and adds only a little new input.[2]

Figure 2. One session tree in time: sub-agents arrive in a burst, and the parent’s cache must survive its wait.[2]

## Challenges: Memory, Scheduling and Compute[#](#challenges-memory-scheduling-and-compute)

### Memory: Keep Reusable History Available[#](#memory-keep-reusable-history-available)

Why long context puts pressure on the cache. Long conversations and overlapping branches compete for the cache capacity left after model weights, runtime buffers and active requests. If useful history is evicted and cannot be recovered from another tier, it must be processed again. Fixed-length, independent-request tests do not expose this cross-request retention pressure. (InferenceX workload overview)

A small hit-rate loss can add thousands of prefill tokens. Here, effective prefix-cache hit rate means the fraction of input tokens reused without recomputing their prefill, including history restored from another tier. For an input of length N, a drop in effective reuse from 96% to 95% (one percentage point) adds approximately N × 1% input tokens to prefill:

  • At the full corpus’s 142,016-token median input length , that is approximately1,420 additional tokens .
  • At its 682,880-token P95 input length —the threshold for roughly the longest 5% of requests, not their average—the same loss adds approximately6,829 tokens .[3]

These are sensitivity calculations on the corpus’s 64-token input-length proxy, not measured cache misses. More prefill tokens do not imply the same proportional change in FLOPs or time to first token (TTFT).

Where that cost lands for one request is simple: a full hit restores the cached prefix and prefills only the uncached suffix; a partial hit resumes from an earlier cached boundary and prefills the rest; a miss prefills the whole input before the first token. The two user-facing metrics in this post follow from it: TTFT, and interactivity, the output tokens per second each user receives (see the InferenceX glossary).

Scheduling: Balance Arrivals and Decode Progress#

Tool results, new branches and cache misses bring new prefills that compete with ongoing decode. Admitting each one immediately keeps interrupting active agents; holding them builds a queue and raises TTFT. The scheduler has to balance the two as load changes. Delays also add up across a session tree: a parent waiting on its sub-agents, as in Figure 2, cannot continue until the last one returns, so a delay on whichever branch returns last delays the whole task.

Compute: Reduce Per-Step Overhead#

A warm prefix avoids recomputation, not attention over history or per-step execution costs:

  • Per-token work. Each generated token still traverses the model, and small decode batches expose kernel-launch overhead and repeated memory traffic.
  • Speculation cost. Speculation adds draft and verification work.

ATOM Optimizations: Memory, Scheduling and Compute# #

ATOM coordinates state and scheduling policies and implements the fused decode kernels; LMCache provides the CPU DRAM tier. Concurrency-specific configurations appear in deployment recommendations.

Scheduling: Coalesce Prefills with PrefillDelayer#

Coalescing policy (scheduling challenge). After each prefill, PrefillDelayer lets a set number of decode passes (not generated tokens) run while it holds new arrivals, so they prefill together instead of pausing decode one at a time. ATOM extends it to TP/DCP. The hold is bounded, but it is not an end-to-end TTFT guarantee. (ATOM #2238)

Figure 7. PrefillDelayer holds new arrivals and prefills them together, so decode streams once instead of twice.

Figure 7 is schematic. In ATOM #2238, enabling the four-pass delay on Kimi-K3 AgentX at C16 cuts P90 inter-token latency (ITL) by 22% (31.6 → 24.6 ms), so active agents get steadier tokens. New arrivals pay for it: median TTFT rises 49% (1,182 → 1,766 ms). The recipe therefore enables the delay from C16 upward, not in the interactive band. (ATOM #2238)

Compute: Fuse Each Decode Step with Mono Decode#

Mono decode for single-session latency (compute challenge). At C1, a speculative step that verifies one request’s seven draft tokens plus one runs the target model over only eight rows. At that size, launch overhead and weight streaming dominate, not arithmetic. Mono decode replaces the chain of small kernels in each eligible KDA layer with two persistent kernels written in FlyDSL and shipped inside ATOM: K1 runs from the attention input through the KDA recurrence, and K2 covers the output projection and the latent MoE, including the tensor-parallel all-reduce; each kernel starts streaming its weights at launch. MLA layers and a few other layers keep the standard path. The design of these FlyDSL mono kernels, and their use in ATOM, vLLM and SGLang, is in FlyDSL Mono Kernels for High-Interactivity LLM Serving in ATOM.[11]

Measured effect. In a single-request test in ATOM #2535 (seven draft tokens, fixed acceptance length, TP8), target-model forward time drops from 22.0 to 16.4 ms against a reference that keeps three KDA projections in BF16. (ATOM #2535)

Figure 8. Mono decode runs each eligible KDA layer as two persistent kernels, with the all-reduce inside.

Where it applies. The first Kimi-K3 mono kernels served decode steps of at most eight rows, which fits one request with seven draft tokens. ATOM #2659 raised the limit to 16 rows and also fused the output projection and MoE of the MLA layers; ATOM #2871 raised it to 64 rows. Steps that include a prefill still take the standard path. The recipe enables Mono decode at C1 only, because the other bands have not yet been measured with it on.

## Results and Deployment Recommendations[#](#results-and-deployment-recommendations)

### Performance and AgentX Deployment[#](#performance-and-agentx-deployment)

Figure 9 shows the throughput–interactivity trade-off reported by InferenceX for Kimi-K3 across hardware/framework combinations. As a rough guide, moving a point up (more tokens per chip) depends mostly on memory, because reused history skips prefill; moving it right (faster responses for each user) depends more on scheduling and per-step compute. This cross-system comparison is not an ablation of individual optimizations.

Figure 9. Kimi-K3 total token throughput per chip versus P90 interactivity across systems. Source: SemiAnalysis InferenceX, updated 2026-10-23.

The Kimi-K3 AgentX recipe[6] gives concurrency-specific starting points for ATOM on 8× AMD Instinct MI355X GPUs, summarized in the table below. All bands use FP8 KV and GPU prefix caching, and Mono decode is on at C1 only. The MI355X/ATOM points in Figure 9 come from the InferenceX run configuration,[8] an ATOM nightly build from 2026-10-13, with FlyDSL mono kernels off at every plotted concurrency, including C1. On that run the KDA state stays in FP16; Mono decode keeps it in FP32. A few environment and quantization settings also differ from the later recipe. The 152 tok/s per user (P90) at C1 and 17.5K tok/s per chip at C56 in this chart therefore do not include Mono decode yet.

In Kimi-K3 AgentX runs at C1 on the same build, turning Mono decode on cuts median ITL from 6.27 to 5.01 ms and raises output throughput by 10.9%. The same FlyDSL mono approach also runs in ATOM for GLM-5.2, DeepSeek-V4.1 and MiniMax-M3.[11]

Serving band Optimization focus Recipe: parallelism and speculation Recipe: cache and scheduling
Interactive, C1/C2/C4 Lowest per-user latency TP8, DCP1, DSpark with 7 draft tokens; synthetic acceptance length (AL) 3.84 GPU prefix cache; no LMCache; ReplaySSM off; Mono decode at C1 only
Mid-concurrency, C8/C12/C14/C16 Steady decode under bursty prefills TP8, DCP8, DSpark with 4 draft tokens; synthetic AL 3.36 LMCache 128 GiB; ReplaySSM on; PrefillDelayer at C16
Throughput, C32–C80 Most retained history and sessions per GB TP8, DCP8, DSpark with 4 draft tokens; synthetic AL 3.36 ReplaySSM on; LMCache 128 GiB at C32–C48, 192 GiB at C56–C80; PrefillDelayer; larger in-flight window

Checkpoint policy. The recipe keeps only prompt-end anchors: --state-checkpoint-interval-tokens -1 turns off the interval ladder (default 8192 tokens), and ATOM_STATE_CHECKPOINT_DEMAND=0 turns off demand checkpoints. (ATOM #2045)

Deployment Checks#

Start a fresh server for each concurrency, and save the model, hardware, software versions and serving configuration for each point. Synthetic acceptance is for performance runs only; turn it off for correctness and task-quality evaluation. Under the target workload, check recomputed prefill, restore failures and kernel dispatch, then test long-context continuation, checkpoint forks and tool use. The bands are recipe choices, not universal thresholds: use these checks to pick an operating point that meets your latency and quality requirements.

Summary# #

In this blog, you explored how ATOM serves agentic coding workloads from the SemiAnalysis InferenceX AgentX benchmark on eight AMD Instinct MI355X GPUs, using Kimi-K3 as the case study. You saw how replayed coding sessions differ from fixed-length request tests: each call re-sends a long, growing history (Figure 1), sub-agents arrive in bursts, and idle gaps separate calls (Figure 2). These patterns pose three serving challenges: keeping history reusable, balancing new prefills against ongoing decode, and cutting per-step cost when batches are small.

You then saw how ATOM answers each challenge. For memory, ATOM shares MLA KV blocks and KDA checkpoints in GPU HBM and extends reuse to CPU DRAM through LMCache, so 94.1% of prompt tokens at C48 are reused instead of prefilled. ReplaySSM compacts speculative KDA state to fit 2.26× as many KV blocks in a seven-draft experiment, and DCP8 stores each MLA KV token on one GPU instead of all eight. For scheduling, PrefillDelayer holds and coalesces new arrivals, cutting P90 ITL by 22% at C16 at the cost of 49% higher median TTFT. For compute, Mono decode fuses each eligible KDA layer into two persistent FlyDSL kernels, cutting target-model forward time from 22.0 to 16.4 ms in a single-request test and median ITL from 6.27 to 5.01 ms in AgentX runs at C1.

On InferenceX, the run of 2026-10-15 measured ATOM on MI355X at 17.5K total tok/s per chip at C56 and 152 tok/s per user (P90) at C1, with FlyDSL mono kernels off. These are operating points of the whole measured stack, not gains attributed to individual techniques. To try these settings, start from the Kimi-K3 AgentX recipe and run the deployment checks against your own traffic.

Looking ahead, we plan to measure Mono decode beyond C1, now that ATOM #2871 supports decode steps of up to 64 rows, and to bring it into the InferenceX runs. We are also working to close the remaining gap to the trace’s 97.2% reuse ceiling and to evaluate SSD as a further cache tier for longer sessions and higher concurrency. Follow the ROCm Blogs for upcoming posts as these results land.

Acknowledgments# #

We thank the ATOM, AITER and FlyDSL teams and the LMCache community for their contributions.

## References[#](#references)

## Disclaimers[#](#disclaimers)

The information presented in this document is for informational purposes only and may contain technical inaccuracies, omissions, and typographical errors. The information contained herein is subject to change and may be rendered inaccurate for many reasons, including but not limited to product and roadmap changes, component and motherboard version changes, new model and/or product releases, product differences between differing manufacturers, software changes, BIOS flashes, firmware upgrades, or the like. Any computer system has risks of security vulnerabilities that cannot be completely prevented or mitigated. AMD assumes no obligation to update or otherwise correct or revise this information. However, AMD reserves the right to revise this information and to make changes from time to time to the content hereof without obligation of AMD to notify any person of such revisions or changes. THIS INFORMATION IS PROVIDED “AS IS.” AMD MAKES NO REPRESENTATIONS OR WARRANTIES WITH RESPECT TO THE CONTENTS HEREOF AND ASSUMES NO RESPONSIBILITY FOR ANY INACCURACIES, ERRORS, OR OMISSIONS THAT MAY APPEAR IN THIS INFORMATION. AMD SPECIFICALLY DISCLAIMS ANY IMPLIED WARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR ANY PARTICULAR PURPOSE. IN NO EVENT WILL AMD BE LIABLE TO ANY PERSON FOR ANY RELIANCE, DIRECT, INDIRECT, SPECIAL, OR OTHER CONSEQUENTIAL DAMAGES ARISING FROM THE USE OF ANY INFORMATION CONTAINED HEREIN, EVEN IF AMD IS EXPRESSLY ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.

Third-party content is licensed to you directly by the third party that owns the content and is not licensed to you by AMD. ALL LINKED THIRD-PARTY CONTENT IS PROVIDED “AS IS” WITHOUT A WARRANTY OF ANY KIND. USE OF SUCH THIRD-PARTY CONTENT IS DONE AT YOUR SOLE DISCRETION AND UNDER NO CIRCUMSTANCES WILL AMD BE LIABLE TO YOU FOR ANY THIRD-PARTY CONTENT. YOU ASSUME ALL RISK AND ARE SOLELY RESPONSIBLE FOR ANY DAMAGES THAT MAY ARISE FROM YOUR USE OF SUCH THIRD-PARTY CONTENT.

AMD, the AMD Arrow logo, AMD Instinct, ROCm, and combinations thereof are trademarks of Advanced Micro Devices, Inc. Other product names used in this publication are for identification purposes only and may be trademarks of their respective companies. © 2026 Advanced Micro Devices, Inc. All rights reserved.

── more in #ai-infrastructure 4 stories · sorted by recency
── more on @amd 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/keep-the-agent-warm-…] indexed:0 read:14min 2026-11-17 · —