Last week when Moonshot AI announced Kimi K3, it also opened more of the infrastructure behind the model. One of those releases was AgentENV, developed with KVCache.ai to run the isolated computer environments used in agentic reinforcement learning.
That release is worth studying on its own. Agentic RL does not train only on text. A coding or computer-use agent acts inside a real operating system, runs tools, changes files, starts services, and sometimes breaks things. Training needs many such environments, strong isolation, fast and resume, and a cheap way to branch one prepared state into several parallel rollouts.
AgentENV arrived with an attention-grabbing claim: incremental snapshots in under 100 milliseconds, even after heavy disk modification. But what exactly completes in those 100 milliseconds? Does a forked microVM immediately own an independent copy of its memory? Where do pages changed by the source VM go before the child starts? We wanted to know whether AgentENV lives up to that headline, so we traced its dirty-page path and measured complete fork-to-first-use latency through 2 GiB of dirty memory.
Today we are going to follow AgentENV from the outside in: first its role as a distributed sandbox platform, then the virtualization concepts underneath it, and finally the detail that dominates its memory-dependent fork cost—how dirty guest pages move from the source VM into the child's immutable memory backing.
AgentENV forks below the guest kernel, so the host does not reconstruct every guest process. That makes process-heavy environments fast to clone. But its current implementation still copies selected dirty guest-memory ranges into a new immutable OverlayBD layer before the fork endpoint returns. Restore is lazy; snapshot capture is not.
What AgentENV is at a high level #
AgentENV is a self-hosted, distributed sandbox runtime for AI agents. Each sandbox is a Firecracker microVM with its own guest kernel. AgentENV can import OCI images, turn them into reusable templates, start isolated sandboxes, and resume them from snapshots, and fork a running sandbox into independent children. Its HTTP API is compatible with E2B clients.
At a high level, AgentENV combines Firecracker microVM isolation with OverlayBD- and ublk
-backed layered storage and reusable snapshots so prepared environments can be d, resumed, and forked efficiently.
The Kimi K3 technical report describes AgentENV as one of several sandbox runtimes used in post-training and evaluation, with the microVM boundary motivated in part by stronger isolation from aggressive agent behavior. The project documentation reports snapshot-backed boot or resume under 50 milliseconds and incremental snapshot operations under 100 milliseconds. Those are project claims for specific internal boundaries; a complete externally observed fork includes more work.
The virtualization ideas needed to understand the fork #
Before we delve into AgentENV's design, we need to briefly recap the virtualization concepts that determine what a microVM fork must capture and what can be restored lazily.
1. What is a microVM, and what does Firecracker do?
A microVM is a deliberately small virtual machine. Like a conventional VM, it runs its own guest kernel and uses hardware virtualization for isolation. Unlike a general-purpose desktop VM, it exposes a minimal virtual-hardware model and is optimized to start quickly with low per-VM overhead.
Firecracker is the virtual machine monitor, or VMM, underneath AgentENV. It uses Linux KVM to run guest vCPUs, allocates the host memory that appears as physical RAM to the guest, implements a small set of virtual devices, and exposes an API for configuring, pausing, snapshotting, restoring, and resuming a microVM. AgentENV orchestrates Firecracker and adds the surrounding template, storage-layer, sandbox-networking, scheduling, and lifecycle machinery needed for an agent platform.
2. Guest RAM is host memory managed by a VMM
Inside the microVM, Linux believes it owns a range of physical memory. On the host, Firecracker represents that guest-physical memory with a virtual-memory mapping. A guest page is therefore also backed by some host page: anonymous memory for a fresh VM, or a page from a snapshot-backed file when a VM is restored.
3. A VM snapshot is several coordinated artifacts
A complete Firecracker snapshot is not one magic blob. It coordinates:
- vCPU, KVM, and supported virtual-device state;
- guest-memory contents;
- the guest's block-device state, which the integrator manages separately.
The guest process tree does not need a separate host-side serializer. Process tables, virtual-memory areas, open files, signals, and scheduler state already live inside guest-kernel memory. This is why microVM fork scales much more gently with guest process count than a CRIU-based container fork.
4. Dirty tracking answers “which pages changed?”
After a base snapshot, most guest pages remain unchanged. Dirty-page tracking records pages written since that baseline. A diff snapshot can store only those pages and inherit everything else from earlier layers. This changes capture work from “copy the entire configured VM memory” to “copy the selected changed ranges.”
There is still an important distinction between tracking a dirty page and publishing its contents. A bitmap or range list tells us which addresses changed. A child also needs stable bytes for those addresses after the source resumes and changes again.
5. File-backed restore can be lazy
Firecracker snapshot restore can map a memory backing file with MAP_PRIVATE
. The child does not copy the complete file into fresh RAM at startup. When a vCPU first touches a page, Linux faults it from the host page cache or backing storage. When the child writes, Linux creates a private anonymous copy.
immutable memory image
↓ MAP_PRIVATE
child reads shared page-cache pages on demand
↓ first child write
private anonymous CoW page
Lazy restore makes child startup fast and lets clones share hot host page-cache pages. It does not tell us how the newest dirty bytes entered the immutable memory image. That happens earlier.
How AgentENV handles dirty memory during fork #
AgentENV's current default enables its direct-OverlayBD memory path. The word “direct” means it avoids first asking Firecracker to write an intermediate raw mem.bin
. It does not mean the child directly maps the source VM's live anonymous pages.
The active flow is:
** the source microVM.** Guest vCPUs stop so memory, device, and disk state describe one consistent point.Save VM state. Firecracker writes vCPU, KVM, and emulated-device state tovm_state.bin
.Obtain dirty guest-memory ranges. AgentENV asks Firecracker which memory ranges changed relative to the parent snapshot.Read the selected bytes from the source. AgentENV usesprocess_vm_readv
to copy those ranges from the source Firecracker process.Create a new immutable memory layer. The page data and address index are written to a temporary OverlayBD commit file, then sealed by renaming it tomem_overlaybd/overlaybd.commit
.Stack it over prior memory layers. A smallmem_image.json
describes the new layer plus its inherited parents.Expose the complete image through Multiple sandboxes can reference the same read-only layered device.ublk
.Start the child Firecracker process. Firecracker receives the device as a file memory backend, maps it privately, and faults pages on demand.Let the branches diverge. Child writes become private anonymous CoW pages while the shared snapshot layers remain immutable.
source Firecracker anonymous memory
↓ process_vm_readv(selected dirty ranges)
new immutable OverlayBD memory layer
↓ stack over parent layers
shared read-only ublk device
↓ MAP_PRIVATE
child Firecracker faults pages lazily
AgentENV does not eagerly read the whole snapshot into the child, but it does synchronously materialize selected dirty source pages into an immutable backing layer. Capture cost grows with dirty bytes; restore cost is deferred to page faults.
The commit layer is a regular host filesystem file, not a purely in-memory object. Its newly written contents can remain hot in the host page cache, so local first use does not necessarily require an SSD write followed by an SSD read. Physical persistence, remote upload, and cold faults are separate costs. But the memory copy from source pages into the layer still consumes memory bandwidth, CPU time, and filesystem work before the child has stable backing.
The filesystem path behaves differently. Guest disk writes already accumulate in a writable OverlayBD upper layer while the VM is running. At snapshot time, AgentENV can seal that layer and open a fresh upper rather than rediscovering and recopying every modified disk block. This helps explain how snapshot time can remain low under heavy disk modification while still growing with dirty memory.
How AgentENV performs in practice #
We tested the complete AgentENV fork-to-first-use boundary on the same host across five fresh sources per point. The first series used a 1 GiB guest from zero through 512 MiB of requested dirty allocation. Extending to 2 GiB required a 4 GiB guest, so we repeated 512 MiB as a bridge and kept the two guest configurations visually separate.
The result was clear. In the 1 GiB guest, median first use grew from 360 milliseconds with no requested dirty allocation to 642 milliseconds at 512 MiB. In the 4 GiB extension, it grew from 715 milliseconds at 512 MiB to 1.75 seconds at 2 GiB.
The extended fit was approximately:
AgentENV first-use latency
≈ 381.6 ms + 0.6728 ms × dirty MiB
R² = 0.996
The fork API return had nearly the same slope, while the gap from API return to first successful child command stayed around 80 milliseconds. That places most of the incremental high-memory cost before the fork endpoint returns, not in later child page faults. In other words, the experiment sees the source-to-layer materialization described by the code path.
This does not contradict AgentENV's narrower under-100-millisecond incremental-snapshot claim. Our number is a complete external operation: source , VM and memory state capture, rootfs and memory-layer publication, source resume, child creation, and first use. The boundaries answer different questions. The experiment does show why one advertised number cannot describe every workload: dirty-memory-per-fork is a first-order variable.
Could the dirty-page copy be avoided? #
It is not a fundamental law that a microVM child must first receive a file containing copied dirty pages. It is a reliability and lifecycle choice in AgentENV's current file-backed architecture.
The classic operating-system alternative is direct copy-on-write sharing. Linux fork()
does not serialize the parent's anonymous pages into a file before the child can run. Parent and child page tables initially reference the same physical pages as read-only CoW mappings. A write on either side allocates one private copy.
source physical page
↙ ↘
source mapping child mapping
↓ first write on either side
one private copy of that page
A microVM service could pursue similar semantics with a source-backed pager, shared sealed memory object, kernel-supported bulk clone, or a VMM architecture that preserves old page generations while the source continues. The child could start first and materialize durable snapshot bytes later.
The hard part is preserving the properties the immutable layer currently provides:
- the child must keep seeing fork-time bytes after the source resumes and writes;
- the child should not depend forever on the source Firecracker process remaining alive;
- memory generations need clear ownership, reference counting, and cleanup;
- migration or remote restart eventually requires self-contained or remotely accessible backing;
- snapshot files and cloned identity, entropy, network, and device state must remain secure and consistent.
So direct CoW can remove byte copying from the local-use critical path, but it does not erase durability work. It shifts materialization later and makes the live sharing relationship more complex. That trade is attractive when low-latency local branching matters more than immediate independent persistence.
How TClone applies direct CoW to a live container #
TClone chooses a different system boundary. Instead of treating the guest machine as the unit of fork, it reconstructs a Linux process tree in a sibling container. That costs more as process, VMA, file-descriptor, and namespace counts grow because CRIU still has to understand those resources.
For memory, however, TClone's locally usable asynchronous path shares the source's resident pages directly. Anonymous memory and supported file/page-cache state begin shared and diverge through CoW. Checkpoint serialization can proceed after the child becomes usable rather than forcing every dirty byte through an immutable memory file first.
In our latest ten-process run, direct TClone latency increased from 531 milliseconds with no requested dirty allocation to 897 milliseconds at 2 GiB. The fitted slope was 0.177 milliseconds per MiB—roughly one quarter of the retained 4 GiB-guest AgentENV first-use slope.
Memory accounting supported the mechanism: at 2 GiB, the child reported about 2 GiB of RSS but roughly half that PSS, while shared-dirty memory tracked the payload size. The child was mapping the source's resident pages, not allocating a separate private 2 GiB copy before return.
This is not a universal TClone victory. AgentENV remained much flatter as process count grew because the guest kernel keeps process state encoded in VM memory. TClone trades a lower dirty-byte cost for semantic process reconstruction. AgentENV trades a lower process-count cost for synchronous changed-memory publication. The better boundary depends on workload shape and on whether “done” means locally usable, host-flushed, or remotely durable.
| Dimension | AgentENV | TClone |
|---|---|---|
| Fork boundary | Complete Firecracker microVM | Live Linux container process tree |
| Guest/workload process count | Mostly hidden inside guest memory | Host reconstructs resources per process |
| Dirty memory before local use | Selected bytes copied into immutable OverlayBD layer | Resident source pages shared directly through CoW |
| Restore/read path | Layered ublk device, MAP_PRIVATE , lazy faults |
|
| Shared mappings established during restore; writes diverge through CoW | ||
| Durability | Immutable snapshot layers form a natural persistence unit | Can move serialization outside the local-use critical path |
The larger lesson #
AgentENV's design is strong because it makes the common case incremental: reuse immutable bases, track changes, seal filesystem layers, map memory lazily, and let many children share the host page cache. It also exposes the remaining irreducible question for any fork system: which changed state must become independently owned before the child may run?
AgentENV answers “dirty guest memory must enter an immutable layer first.” TClone's asynchronous local-use path answers “the child may initially share live source pages, and persistence can follow.” Neither answer is free. They optimize different contracts.
Want to experiment with live container fork and direct CoW memory sharing?
Explore TClone's kernel and runtime implementation in os4agent, or see how Gensee Crate can use live forks for coding-agent environment branching, experimentation, and rollback.
Frequently asked questions #
What is AgentENV?
AgentENV is a distributed sandbox runtime for AI agents. It runs isolated Firecracker microVMs, imports OCI environments through OverlayBD, supports snapshot-backed , resume, and fork, and exposes an E2B-compatible API.
Does AgentENV copy dirty guest memory during fork?
Yes in the current direct-OverlayBD implementation. AgentENV reads selected dirty guest-memory ranges from the source Firecracker process and writes them into a new immutable OverlayBD memory layer before the fork endpoint returns.
Does the child eagerly read the complete memory snapshot?
No. A read-only ublk
device exposes the layered memory image, Firecracker maps it privately, and pages fault on demand through the host page cache. Child writes become private anonymous CoW pages.
Could AgentENV start the child before copying dirty pages?
Potentially, with a different source-backed CoW or lazy-pager design. That design would need to preserve fork-time bytes while the source continues, survive source failure when required, manage page generations safely, and eventually materialize state for independent durability or migration.
Sources and further reading #
Primary references include the Kimi K3 repository and technical report, the AgentENV repository and documentation, Firecracker's snapshot-support documentation, the AgentENV direct-OverlayBD configuration and snapshot implementation, the TClone paper, and the open-source GenseeAI/os4agent repository.