{"slug": "linux-sandboxes-from-s3-93-99-cheaper-than-always-on-for-ai-agents", "title": "Linux sandboxes from S3. 93-99% cheaper than always on for AI Agents.", "summary": "The open-source Python library exec-sandbox runs untrusted Python, JavaScript or shell code — including code written by AI agents — in QEMU microVMs on a user's own Linux or macOS hosts, with no cloud account or API key required. Its author claims sessions start in under 1 ms from a warm pool and about 50-60 ms from an L1 memory snapshot, and that saved instances written encrypted to S3 make the setup 93-99% cheaper than always-on sandboxes for AI agents. The library requires Python 3.12-3.14, QEMU 11.0+ for L1 memory snapshots and saved instances (8.0+ for the rest), and KVM on Linux or HVF on macOS for roughly 5-8x faster execution.", "body_md": "exec-sandbox runs untrusted Python, JavaScript or shell code in QEMU microVMs on your own Linux or macOS hosts, for example code that an AI agent wrote. It is a Python asyncio library with a command line (`sbx`). It needs no cloud account and no API key.\n\n``` python\nimport asyncio\n\nfrom exec_sandbox import Language, Scheduler, VmSpec\n\nasync def main() -> None:\n    async with Scheduler() as scheduler:\n        # Language.PYTHON, Language.JAVASCRIPT or Language.RAW\n        async with await scheduler.new(VmSpec(language=Language.PYTHON)) as session:\n            result = await session.exec(\"print('Hello from a secure VM!')\")\n            print(result.stdout)  # Hello from a secure VM!\n            print(result.exit_code)  # 0\n\nasyncio.run(main())\n```\n\nThe other Python examples use `async with` and `await` at the top level: run them in an `async def` function, as `main()` above.\n\n- **Start:**[Features](#features) ·[Install](#install) ·[Quick start](#quick-start) ·[Key terms](#key-terms)\n- **Use:**[Sessions](#sessions) ·[Files](#files) ·[Packages](#packages) ·[Network](#network) ·[Saved instances](#saved-instances) ·[Configuration](#configuration) ·[Errors and recovery](#errors-and-recovery)\n- **Operate:**[Performance](#performance) ·[Caches and disk use](#caches-and-disk-use) ·[VM images and offline use](#vm-images-and-offline-use) ·[Linux host setup](#linux-host-setup) ·[Instance store operations](#instance-store-operations)\n- **Reference:**[CLI reference](#cli-reference) ·[API reference](#api-reference) ·[Limits](#limits) ·[Guest environment](#guest-environment)\n- **Internals:**[How it works](#how-it-works) ·[Security](#security) ·[Contributing](#contributing) ·[License](#license)\n\n- **Hardware isolation.** Each session runs in its own QEMU VM, not in a container, with KVM or HVF acceleration. Up to 8 layers of isolation ([Security](#security) ).\n- **Fast start.** Under 1 ms from the warm pool, about 50-60 ms from an L1 memory snapshot. L2 disk snapshots keep the installed packages, and L3 shares them in S3 with all hosts ([Performance](#performance) ).\n- **Self-hosted.** The first use downloads the VM images from GitHub Releases. For hosts with no internet access, use`sbx prefetch` .\n- **Python API and CLI.** Run code once or in several steps, stream stdout and stderr, and read and write files.\n- **Saved instances.**`save()` writes the RAM, the processes and`/home/user` of a VM, encrypted, to S3 or a local folder.`reuse()` continues it later on any worker of the same host kind.\n- **Network control.** No network access by default. Per spec, an allow-list of domains, checked by DNS filtering and TLS SNI inspection.\n- **Memory density.** zram in the guest. On Linux, KSM page merging and memory reclaim of idle warm-pool VMs.\n\n- **Python:** 3.12, 3.13, 3.14 (including free-threaded).\n- **OS:** Linux or macOS, x64 or arm64. On Windows, use WSL2 (on native Windows,`import exec_sandbox` raises`ConfigError`` PLATFORM_UNSUPPORTED` ).\n- **QEMU:** 11.0+ for L1 memory snapshots and saved instances; 8.0+ for the rest. ARM64 without KVM or HVF needs 9.0.4+.\n- **Hardware acceleration:** KVM (Linux) or HVF (macOS), recommended (~5-8x faster).\n- **x86 CPU with KVM or HVF:** x86-64-v3 (Intel Haswell, AMD Zen 1 or newer). On a CPU without it, the`Scheduler` start raises`ConfigError`` HOST_DEPENDENCY` with the missing features. TCG runs on any x86 CPU.\n\n```\nuv add exec-sandbox              # Core library\nuv add \"exec-sandbox[cli]\"       # + the sbx command line and the local instance store\nuv add \"exec-sandbox[s3]\"        # + the L3 snapshot cache and the s3 instance store (and the local store)\n```\n\nWith pip, use `pip install` with the same names. For `sbx` with the `s3` store, install `exec-sandbox[cli,s3]`.\n\n```\nbrew install qemu                             # macOS\napt install qemu-system qemu-utils            # Ubuntu/Debian\ntest -r /dev/kvm -a -w /dev/kvm && echo ok    # Linux: ok means your user can use KVM\nsysctl kern.hv_support                        # macOS: kern.hv_support: 1 means HVF is available\n```\n\nexec-sandbox uses `qemu-system-x86_64` or `qemu-system-aarch64` from `/usr/bin/`, else from `PATH` (`EXEC_SANDBOX_QEMU_BIN_X86` and `EXEC_SANDBOX_QEMU_BIN_ARM` set another file), and `qemu-img` and `qemu-storage-daemon` from `PATH`. A missing QEMU binary, or one whose version or machine types this user cannot read, raises `ConfigError` (`HOST_DEPENDENCY`) when the `Scheduler` starts; a missing `qemu-img` raises it at its first use. Check the version with `/usr/bin/qemu-system-x86_64 --version` (or `qemu-system-aarch64`): Ubuntu 24.04 ships 8.2.2. Saved instances and L1 snapshots need QEMU 11.0 or later: where the distribution ships an older one, build QEMU from its source release ([qemu.org/download](https://www.qemu.org/download/#source)) and set `EXEC_SANDBOX_QEMU_BIN_X86` or `EXEC_SANDBOX_QEMU_BIN_ARM` to the binary.\n\nOn Linux, give your user access to KVM with `sudo usermod -aG kvm $USER`, then log in again. Without this access, exec-sandbox logs a warning and uses TCG. To run QEMU as its own user, with cgroup limits and namespaces, see [Linux host setup](#linux-host-setup).\n\nThe first run downloads the VM images ([How images download](#how-images-download)). For Python, see the example at the top and [Sessions](#sessions). `sbx` comes with the `cli` extra; all its options are in the [CLI reference](#cli-reference).\n\n```\nsbx run 'print(\"Hello from sandbox\")'                                  # Python\nsbx run -l javascript 'console.log(\"Hello from sandbox\")'              # JavaScript\nsbx run script.py                                                      # A file: the extension gives the language\nsbx run -p pandas==3.0.1 'import pandas; print(pandas.__version__)'    # With packages\nsbx run --network --allow-domain api.example.com fetch_data.py         # Network, with an allow-list\nsbx run --instance my-vm -c 'x = 41'                                   # Keep the VM state (Saved instances)\nsbx run --instance my-vm -c 'print(x + 1)'                             # 42\n```\n\n| Term | Meaning | \n|---|---|\n| Spec | A `VmSpec` : what a VM is made of ([VmSpec](#vmspec) ) | \n| REPL | The interpreter in the VM that runs your code | \n| Session | One VM that runs your code: `new()` or`reuse()` gives it,`close()` ends the VM ([Sessions](#sessions) ) | \n| Saved instance | The state of a session that `save()` stored (RAM, processes, files in`/home/user` ):`reuse()` continues it on any worker of its host kind ([Saved instances](#saved-instances) ) | \n| Warm pool | VMs started before the request ( [Warm pool](#warm-pool) ) | \n| L1, L2, L3 | The snapshot cache: memory snapshots, disk snapshots with the packages installed, and a copy of L2 in S3 ( [Snapshot cache](#snapshot-cache) ). It makes new VMs start faster; it holds no saved instance | \n| Instance store | Where saved instances live: your S3 bucket ( `s3` ) or a folder on this host (`local` ) ([Choose a store](#choose-a-store) ) | \n| Generation | The save number of an instance: 1, then +1 for each save | \n| Head | The object in the store that names the current state of an instance | \n| VM images | The kernel, the initramfs and the base disk image of each language ( [VM images and offline use](#vm-images-and-offline-use) ) | \n| Assets | The VM images and the gvproxy-wrapper binary: what `sbx prefetch` downloads | \n| Guest images | The disk images of one VM: the base image and, with packages, the L2 package image | \n| Worker | A process that runs a `Scheduler` , on its host (a machine or a container) | \n| Host kind | What a saved instance must match to load ( [Rules for save and reuse](#rules-for-save-and-reuse) ) | \n| Admission | The wait until the host has the memory, CPU and disk for a VM ( [Limits](#limits) ) | \n| KVM, HVF, TCG | Hardware acceleration on Linux (KVM) and macOS (HVF). TCG is QEMU's software emulation | \n\n`scheduler.new(spec)` starts a new VM and returns its session. The session keeps variables, imports, files and background processes between `exec()` calls ([Limits](#limits)), until `close()` destroys the VM. Some errors close a session ([What closes a session](#what-closes-a-session)).\n\n```\nasync with Scheduler() as scheduler:\n    async with await scheduler.new(VmSpec(language=Language.PYTHON)) as session:\n        await session.exec(\"import math\")\n        await session.exec(\"x = math.pi * 2\")\n        result = await session.exec(\"print(f'{x:.4f}')\")\n        print(result.stdout)  # 6.2832\n        print(session.exec_count)  # 3\n```\n\nJavaScript sessions (also TypeScript) keep variables and functions. Shell sessions (`raw`, Bash) keep environment variables, the working folder and functions.\n\nA session closes after `session_idle_timeout_seconds` (default 300) with no call. For one session, pass `idle_timeout_seconds` (10-3600) to `new()` or `reuse()`; a value out of range raises `InvalidInputError` (`ARGUMENT_INVALID`).\n\nTo get the output while the code runs, pass `on_stdout` and `on_stderr` to `exec()`: `await session.exec(code, on_stdout=lambda chunk: print(chunk, end=\"\"))`. To get the boot output of the kernel, tiny-init and the guest agent, pass `on_boot_log` to `new()`; the VM then always cold boots (no warm pool, no L1). Keep each callback short: a slow callback blocks the event loop.\n\nPaths are relative to `/home/user`, the folder where your code runs.\n\n```\nawait session.write_file(\"input.csv\", b\"name,score\\nAlice,95\")  # From bytes, a local Path or an IO[bytes] stream\nawait session.write_file(\"run.sh\", Path(\"./run.sh\"), make_executable=True)  # Mode 0755 (default 0644)\nawait session.exec(\"open('output.csv', 'w').write(open('input.csv').read().upper())\")\nawait session.read_file(\"output.csv\", destination=Path(\"./output.csv\"))  # To a local Path or an IO[bytes] stream\nfor f in await session.list_files(\"\"):  # \"\" is /home/user\n    print(f.name, f.is_dir, f.size)\nsbx run --upload ./local.csv:input.csv --download output.csv:./result.csv -c \"open('output.csv','w').write(open('input.csv').read().upper())\"\n```\n\nSet `packages` in the spec. The first VM of a spec installs the packages and caches the result; later VMs of that spec start from the cache ([Snapshot cache](#snapshot-cache)). Count and size limits: [Limits](#limits). Runtimes and tools: [Guest environment](#guest-environment).\n\n- Give each package a version: `VmSpec(language=Language.PYTHON, packages=(\"pandas==3.0.1\",))` ,`VmSpec(language=Language.JAVASCRIPT, packages=(\"lodash@4.17.21\",))` . Other operators (`>=` ,`~=` ) pass too;`==` gives a stable cache entry.\n- A package without a version, a bad name, or packages for `raw` raise`InvalidInputError` (`ARGUMENT_INVALID` ). Scoped npm names (`@scope/name` ) are refused.\n- A package outside the allow-list (the top 10,000 of PyPI or npm) raises `InvalidInputError` (`PACKAGE_NOT_ALLOWED` ). To skip this check, set`enable_package_validation=False` (`sbx run --no-validation` ).\n- Python packages install into Python 3.14 on musl with no build step: use packages with pure-Python or musllinux wheels.\n- The install reaches PyPI or npm, whatever the network settings of the spec.\n\n`allow_network=True` turns on outbound access. `allowed_domains` then sets what the guest can reach:\n\n- `None` (default): the package registries of the language (python:`pypi.org` ,`files.pythonhosted.org` ; javascript:`registry.npmjs.org` ).`raw` reaches every host.\n- `()` (empty): every host.\n- A list of domains, such as `(\"httpbin.org\",)` : these domains and their subdomains (`example.com` also allows`api.example.com` ).\n\n``` python\nfrom exec_sandbox import Language, PortMapping\n\n# Guest port 8080 on host port 3000\nspec = VmSpec(language=Language.PYTHON, expose_ports=(PortMapping(internal=8080, external=3000),))\nasync with await scheduler.new(spec) as session:\n    print(session.exposed_ports[0].url)  # http://127.0.0.1:3000\n```\n\n- A fixed host port must be 1024-65535. An int in place of a `PortMapping` (`expose_ports=(8080,)` ) is the guest port: the OS picks the host port.`protocol` is`\"tcp\"` (default) or`\"udp\"` .\n- Ports bind to 127.0.0.1. For access from other machines, put a reverse proxy on the host.\n- Port forwarding works without `allow_network` ; all outbound TCP and UDP then stays blocked.\n- After `reuse()` , the ports open again: a fixed host port must be free on this host, and a dynamic one can change (read`exposed_ports` again).\n\n`save()` writes the state of a VM (RAM, processes, files in `/home/user`), compressed and encrypted, to the instance store. `reuse()` continues it later, on any worker of the same host kind. Every worker needs QEMU 11.0 or later (else the `Scheduler` start fails with `ConfigError` `MIGRATION_UNSUPPORTED`) and a keyring: set `EXEC_SANDBOX_INSTANCE_KEYRING`, or pass `instance_keyring=` in code ([Keys](#keys)).\n\n``` python\nimport os\n\nfrom exec_sandbox import Language, Scheduler, SchedulerConfig, VmSpec\n\nKEYRING = os.environ[\"MY_KEYRING\"]  # from your secret store: see Keys\nconfig = SchedulerConfig(instance_store=\"local\", instance_keyring=KEYRING)  # \"s3\" for several workers\nasync with Scheduler(config) as scheduler:\n    async with await scheduler.new(VmSpec(language=Language.PYTHON)) as session:\n        await session.exec(\"x = 41\")\n        ref = await session.save()  # InstanceRef(instance_id=..., generation=1)\n    async with await scheduler.reuse(ref.instance_id) as session:  # Later, with the same keyring\n        result = await session.exec(\"x += 1; print(x)\")\n        print(result.stdout)  # 42\n        await session.save()  # generation 2\n```\n\n`save()` keeps the VM running: the session goes on after the save.\n\nMake a key with `python3 -c \"import base64, secrets; print(base64.b64encode(secrets.token_bytes(32)).decode())\"`. The keyring is a comma-separated list of keys, newest first: the first key encrypts new saves, the others only open older instances. Keep the keyring in your secret store, and give every worker the same keyring. Keep every key that saved an instance: without it, `reuse()` raises `ConfigError` (`INSTANCE_KEY_MISSING`). To change keys: [Key rotation](#key-rotation).\n\nTo choose the id, pass it to `new()`: `scheduler.new(spec, instance_id=\"user-42\")`. An id has 1 to 64 lowercase letters, digits, `.`, `_` or `-`, and starts with a letter or digit. If the id exists, the first `save()` fails with `INSTANCE_CONFLICT`. Without an id, the first save makes a random one (` ref.instance_id`).\n\n`session.save(instance_id=\"run-17\")` saves the VM as generation 1 of a new instance, then the session saves to that instance (`session.instance`). The instance the session had stays as its last save left it. An id that exists raises `INSTANCE_CONFLICT`, and nothing is overwritten. The session's own id gives a plain save.\n\nTo start many sessions from one save, call `reuse()` of the same instance in each, and give each session its own id at its first save:\n\n```\nchildren = [await scheduler.reuse(checkpoint.instance_id) for _ in range(3)]\nfor n, child in enumerate(children):\n    await child.exec(f\"attempt({n})\")\n    await child.save(instance_id=f\"attempt-{n}\")  # the checkpoint stays as it was\n```\n\nEach VM that starts from the same save gets new values from its first `exec()`: the kernel random source (` os.urandom`, `/dev/urandom`), Python `ssl` and the module functions of `random`, the random functions of the JavaScript `crypto` and `node:crypto`, `Math.random` (also in the `node:vm` contexts that the code makes), the keys of the `Sync` functions of `node:crypto` (such as `generateKeyPairSync`), and bash `$RANDOM`. Python `random` gets a new seed at each `reuse()`: to repeat a seeded run, call `random.seed()` after the load. These keep the state of the save in each copy: values made before the save, programs that run since before the save, `random.Random()` objects and NumPy generators. In Bun, these can give the same values in each copy: the keys, primes and signatures of the functions that take a callback or return a promise (such as `generateKeyPair`, `generatePrime` and `crypto.subtle`), TLS and `fetch`, and Workers. To get other values in each copy, make keys with the `Sync` functions, and seed the other generators in the copy.\n\nThe `local` store (`instance_store=\"local\"`) is a folder on this host, for one host. For the `s3` store, set up the bucket ([Set up the S3 bucket](#set-up-the-s3-bucket)), then give every worker the same keyring and the same config: `SchedulerConfig(instance_store=\"s3\", s3_bucket=\"my-bucket\")`. At start, the `Scheduler` checks the bucket and logs \"Instance store ready\" with its key ids.\n\n`sbx run --instance ID` continues the instance `ID`, or makes it when it does not exist:\n\n```\nsbx run --instance my-vm -c 'x = 41'            # Created instance my-vm, generation 1\nsbx run --instance my-vm -c 'x += 1; print(x)'  # 42, then: Saved instance my-vm, generation 2\nsbx instance list                               # One id on each line (--json for an array)\nsbx instance list --long                        # One JSON object for each instance: as describe, or its error\nsbx instance describe my-vm                     # JSON: generation, recent commits, key id, access, writer release; spec, host (with the device layout), guest and egress domains when access is full\nsbx instance keys                               # The id of each key of the keyring, newest first\nsbx instance check my-vm                        # \"my-vm ok\", or why this worker cannot continue it\nsbx worker describe                             # JSON: this worker's facts (WorkerFacts), for a broker\nsbx instance rm my-vm other-vm                  # In order, stops at the first error; exit code 0 when they are gone\n```\n\n- **Store.**`sbx` reads the environment variables of[SchedulerConfig](#schedulerconfig) . By default, it uses the local store in`instance_store_dir` (`EXEC_SANDBOX_INSTANCE_STORE_DIR` ). For S3, set`EXEC_SANDBOX_INSTANCE_STORE=s3` ,`EXEC_SANDBOX_S3_BUCKET` and`EXEC_SANDBOX_INSTANCE_KEYRING` , and the other`EXEC_SANDBOX_S3_*` variables as needed. Only`--instance` and`sbx instance` open the store.\n- **Key.**`sbx` reads`EXEC_SANDBOX_INSTANCE_KEYRING` ; set it on a host with no OS keyring, such as a server or a container. Else, for the local store,`sbx` makes one key at the first use and keeps it in the OS keyring (macOS Keychain; Secret Service or KWallet on Linux; no other`keyring` backend), entry`exec-sandbox` /`instance-keyring` . Back it up with`keyring get exec-sandbox instance-keyring` , restore it with`keyring set exec-sandbox instance-keyring` (the`keyring` command of the`cli` extra, in the same Python environment). To open these instances from code, pass`instance_keyring=keyring.get_password(\"exec-sandbox\", \"instance-keyring\")` .\n- **Save.** A run saves after the uploads, the code and the downloads, also when the code exits with an error or a download fails. It saves nothing after a timeout, a VM that stopped, or a failed upload or run; stderr tells why.`-q` hides the \"Created/Saved\" line.\n- **VM settings.**`-l` ,`-p` ,`-m` ,`--network` ,`--allow-domain` and`--expose` are fixed when the instance is made. Leave them out or give the same value; another value exits with code 2. With`--allow-domain` , also give`--network` .`--instance` takes one source (`-c` , a file or`-` ).\n- **Exit codes.** A bad id exits with code 2. If two runs save at the same time, the second save fails with`INSTANCE_CONFLICT` and exit code 125; its output stays, its state does not. Others:[Exit codes](#exit-codes) .\n\n```\ninfo = await scheduler.describe_instance(\n    instance_id\n)  # Generation, recent commits, key id; spec and host when this keyring opens it\ndeleted = await scheduler.delete_instance(instance_id)  # True if it existed\nasync for instance_id in scheduler.list_instances():\n    ...\n```\n\nTo do this without a `Scheduler` (no VM, no guest image, no QEMU: a cleanup job on any host), open the store with `instance_admin`:\n\n``` python\nfrom exec_sandbox import SchedulerConfig, instance_admin\n\nasync with instance_admin(SchedulerConfig(instance_store=\"s3\", s3_bucket=\"my-bucket\")) as admin:\n    async for instance_id in admin.list_ids():\n        info = await admin.describe(instance_id)\n```\n\n- **One writer.**`save()` fails with`INSTANCE_CONFLICT` if another save of the instance was stored since this session loaded or last saved it. The session stays open: save it under a new id to keep its work, or close it. If two requests can reach one instance at the same time, use your own lock.\n- **Network.** A reused instance reaches the domains its last save allowed (`describe_instance(id).egress_domains` ; empty: all), also after a release that changes the default list of`allowed_domains=None` .\n- **`close()` never saves.** After an`exec()` or`write_file()` with no later save,`close()` warns with`InstanceNotSavedWarning` (not after another save of the instance won: no plain save of that session can store its work). The idle timeout closes the session with no save (it logs a warning in the cases where`close()` warns): save before a long pause.\n- **Host kind.** An instance loads only on a worker with the same CPU architecture, accelerator (`kvm` ,`hvf` or`tcg` ), guest CPU model, host CPU (ARM with KVM or HVF) and guest clock rate (any rate on x86 KVM with TSC scaling), and a QEMU that has the saved machine type (for example`virt-11.1` ) and accepts the saved state. The release of the worker must run the saved devices (`describe_instance(id).host.device_layout` ). Other workers raise`RouteError` ; its`required` field gives the values (empty for the devices: load it on a worker of another release).\n- **After `reuse()`.** The network connections that were open at the save are closed.\n\n| `action` (class) | Codes | Do this | \n|---|---|---|\n| `RETRY` (`TransientError` ) | `INSTANCE_STORE_UNAVAILABLE` ,`LOCAL_DISK` | Call again later. A failed save stored nothing. | \n| `ROUTE` (`RouteError` ) | `INSTANCE_HOST_MISMATCH` | Use a worker that satisfies `required` ([Upgrades](#upgrades) ). Keep the instance. | \n| `RECREATE` (`InstanceGoneError` ) | `INSTANCE_NOT_FOUND` ,`INSTANCE_EXPIRED` ,`INSTANCE_INTEGRITY` | Start a new instance. To use the same id again, call `delete_instance(id)` first. A save that raises`INSTANCE_NOT_FOUND` keeps its session open: save it under a new id to keep its work. | \n| `VERIFY` (`OutcomeUnknownError` ) | `INSTANCE_SAVE_OUTCOME_UNKNOWN` | The session stays open. The save was stored if `describe_instance(id).recent_commits` has`error.context[\"commit_id\"]` : check before you do the work again. A retry of the same save (the same`instance_id` ) never stores a second copy. | \n| `VERIFY` (`OutcomeUnknownError` ) | `INSTANCE_CONFLICT` | This save was not stored, and the session stays open. If an earlier save of the session had an unknown outcome, look for its `commit_id` in`describe_instance(id).recent_commits` first: it may be the save that is stored. Else another save holds the instance: save this session under a new id to keep its work, or`reuse()` the instance (it has the other save) and do the work again. For a new id, or the first save of`new(instance_id=...)` , the id is taken: use another id. | \n| `FIX_CONFIG` (`ConfigError` ) | `INSTANCE_CONFIG` ,`INSTANCE_KEY_MISSING` ,`MIGRATION_UNSUPPORTED` ,`IMAGE_DAMAGED` | `INSTANCE_CONFIG` : fix the bucket, store folder, permissions or keyring, or use a store with conditional writes.`INSTANCE_KEY_MISSING` : add the named key; keep the instance.`MIGRATION_UNSUPPORTED` : upgrade QEMU to 11.0+.`IMAGE_DAMAGED` : download or rebuild the images. | \n| `FIX_INPUT` (`InvalidInputError` ) | `INSTANCE_INVALID_ID` | Fix the id ( [Instance ids](#instance-ids) ). | \n\n`save()` and `reuse()` can also raise the other codes of `ErrorCode`: [Error codes](#error-codes).\n\n`VmSpec` says what a VM is made of. It is frozen and hashable: keep one spec per workload.\n\n| Field | Default | Description | \n|---|---|---|\n| `language` | required | `\"python\"` ,`\"javascript\"` or`\"raw\"` (shell) | \n| `memory_mb` | 192 | VM memory, 128 MB minimum. See [Memory and file systems](#memory-and-file-systems) | \n| `cpu_cores` | 1 | Guest vCPUs | \n| `disk_mb` | 1024 | Home disk size ( `/home/user` , ext4), 128 MB minimum. Only written blocks use host disk | \n| `packages` | `()` | Packages, each with a version ( `\"pandas==3.0.1\"` ,`\"lodash@4.17.21\"` ). See[Packages](#packages) | \n| `allow_network` | False | Outbound network access | \n| `allowed_domains` | None | With `allow_network` : the domains the guest can reach. See[Outbound access](#outbound-access) | \n| `expose_ports` | `()` | Guest ports to expose on the host ( `PortMapping` or an int), at most 10. See[Port forwarding](#port-forwarding) | \n\nSettings for all VMs of a `Scheduler`: `Scheduler(SchedulerConfig(warm_pool_size=1))`. A field that you do not set in code reads its environment variable. An invalid value fails the `Scheduler` start with `ConfigError` (`SETTING_INVALID`).\n\n| Field | Default | Env var | Description | \n|---|---|---|---|\n| **Sessions and warm pool** |  |  |  | \n| `warm_pool_size` | 0 | — | Warm-pool VMs per language, for `VmSpec(language=...)` with no other field. 0: no warm pool | \n| `default_timeout_seconds` | 30 | — | Execution timeout (1-300 s) | \n| `session_idle_timeout_seconds` | 300 | — | Close a session after this long without a call (10-3600 s) | \n| **Admission** |  |  |  | \n| `memory_overcommit_ratio` | 5.0 | `EXEC_SANDBOX_MEMORY_OVERCOMMIT_RATIO` | 1.0-10.0. Budget = host memory × (1 - reserve) × ratio | \n| `cpu_overcommit_ratio` | 2.0 | `EXEC_SANDBOX_CPU_OVERCOMMIT_RATIO` | 1.0-20.0. Budget = (host CPUs - reserve) × ratio | \n| `disk_overcommit_ratio` | 4.0 | `EXEC_SANDBOX_DISK_OVERCOMMIT_RATIO` | 1.0-64.0. Each VM reserves the unwritten part of its `disk_mb` / ratio | \n| `host_memory_reserve_ratio` | 0.08 | `EXEC_SANDBOX_HOST_MEMORY_RESERVE_RATIO` | Fraction of host memory kept for the OS (0.01-0.5) | \n| `host_cpu_reserve_cores` | 0.5 | `EXEC_SANDBOX_HOST_CPU_RESERVE_CORES` | CPU cores kept for host processes (0.0-16.0) | \n| **Packages** |  |  |  | \n| `enable_package_validation` | True | — | Check packages against the allow-list (top 10,000 of PyPI and of npm) | \n| **Images and caches** |  |  |  | \n| `images_dir` | None | `EXEC_SANDBOX_IMAGES_DIR` | Folder of the VM images. None: found automatically ( [How images download](#how-images-download) ) | \n| `auto_download_assets` | True | — | Download missing VM images from GitHub Releases | \n| `disk_snapshot_cache_dir` | `<cache dir>/disk-snapshots/` | `EXEC_SANDBOX_DISK_SNAPSHOT_CACHE_DIR` | L2 cache folder ( [Where files go](#where-files-go) ) | \n| `memory_snapshot_cache_dir` | `<cache dir>/memory-snapshots/` | `EXEC_SANDBOX_MEMORY_SNAPSHOT_CACHE_DIR` | L1 cache folder | \n| **S3** |  |  |  | \n| `s3_bucket` | None | `EXEC_SANDBOX_S3_BUCKET` | Bucket of the L3 snapshot cache and of the `s3` instance store | \n| `s3_region` | us-east-1 | `EXEC_SANDBOX_S3_REGION` | AWS region | \n| `s3_endpoint_url` | None | `EXEC_SANDBOX_S3_ENDPOINT_URL` | S3-compatible endpoint (R2, MinIO, Ceph RGW); None for AWS | \n| `s3_snapshot_prefix` | snapshots/ | `EXEC_SANDBOX_S3_SNAPSHOT_PREFIX` | Key prefix of the L3 snapshot cache | \n| `s3_instance_prefix` | instances/ | `EXEC_SANDBOX_S3_INSTANCE_PREFIX` | Key prefix of saved instances ( [Set up the S3 bucket](#set-up-the-s3-bucket) ) | \n| `max_concurrent_s3_uploads` | 128 | `EXEC_SANDBOX_MAX_CONCURRENT_S3_UPLOADS` | Max concurrent background S3 uploads (1 or more) | \n| **Saved instances** |  |  |  | \n| `instance_store` | None | `EXEC_SANDBOX_INSTANCE_STORE` | `s3` (`s3_bucket` ,`s3` extra) or`local` (`instance_store_dir` ,`cli` or`s3` extra).`None` set in code turns saved instances off | \n| `instance_store_dir` | OS data folder | `EXEC_SANDBOX_INSTANCE_STORE_DIR` | Root of the `local` store. macOS:`~/Library/Application Support/exec-sandbox/instances/` . Linux:`~/.local/share/exec-sandbox/instances/` (under`$XDG_DATA_HOME` when it is an absolute path) | \n| `instance_keyring` | None | `EXEC_SANDBOX_INSTANCE_KEYRING` | Required with `instance_store` : comma-separated base64 32-byte keys, newest first. Never logged or passed to child processes | \n\nFolder rules: `instance_store_dir` must be a folder of this user, mode 0700, with no ACL entry (exec-sandbox makes it when it is missing). Only root and this user can write the parent folders of `instance_store_dir`, the cache folders and `EXEC_SANDBOX_HOME_DISK_DIR`. Sticky folders such as `/tmp` pass, and on Linux, so does a folder that only this user's private group can write. For a root volume that others can write (for example a Kubernetes `emptyDir`), set the sticky bit (` chmod +t`) as root, for example in an init container.\n\nVariables that are not `SchedulerConfig` fields:\n\n| Variable | Default | Description | \n|---|---|---|\n| `EXEC_SANDBOX_CACHE_DIR` | OS cache folder | Root of the cache folders ( [Where files go](#where-files-go) ) | \n| `EXEC_SANDBOX_HOME_DISK_DIR` | `<cache dir>/home-disks` | Home disks of running VMs, and work files of `save()` and`reuse()` . A folder of this user that group and others cannot write, with no ACL entry, on a disk (not tmpfs) | \n| `EXEC_SANDBOX_INSTANCE_ASSET_DIR` | `<cache dir>/instance-assets` | Guest images downloaded from the instance store | \n| `EXEC_SANDBOX_DISK_FREE_FLOOR_MB` | 1024 | Free space that admission keeps on the disk of the home disk folder | \n| `EXEC_SANDBOX_AVAILABLE_MEMORY_FLOOR_MB` | 0 | Available host memory that admission keeps free | \n| `EXEC_SANDBOX_HOST_MEMORY_MB` ,`EXEC_SANDBOX_HOST_CPU_COUNT` | detected | Host size for the admission budgets | \n| `EXEC_SANDBOX_QEMU_BIN_X86` ,`EXEC_SANDBOX_QEMU_BIN_ARM` | `/usr/bin/qemu-system-x86_64` ,`/usr/bin/qemu-system-aarch64` | QEMU binary ( [Install QEMU](#install-qemu) ) | \n| `EXEC_SANDBOX_FORCE_EMULATION` | false | Use TCG even when KVM or HVF is available | \n| `EXEC_SANDBOX_SNAPSHOT_CACHE_TTL_DAYS` | 14 | Value of the `ttl_days` tag on L3 uploads | \n| `EXEC_SANDBOX_OFFLINE` | unset | `1` : never download assets (a missing asset fails) | \n| `EXEC_SANDBOX_ASSET_VERSION` | unset | Use the assets of this release version | \n| `EXEC_SANDBOX_LOG_LEVEL` | unset | Level of the `exec_sandbox` logger (`DEBUG` ,`INFO` ,`WARNING` ,`ERROR` ) | \n| `GITHUB_TOKEN` | unset | Sent to the GitHub API for asset downloads (higher rate limit) | \n\nThe library logs to the `exec_sandbox` logger and adds no handler. `configure_logging(level=\"INFO\")` adds a stderr handler (the CLI does this).\n\nEvery run-time error is a `SandboxError` with an `action` (what to do next, from its class), a `code` (an `ErrorCode`), a `message`, a `hint`, an `instance_id` and a `context`. After a QEMU crash or a boot timeout, `context[\"console_log\"]` holds the VM console. A bad value in `VmSpec`, `SchedulerConfig` or `PortMapping` raises pydantic's `ValidationError`.\n\n| Name | Action | When | \n|---|---|---|\n| `SandboxError` | — | Base class (never raised itself) | \n| `TransientError` | `RETRY` | The call did nothing you rely on (exec: the code was not sent). Repeat it, on this worker or another, a bounded number of times (for example 3, with backoff). If the session closed, redo its unsaved work on a new session first | \n| `OutcomeUnknownError` | `VERIFY` | The code may have run, or a save's outcome is unknown, or another save was stored first. Check the state first | \n| `SessionClosedError` | `RESTART` | The session is closed. Open a new one with `new()` or`reuse()` | \n| `InvalidInputError` | `FIX_INPUT` | The call is wrong, or the scheduler cannot take it now. The session stays open | \n| `ConfigError` | `FIX_CONFIG` | This worker's setup is wrong: settings, keyring, store, binaries, images, Python extras, permissions or OS | \n| `RouteError` | `ROUTE` | Another worker can serve the call: `required` gives the facts the call checked,`found` this worker's values. A worker satisfies`required` when each key holds by its rule (`RouteKey.rule` :`equal` ,`at_least` ,`contains` or`equal_or_any` ); it passes the checks that raised the error. An empty`required` (another release wrote the instance) names no worker: use a worker of the release in`describe_instance(id).writer_release` , or in the error's`context[\"writer_release\"]` when`describe_instance` raises it too and the data names its release | \n| `InstanceGoneError` | `RECREATE` | The saved instance is gone, expired or damaged. Start a new one | \n| `InternalError` | `REPORT` | A bug or a broken host. Report it with the log | \n| `ErrorAction` |  | The 8 actions above | \n| `ErrorCode` |  | See [Error codes](#error-codes) | \n| `InstanceNotSavedWarning` |  | Warning: `close()` of a saved-instance session with`exec()` or`write_file()` calls after its last`save()` | \n\n`RouteKey` names the keys of `RouteError.required` and of `WorkerFacts.values` (`describe_worker()`, `sbx worker describe`); `RouteRule` names their rules. In `WorkerFacts.values`, a `contains` key holds the set of values this worker has (a sorted list in JSON), and `ROUTE_ANY` (`\"any\"`) is the value of an `equal_or_any` key that takes every value.\n\n| Key | Rule | Value | \n|---|---|---|\n| `arch` | `equal` | `x86_64` or`aarch64` | \n| `accelerator` | `equal` | `kvm` ,`hvf` or`tcg` | \n| `cpu_model` | `equal` | The guest CPU model (QEMU `-cpu` value) | \n| `cpu_id` | `equal` | The host CPU an ARM guest sees with KVM or HVF; absent on x86 and ARM TCG | \n| `counter_hz` | `equal_or_any` | Rate in Hz of the guest clock's counter, set at its first boot; absent when that host had no exact rate. A worker's `any` : x86 KVM with TSC scaling, which takes every rate | \n| `qemu_version` | `at_least` | `MAJOR.MINOR.PATCH` , compared part by part | \n| `machine_type` | `contains` | A QEMU machine type ( `virt-11.1` ,`microvm` ) | \n| `base_image` | `contains` | SHA-256 digest (64 hex characters) | \n| `packages_image` | `contains` | SHA-256 digest (64 hex characters) | \n| `memory_mb` | `at_least` | Whole MB; a worker's `null` : no limit | \n| `cpu` | `at_least` | Cores with two decimals; a worker's `null` : no limit | \n\nCatch the class of each action that you handle:\n\n``` python\nfrom exec_sandbox import ErrorCode, InvalidInputError, OutcomeUnknownError, SandboxError, TransientError\n\ntry:\n    async with await scheduler.new(VmSpec(language=Language.PYTHON)) as session:\n        result = await session.exec(\"print('hello')\", timeout_seconds=5)  # A timeout does not raise: exit_code -1\nexcept InvalidInputError as e:  # FIX_INPUT: change the call\n    if e.code is ErrorCode.PACKAGE_NOT_ALLOWED:\n        ...\nexcept TransientError:  # RETRY: the same call can succeed later\n    ...\nexcept OutcomeUnknownError:  # VERIFY: the code may have run\n    ...\nexcept SandboxError as e:  # The other actions\n    print(f\"{e.code} ({e.action}): {e}\")\n```\n\n`result.exit_code` is 0 for success and -1 for a timeout. A value above 128 usually means that signal N (exit code - 128) killed the code (137 = SIGKILL, 139 = SIGSEGV). Other values mean that the program failed. It is a process status (0-255): `sys.exit(-1)` gives 255. Infrastructure failures raise an error and return no result. For the exit codes of `sbx`, see [Exit codes](#exit-codes).\n\n- **Stays open, state kept:** an exception, a syntax error,`sys.exit(n)` with n ≠ 137, and`InvalidInputError` (for example`OUTPUT_LIMIT` , unless its message ends with \"the REPL then exited with code n\": see the next line).\n- **Stays open, the next `exec()` runs in a new REPL:** a timeout (-1), a signal other than SIGKILL (128 + signal),`os._exit(n)` with n ≠ 137 (a REPL exit or signal after output past the limit:`OUTPUT_LIMIT` whose message ends with \"the REPL then exited with code n\"), and a REPL exit between two calls with a code other than 137.\n- **Closes, state lost:** exit code 137 (for example SIGKILL), also between two calls (the next`exec()` raises`GUEST_UNAVAILABLE` ); a VM communication failure; QEMU stops; a full host disk under the home disk. Every later call raises`SessionClosedError` .\n- Other errors: in `exec()` , they close the session. File calls close it only on cancellation,`OutcomeUnknownError` , a dead VM or a full host disk. For`save()` , see its docstring.\n\n| Code | Class | Meaning | \n|---|---|---|\n| `VM_BOOT_TIMEOUT` | `TransientError` | The guest agent did not answer in time at boot. | \n| `QEMU_CRASHED` | `TransientError` | QEMU exited or did not come up. | \n| `OVERLAY_FAILED` | `TransientError` | qemu-img or qemu-storage-daemon could not make a disk image. | \n| `GVPROXY_FAILED` | `TransientError` | gvproxy did not start or become ready. | \n| `CAPACITY` | `TransientError` | No resources became free within the admission timeout. | \n| `GUEST_UNAVAILABLE` | `TransientError` | The guest could not take the call or start the code. The code did not run. | \n| `GUEST_SILENT_AFTER_RESTORE` | `TransientError` | The guest did not answer after a memory snapshot restore, or could not reseed its kernel random source. | \n| `LOCAL_DISK` | `TransientError` | A local file is missing (a path that you gave: `ARGUMENT_INVALID` ), or the host cannot read or write it now (disk full, quota, I/O error, file handle or memory limit), or the host disk has no room for a home disk. | \n| `PACKAGE_DOWNLOAD` | `TransientError` | The package manager could not download (network error or timeout). | \n| `MIGRATION_FAILED` | `TransientError` | QEMU did not save or load the VM state. | \n| `ASSET_DOWNLOAD` | `TransientError` | A VM image download failed after its retries. | \n| `INSTANCE_STORE_UNAVAILABLE` | `TransientError` | The instance store failed, answered badly, or kept changing. | \n| `GUEST_OUTCOME_UNKNOWN` | `OutcomeUnknownError` | The transport was lost after the call reached the guest. The code may have run. | \n| `INSTANCE_CONFLICT` | `OutcomeUnknownError` | Another save was stored first, or the id of a first save is taken. This save was not stored; the session stays open. | \n| `INSTANCE_SAVE_OUTCOME_UNKNOWN` | `OutcomeUnknownError` | The store did not confirm the save, and a read-back could not tell. | \n| `SESSION_CLOSED` | `SessionClosedError` | The session is closed. | \n| `CODE_INVALID` | `InvalidInputError` | The code is empty, only spaces, holds null bytes, or is too large. | \n| `ENV_VAR_INVALID` | `InvalidInputError` | An env var name or value breaks a rule. | \n| `OUTPUT_LIMIT` | `InvalidInputError` | stdout or stderr went above the guest limit. | \n| `PATH_INVALID` | `InvalidInputError` | A guest path breaks a rule (empty, too long, outside the sandbox, a folder). | \n| `GUEST_IO` | `InvalidInputError` | The guest could not read, list or write the path (not found, no access, disk full). | \n| `ARGUMENT_INVALID` | `InvalidInputError` | An argument breaks a documented limit: timeout range, content size, stream, local path, package without a version, packages for `raw` . | \n| `PACKAGE_NOT_ALLOWED` | `InvalidInputError` | A package is not in the allow-list. | \n| `PACKAGE_INSTALL_FAILED` | `InvalidInputError` | The package manager refused the packages. | \n| `INSTANCE_INVALID_ID` | `InvalidInputError` | The value breaks the instance id rule. | \n| `SCHEDULER_STATE` | `InvalidInputError` | The scheduler is not started, already started, or stopping. | \n| `HOST_DEPENDENCY` | `ConfigError` | A binary (QEMU, qemu-img, qemu-storage-daemon, gvproxy-wrapper), system user, cgroup or ACL support is missing or too old, or this user cannot run it; with KVM or HVF, the host CPU cannot run the x86 guest CPU, or the ARM host CPU identity cannot be read. | \n| `EXTRA_MISSING` | `ConfigError` | A Python extra is not installed (the message gives the pip command). | \n| `SETTING_INVALID` | `ConfigError` | A worker setting is not valid. | \n| `ASSET_NOT_FOUND` | `ConfigError` | A VM image is not on this host and was not downloaded. | \n| `IMAGE_DAMAGED` | `ConfigError` | A local guest image does not match its digest: download or rebuild the images. | \n| `PLATFORM_UNSUPPORTED` | `ConfigError` | The OS is not Linux or macOS. | \n| `MIGRATION_UNSUPPORTED` | `ConfigError` | This worker's QEMU cannot save or load VM state (below QEMU 11.0, or a capability refused). | \n| `INSTANCE_CONFIG` | `ConfigError` | The instance settings are wrong (store, keyring, store check). | \n| `INSTANCE_KEY_MISSING` | `ConfigError` | The instance's key is not in this keyring. The data is intact: do not delete the instance. | \n| `INSTANCE_HOST_MISMATCH` | `RouteError` | This worker cannot load the saved instance: a value in `required` differs, or (empty`required` ) this release cannot read the stored data or runs other devices for it (`describe_instance(id).host.device_layout` ). | \n| `CAPACITY_NEVER_FITS` | `RouteError` | The VM is larger than this worker's whole budget. | \n| `MIGRATION_REJECTED` | `RouteError` | QEMU refused the saved state. It shows only in the cause chain of `INSTANCE_HOST_MISMATCH` (this QEMU is older than the saver's) or of`InternalError`` INTERNAL` (the same or a newer QEMU). | \n| `INSTANCE_NOT_FOUND` | `InstanceGoneError` | No instance with this id (deleted, expired, or never saved). | \n| `INSTANCE_EXPIRED` | `InstanceGoneError` | Part of the saved state is gone from the store. | \n| `INSTANCE_INTEGRITY` | `InstanceGoneError` | Stored data failed its checks. A head this release authenticates but cannot read gives `RouteError` (`INSTANCE_HOST_MISMATCH` ), not this code. | \n| `CLEANUP_UNCONFIRMED` | `InternalError` | The end of a QEMU or gvproxy process, or the release of VM resources, was not confirmed. | \n| `GUEST_PROTOCOL` | `InternalError` | The guest sent a protocol error or an answer of the wrong type. | \n| `INTERNAL` | `InternalError` | A bug. | \n\nThe start time of a VM depends on its source: [Snapshot cache](#snapshot-cache).\n\nSet `warm_pool_size` to keep that many VMs ready for each language. Only `VmSpec(language=...)` with no other field comes from the warm pool.\n\n```\nasync with Scheduler(SchedulerConfig(warm_pool_size=1)) as scheduler:\n    await scheduler.wait_pool_ready()  # Waits until the pool is full, 120 s at most\n    async with await scheduler.new(VmSpec(language=Language.PYTHON)) as session:  # From the warm pool\n        ...\n```\n\nEach warm-pool VM takes 272 MB of the memory budget (192 MB + 80 MB overhead): `warm_pool_size=5` takes about 4.1 GB (5 VMs × 3 languages).\n\nThe default overcommit ratios come from tuning on production KVM hosts. To serve more requests at once, add hosts; do not raise the ratios. On a desktop, find the best values with `make bench-optimizer` in a clone of the repository ([Contributing](#contributing)).\n\nThe cache folder (`<cache dir>`) is `~/Library/Caches/exec-sandbox/` on macOS and `~/.cache/exec-sandbox/` on Linux (or `$XDG_CACHE_HOME/exec-sandbox/` when that is an absolute path). It holds:\n\n- `v<version>/` : the download cache (VM images and gvproxy-wrapper)\n- `memory-snapshots/` (L1) and`disk-snapshots/` (L2)\n- `home-disks/` : the home disks of running VMs\n- `instance-assets/` : guest images from the instance store (removed oldest first when the home disks need the space, never one a VM runs on)\n\n`EXEC_SANDBOX_CACHE_DIR` moves all of them (the download cache is then the folder itself, with no `v<version>`). Use one cache folder per host. Put the home disk folder and the L1 cache on a disk, not on tmpfs (`/tmp` on many Linux hosts): on tmpfs, QEMU writes count against the memory of the VM.\n\nA new VM comes from the first source that has it (times with KVM or HVF):\n\n| Source | Start time | \n|---|---|\n| Warm pool | Under 1 ms | \n| L1 memory snapshot (a spec that ran before; QEMU 11.0 or later) | About 50-60 ms | \n| L2 disk snapshot (packages already installed) | About 200 ms boot, then 0.1-1 s REPL start | \n| L3 snapshot cache (L2 from S3) | Download, then as L2 | \n| Cold boot | As L2, plus the package install | \n\nAfter a boot, the REPL starts in the first `exec()` (counted in `timing.execute_ms`, not in `startup_timing`).\n\nL3 shares L2 entries between hosts through S3. Set `s3_bucket` to turn it on (keys under `s3_snapshot_prefix`). It needs `s3:GetObject`, `s3:PutObject`, `s3:PutObjectTagging` and `s3:AbortMultipartUpload` on `<s3_snapshot_prefix>*`. Add two lifecycle rules on that prefix: one that expires objects (uploads have the tag `ttl_days=14`, `EXEC_SANDBOX_SNAPSHOT_CACHE_TTL_DAYS`), and `AbortIncompleteMultipartUpload` (1 day). An L3 error only logs a warning.\n\nL1 and L2 have no size limit. exec-sandbox removes entries (oldest last use first) only when the disk is full or a new home disk needs the space, and never an entry that a running VM uses. When no `Scheduler` runs, you can delete L1 and L2 entries, old `v<version>` image folders and the files in `instance-assets`.\n\nexec-sandbox needs VM images (kernel, initramfs, qcow2 base images) and the gvproxy-wrapper binary. It downloads them from GitHub Releases at first use.\n\nEach `Scheduler` start uses the first of these folders that holds both the kernel and the initramfs:\n\n1. `images_dir` ([SchedulerConfig](#schedulerconfig) )\n2. `EXEC_SANDBOX_IMAGES_DIR`\n3. `images/dist/` of a clone of the repository (a local build)\n4. the download cache ([Where files go](#where-files-go) )\n\nIf no folder holds them, exec-sandbox downloads them to the download cache (with `auto_download_assets=False`: `ConfigError` `ASSET_NOT_FOUND`). It looks for gvproxy-wrapper in the folder of the boot files, then in `gvproxy-wrapper/bin/` of a clone, then in the folders above. It looks for the base image of a language, at the first VM of that language, in the folder of the boot files, then in the download cache. With downloads on, a missing asset downloads to the download cache.\n\nDownloads come from the GitHub release of this version (`v{__version__}`) over HTTPS, and each is checked against its SHA-256 digest from the GitHub API. Set `GITHUB_TOKEN` for a higher GitHub API rate limit.\n\n`sbx prefetch` downloads all assets before use:\n\n```\nsbx prefetch                    # All assets for this architecture\nsbx prefetch --arch aarch64     # Assets of another architecture\nsbx prefetch -q                 # No progress output (CI, Docker)\nFROM ghcr.io/astral-sh/uv:python3.12-bookworm\n# The image also needs QEMU (see Install)\nRUN uv pip install --system \"exec-sandbox[cli]\"\nRUN sbx prefetch -q\nENV EXEC_SANDBOX_OFFLINE=1\n# A spec with packages still needs PyPI or npm for its first build (L2 cache miss).\n# Run the container with --device /dev/kvm for hardware acceleration.\n```\n\nFrom a clone of the repository, `./scripts/build-images.sh` writes `python-3.14-base-{arch}.qcow2`, `node-1.4-base-{arch}.qcow2` and `raw-base-{arch}.qcow2` to `./images/dist/`.\n\nFor more isolation on Linux, exec-sandbox can run QEMU as an unprivileged `qemu-vm` user:\n\n```\n# Create qemu-vm system user, add it to the kvm group (hardware acceleration), and add your user to its group (socket access)\nsudo useradd --system --no-create-home --shell /usr/sbin/nologin qemu-vm\nsudo usermod -aG kvm qemu-vm\nsudo usermod -aG qemu-vm $USER\n\n# Let your user run commands as qemu-vm, and chown and rm as root, with no password\necho \"$USER ALL=(qemu-vm) NOPASSWD: ALL\" | sudo tee /etc/sudoers.d/qemu-vm\necho \"$USER ALL=(root) NOPASSWD: $(command -v chown), $(command -v rm)\" | sudo tee -a /etc/sudoers.d/qemu-vm\nsudo chmod 0440 /etc/sudoers.d/qemu-vm\n\n# setfacl gives qemu-vm access to the home disks and the L1 cache files\nsudo apt install acl\n\n# Log in again, or start a shell with the new group (for a service: sg qemu-vm -c '<command>')\nnewgrp qemu-vm\n\n# Check: each line must print ok\nsudo -n -k -u qemu-vm /bin/sh -c 'umask 007 && exec \"$@\"' qemu-vm true && echo ok\nid -nG | grep -qw qemu-vm && echo ok\nsudo -n -k -l chown >/dev/null && sudo -n -k -l rm >/dev/null && echo ok\n# the group's members, and the accounts whose primary group it is: only qemu-vm and you\ngid=$(getent group qemu-vm | cut -d: -f3); { getent group qemu-vm | cut -d: -f4 | tr , '\\n'; awk -F: -v g=\"$gid\" '$4 == g {print $1}' /etc/passwd; } | grep -vx -e '' -e qemu-vm -e \"$(id -un)\" | grep -q . || echo ok\n# in the shell that starts the service, after the cgroup setup below: makes a VM-like cgroup and moves a process into it\nd=/sys/fs/cgroup/code-exec/check-$$; mkdir \"$d\" && sh -c 'echo 0 > \"$1/cgroup.procs\"' sh \"$d\" 2>/dev/null && test -e \"$d/cgroup.kill\" && echo ok; rmdir \"$d\"\n```\n\n- Give these sudo rights to the service account only: `chown` and`rm` as root reach every file. A`(qemu-vm)` rule limited to some commands must allow`/bin/sh` .\n- The `qemu-vm` group must have no members other than`qemu-vm` and your user.\n- The service must run in the `qemu-vm` group (or as root), inside the cgroup v2 subtree of[Resource limits (cgroups)](#resource-limits-cgroups) , on Linux 5.14 or later.\n- exec-sandbox runs `chmod a+x` on the folders above the images, the VM files and the L1 cache files (for example your home folder).\n\nEach process checks this setup at its first `Scheduler` start. If a check fails, its VMs run QEMU as your user, and a warning names the failed check.\n\ncgroup limits (layer 6) need a cgroup v2 subtree that your user owns, and your service must start inside it. A reboot removes this setup:\n\n```\nsudo mkdir -p /sys/fs/cgroup/code-exec/service\necho \"+memory +cpu +pids\" | sudo tee /sys/fs/cgroup/cgroup.subtree_control\necho \"+memory +cpu +pids\" | sudo tee /sys/fs/cgroup/code-exec/cgroup.subtree_control\nsudo chown -R \"$USER\": /sys/fs/cgroup/code-exec\necho $$ | sudo tee /sys/fs/cgroup/code-exec/service/cgroup.procs   # then start the service from this shell\n```\n\nDo not name your service cgroup `exec-sandbox`, `exec-sandbox-probe`, `exec-sandbox-watcher`, `warm-pool` or `l1-cache`: exec-sandbox uses these names. Without this setup, exec-sandbox logs \"cgroup v2 unavailable, resource limits disabled\" and uses `ulimit` only.\n\nNamespaces (layer 7) need `unshare` from util-linux 2.32 or later. Check: `/usr/bin/unshare --pid --fork --kill-child=SIGKILL -- /usr/bin/true && echo ok`. Without it, exec-sandbox logs \"unshare unavailable (namespace isolation disabled)\" and starts QEMU without namespaces.\n\n`setpriv --pdeathsig` (util-linux 2.33 or later) makes gvproxy, qemu-storage-daemon and a QEMU that runs as your user end with the service. Check: `setpriv --pdeathsig KILL -- /bin/sh -c : && echo ok`. Without it, exec-sandbox logs \"setpriv --pdeathsig unavailable\" and starts these processes more slowly when the service uses much memory.\n\n`<prefix>` below is `s3_instance_prefix` (default `instances/`).\n\n- \n**Conditional writes.** The store must support`If-None-Match` and`If-Match` on PUT: AWS S3, Cloudflare R2, MinIO and Ceph RGW 19.2.7 or later do. The`Scheduler` checks this at start.\n- \n**Credentials.** The standard AWS chain (`AWS_ACCESS_KEY_ID` and`AWS_SECRET_ACCESS_KEY` ,`AWS_PROFILE` ,`~/.aws/credentials` , or the machine role). For R2, MinIO or Ceph RGW, set`s3_endpoint_url` .\n- \n**L3 cache.**`s3_bucket` also turns on the L3 snapshot cache in the same bucket (keys under`s3_snapshot_prefix` ): give the workers its permissions too ([L3: S3 snapshot cache](#l3-s3-snapshot-cache) ), else each new L2 entry logs a warning.\n- \n**Permissions.**`s3:GetObject` ,`s3:PutObject` ,`s3:DeleteObject` and`s3:AbortMultipartUpload` on`<prefix>*` , and`s3:ListBucket` on the bucket (without it, S3 answers 403 for a missing key). An access error raises`ConfigError` (`INSTANCE_CONFIG` ). Example for`instances/` :\n\n```\n{\n  \"Version\": \"2012-10-17\",\n  \"Statement\": [\n    {\"Effect\": \"Allow\", \"Action\": [\"s3:GetObject\", \"s3:PutObject\", \"s3:DeleteObject\", \"s3:AbortMultipartUpload\"], \"Resource\": \"arn:aws:s3:::my-bucket/instances/*\"},\n    {\"Effect\": \"Allow\", \"Action\": \"s3:ListBucket\", \"Resource\": \"arn:aws:s3:::my-bucket\"}\n  ]\n}\n```\n\n- \n**Lifecycle rules.**`AbortIncompleteMultipartUpload` (1 day) on`<prefix>` , and an expiration of 1 day on`<prefix>probe/` . To remove unused instances, add an expiration of X days on`<prefix>s/` ([Expiration](#expiration) ); without it, instances stay until`delete_instance()` . Never expire`<prefix>assets/` : all instances share these guest images. Delete an image there only when no instance uses it (`instance_admin` :`list_assets()` ,`delete_asset(digest)` ;`describe(id).guest` names the images of an instance whose`access` is`\"full\"` : keep every image while one is not). A save stores its images before its head: keep the images whose`LastModified` in the bucket is newer than your longest save.\n- \n**No versioning.** With versioning, deleted instances still use storage. If versioning was on, suspend it and add a lifecycle rule that deletes noncurrent versions.\n- \n**No archive storage class** (such as S3 Glacier Flexible Retrieval or Deep Archive): a load cannot read an archived object and fails with`ConfigError` (`INSTANCE_CONFIG` ).\n- \n`s3_instance_prefix` and`s3_snapshot_prefix` must not overlap.\n- \nGive write access to the workers only: anyone who can write to the bucket can put back an older head and so roll an instance back.\n\nAn expiration rule of X days on `<prefix>s/` deletes each object X days after its write (S3 can take 1-2 more days). Each save writes the head again, and a full copy or a pack (the blocks that changed). A new full copy comes when the packs since the last one reach its size, after 64 packs, or after a key rotation. An instance stays loadable for at least X days after its **last full copy**.\n\nSo an instance saved at each request must be used at least every **X / (n + 1) days**, where n = min(full copy size / typical pack size, 64). Example: X = 30 days, a 45 MiB full copy and 1 MiB packs give n = 45: use the instance at least every 15 hours.\n\nWhen an object is gone, `reuse()` fails with `INSTANCE_EXPIRED` (`action == \"RECREATE\"`). The next save of a session that is still open repairs the stored instance.\n\nAn instance moves to the newest key at its next save. To rotate the key on all workers:\n\n1. Add the new key at the end of `instance_keyring` on every worker (`OLD,NEW` ).\n2. When every worker has it, move it to the front (`NEW,OLD` ). If one worker puts it first too early, the others fail to load the instances it saves (`INSTANCE_KEY_MISSING` ).\n3. Remove the old key when no instance uses it: for each id of `list_instances()` ,`describe_instance(id).key_id` is not the old key's id. An instance that is never saved again keeps the old key.`sbx instance keys` (or`describe_worker().key_ids` ) shows the id of each key of a keyring, newest first.\n\nA saved instance loads on other exec-sandbox releases and on the same or a newer QEMU: it keeps the QEMU machine type of its first boot (`describe_instance(id).host.machine_type`). On x86 that type (` microvm`) has no version: give x86 workers the QEMU version of the save. A release that changes the devices of the VM or their migration capabilities (`describe_instance(id).host.device_layout`) cannot load the instances of an earlier one: `RouteError` with an empty `required`, before any download. Data that a release cannot read (another release changed the stored format) gives `RouteError` (`INSTANCE_HOST_MISMATCH`) with an empty `required`: load it on a worker of another release. `describe_instance(id).writer_release` names the release of the last save. Old L1 snapshots and VM images stay on disk ([Clean up](#clean-up)).\n\n- **Store.** Each instance holds its full copy and its packs (at most about twice the full copy) under`<prefix>s/<instance_id>/` . The store holds one copy of each guest image under`<prefix>assets/` .\n- **Encryption.** The RAM, the home disk, the`VmSpec` and the host values use AES-256-GCM. The key names, instance id, generation, commit ids and key id are in clear. Guest images are not encrypted.\n- **Worker disk.** Each open session of a saved instance, and each save and load, keeps a RAM file and a home disk copy (not encrypted) in a private folder (mode 0700) under the home disk folder (`EXEC_SANDBOX_HOME_DISK_DIR` , default`<cache dir>/home-disks/` ). exec-sandbox deletes them when the session closes, or at the next start after a crash. Guest images from the store stay in`<cache dir>/instance-assets/` ([Clean up](#clean-up) ).\n\n`sbx run --help` prints the same list.\n\n| Option | Short | Description | Default | \n|---|---|---|---|\n| `--code` | `-c` | Code to run (repeatable, in place of SOURCES) | - | \n| `--language` | `-l` | python, javascript or raw | from the file extension, else python | \n| `--timeout` | `-t` | Timeout in seconds (1-300) | 30 | \n| `--env` | `-e` | Environment variable KEY=VALUE (repeatable) | - | \n| `--memory` | `-m` | Memory in MB (128 or more) | 192 | \n| `--package` | `-p` | Package to install (repeatable) | - | \n| `--no-validation` |  | Skip the package allow-list check | false | \n| `--network` |  | Outbound network. Without `--allow-domain` : python reaches PyPI only, javascript npm only, raw any host | false | \n| `--allow-domain` |  | Allowed domain and its subdomains (repeatable; needs `--network` ) | - | \n| `--expose` |  | Expose port: `INTERNAL[:EXTERNAL][/tcp\\|udp]` (repeatable) | - | \n| `--upload` |  | Upload file `LOCAL:GUEST` (repeatable) | - | \n| `--download` |  | Download file `GUEST:LOCAL` , or`GUEST` to`./<file name>` (repeatable) | - | \n| `--instance` |  | Saved instance to continue (made at the first run), saved after the code runs ( [Instance ids](#instance-ids) ) | - | \n| `--json` |  | JSON output ( [JSON output](#json-output) ) | false | \n| `--quiet` | `-q` | Hide progress output | false | \n| `--debug` |  | DEBUG logging, and kernel and init boot logs on stderr (skips the warm pool and L1) | false | \n\n| Code | When | \n|---|---|\n| 0-255 | The exit code of the code ( [Exit codes and timeouts](#exit-codes-and-timeouts) ) | \n| 2 | Usage error (also `--allow-domain` without`--network` , or an invalid VM option such as`-m 100` or more than 10`--expose` ), or an error with action`FIX_INPUT` or`FIX_CONFIG` | \n| 124 | The code timed out (with `--json` ,`result.exit_code` stays -1), or the VM did not boot in time (`VM_BOOT_TIMEOUT` ) | \n| 125 | Other sandbox and instance errors | \n| 130 | Ctrl+C (a second Ctrl+C stops the VMs at once) | \n\nWith several sources, the run exits with 2 if a source has a `FIX_INPUT` or `FIX_CONFIG` error, else 124 if a source timed out or did not boot in time, else the highest exit code of the sources. When the code itself exits with 2 or 124, that is only its exit code. In the JSON array, the `exit_code` of a source that timed out is 124.\n\n| Sources | `--json` prints | Fields | \n|---|---|---|\n| One | One object | `result` (`stdout` ,`stderr` ,`exit_code` and the timings),`startup` ,`exposed_ports` ,`instance` and`total_ms` | \n| Several | An array of objects | `index` ,`source` ,`run` (the object above, or`null` ),`error` ,`code` ,`action` and`exit_code` | \n\nA single source that fails before its code runs prints no JSON: the error goes to stderr.\n\n| Command | Use | Exit codes | \n|---|---|---|\n| `sbx instance list\\|describe\\|keys\\|rm` | [Use instances from the command line](#use-instances-from-the-command-line) | 0 on success; 2 for invalid settings, a bad id, or a `FIX_INPUT` or`FIX_CONFIG` error; 125 for the other errors; 130 after Ctrl+C | \n| `sbx instance check` | [Use instances from the command line](#use-instances-from-the-command-line) | 0 when every instance passes; 1 when one fails; otherwise as `sbx instance list` | \n| `sbx worker describe` | [Use instances from the command line](#use-instances-from-the-command-line) | As `sbx instance list` | \n| `sbx prefetch` | [Offline hosts and Docker](#offline-hosts-and-docker) | 1 when a download fails | \n\n`sbx ARGS...` is `sbx run ARGS...`. `sbx -V` shows the version. `exec-sandbox` is the same command as `sbx`.\n\nThe docstring of each method (`help(Scheduler.new)`) lists its errors.\n\n| Member | Description | \n|---|---|\n| `Scheduler(config=None)` | Use as `async with Scheduler(config) as scheduler:` ;`None` uses the defaults ([SchedulerConfig](#schedulerconfig) ) | \n| `new(spec, *, idle_timeout_seconds=None, on_boot_log=None, instance_id=None)` | Start a new VM; returns its `Session` ([Sessions](#sessions) ) | \n| `reuse(instance_id, *, idle_timeout_seconds=None)` | Start a VM from a saved instance; returns the `Session` that continues it ([Saved instances](#saved-instances) ) | \n| `wait_pool_ready(timeout=120)` | Wait until the warm pool is full, or until `timeout` seconds pass ([Warm pool](#warm-pool) ) | \n| `describe_instance(instance_id)` | `InstanceInfo` of a saved instance ([List, describe and delete instances](#list-describe-and-delete-instances) ) | \n| `list_instances()` | Async iterator of the instance ids in the store | \n| `delete_instance(instance_id)` | Delete a saved instance; `True` if it existed | \n| `describe_worker()` | `WorkerFacts` of this worker: what a call that raised`RouteError` needs;`facts.satisfies(error.required)` says whether this worker can serve it | \n| `config` | The `SchedulerConfig` as given in code: a value set by an`EXEC_SANDBOX_*` variable only is not in it (the`Scheduler` uses it) | \n\n`instance_admin(config)` opens the instance store alone: `async with instance_admin(config) as admin:`, then `admin.describe(instance_id)`, `admin.list_ids()` and `admin.delete(instance_id)`, as the three `Scheduler` methods above ([List, describe and delete instances](#list-describe-and-delete-instances)).\n\n| Member | Description | \n|---|---|\n| `exec(code, *, timeout_seconds=None, env_vars=None, on_stdout=None, on_stderr=None)` | Run code; returns an `ExecutionResult` .`timeout_seconds` defaults to`default_timeout_seconds` ;`env_vars` is a dict ([Exit codes and timeouts](#exit-codes-and-timeouts) ) | \n| `write_file(path, content, *, make_executable=False)` | Write a file from bytes, a local `Path` or an`IO[bytes]` stream ([Files](#files) ) | \n| `read_file(path, *, destination)` | Read a file to a local `Path` or an`IO[bytes]` stream | \n| `list_files(path=\"\")` | List a folder; returns `FileInfo` entries (`name` ,`is_dir` ,`size` in bytes, 0 for folders) | \n| `save(*, instance_id=None)` | Save the VM state; returns an `InstanceRef` .`instance_id` : save under a new id and continue it ([Save under a new id](#save-under-a-new-id) ) | \n| `close()` | Destroy the VM; it never saves ( [Rules for save and reuse](#rules-for-save-and-reuse) ) | \n| `spec` | `VmSpec` : what the VM is made of | \n| `startup_timing` | `StartupTiming` : how the VM started (below) | \n| `exposed_ports` | `list[ExposedPort]` :`.internal` ,`.external` ,`.host` ,`.protocol` ,`.url` | \n| `exec_count` | `int` : number of`exec()` calls that returned a result | \n| `closed` | `bool` : the session is closed | \n| `vm_id` | `str` : id of the VM | \n| `instance` | `InstanceRef \\| None` : the saved instance this session continues, set by`reuse()` and by each`save()` | \n\n| Field | Type | Description | \n|---|---|---|\n| `stdout` | str | Output (at most 1 MB) | \n| `stderr` | str | Errors (at most 100 KB) | \n| `exit_code` | int | 0-255 (0 = success, 128+N = usually killed by signal N), or -1 for a timeout | \n| `execution_time_ms` | int \\| None | Run time that the VM reports | \n| `external_cpu_time_ms` | int \\| None | CPU time that the host measures | \n| `external_memory_peak_mb` | int \\| None | Peak memory that the host measures | \n| `external_cpu_nr_throttled` | int \\| None | CFS throttle events (host cgroup) | \n| `timing.execute_ms` | int | Connect + run + response, measured on the host | \n| `timing.connect_ms` | int \\| None | Connection to the guest agent (None when not measured) | \n| `spawn_ms` | int \\| None | Time to get the REPL process (guest) | \n| `process_ms` | int \\| None | Time from spawn to process exit (guest) | \n\nTimes in ms of the `new()` or `reuse()` call, measured on the host. After a cold boot, the interpreter starts in the first `exec()` and counts in its `timing.execute_ms`.\n\n| Field | Type | Description | \n|---|---|---|\n| `total_ms` | int | The whole call | \n| `admission_ms` | int | Wait for a resource slot | \n| `setup_ms` | int | Cache lookup, overlay, cgroup; after `reuse()` , the restore of the saved state | \n| `boot_ms` | int | Cold boot only | \n| `warm_pool_hit` | bool | The VM came from the warm pool | \n| `l1_cache_hit` | bool | The VM came from an L1 memory snapshot | \n| `l1_restore_ms` | int \\| None | L1 restore (None unless `l1_cache_hit` ) | \n| `instance_load_ms` | int \\| None | `reuse()` only: read, check and download of the saved state | \n| `overlay_ms` ,`qemu_cmd_build_ms` ,`gvproxy_start_ms` ,`qemu_fork_ms` ,`guest_wait_ms` ,`boot_retries` | int | Parts of a cold boot | \n\n`save()` returns an `InstanceRef` (`instance_id`, `generation`); pass the id to `reuse()`. `describe_instance()` returns an `InstanceInfo`:\n\n| Field | Type | Description | \n|---|---|---|\n| `instance_id` | str | The id given to `new(instance_id=...)` or`save(instance_id=...)` , else the one the first save made (32 lowercase hex characters) | \n| `generation` | int | 1 after the first save; +1 for each later save | \n| `recent_commits` | tuple[RecentCommit, ...] | The last saves ( `generation` ,`commit_id` ), newest first, at most 16 | \n| `key_id` | str | Id of the keyring key that protects the instance | \n| `access` | `\"full\"` ,`\"other_release\"` or`\"no_key\"` | `\"full\"` : this keyring opened the head and this release read it.`\"other_release\"` : the head opened, but another release wrote it and this release cannot read it.`\"no_key\"` : this keyring does not have the key, and the other fields are not checked | \n| `writer_release` | str \\| None | The exec-sandbox release of the last save; None when `access` is`\"no_key\"` , or when the body names no release | \n| `spec` | VmSpec \\| None | What the VM is made of (None unless `access` is`\"full\"` ) | \n| `host` | HostFacts \\| None | The host of the last save: `arch` ,`accelerator` ,`cpu_model` ,`cpu_id` ,`qemu_version` ,`machine_type` ,`release` ,`device_layout` (one line for each part of the VM the guest sees: the board, the RAM, each drive and device; then the migration capabilities) (None unless`access` is`\"full\"` ) | \n| `guest` | GuestFacts \\| None | What the guest got at its first boot: the SHA-256 digests of its files, `base_image` ,`packages_image` (None without packages),`kernel` ,`initramfs` , and`counter_hz` , the rate of its clock's counter (None when that host had no exact rate); every save carries them (None unless`access` is`\"full\"` ) | \n| `egress_domains` | tuple[str, ...] \\| None | The domains the VM may reach with the network on, as its last save resolved them; each reuse applies them (empty: all; None unless `access` is`\"full\"` ) | \n\n| Resource | Limit | \n|---|---|\n| Code size | 1 MiB | \n| stdout / stderr | 1 MB / 100 KB | \n| Packages | 50 (214 bytes each) | \n| Env vars | 100 (name: 256 bytes, value: 4096 bytes; no control characters except tab) | \n| Exposed ports | 10 | \n| File I/O | 500 MiB per file; path 4096 bytes (255 per component); 120 s wait for a guest answer (the end of a write, each chunk of a read, a listing) | \n| Timeouts | Execution 1-300 s; package install 300 s per try; VM boot 45 s per try, up to 4 tries | \n| VM memory | 128 MB minimum (no upper bound) | \n| Processes and threads of your code (user 1000) | 1024 ( `RLIMIT_NPROC` ) | \n| Concurrent VMs | Set by admission (host memory, CPU and disk: [SchedulerConfig](#schedulerconfig) ,[Disk admission](#disk-admission) ). A wait of 120 s fails with`TransientError` (`CAPACITY` ); a home disk that cannot fit fails at once with`TransientError` (`LOCAL_DISK` ) | \n| One saved-state object in S3 | About 78 GiB | \n\nPre-built images from [GitHub Releases](https://github.com/dualeai/exec-sandbox/releases), all based on Alpine Linux (musl libc):\n\n| Image | Runtime | Package manager | Size | \n|---|---|---|---|\n| `python-3.14-base` | Python 3.14 ( [python-build-standalone](https://github.com/astral-sh/python-build-standalone) , musl), cloudpickle 3.1.2 | uv 0.13 | ~75 MB | \n| `node-1.4-base` | Bun 1.4 | bun | ~71 MB | \n| `raw-base` | GNU Bash | None | ~17 MB | \n\nAll images include `bash`; the GNU `coreutils`, `grep` (PCRE with `-P`), `findutils`, `sed`, `gawk` and `diffutils`; and `git`, `curl`, `jq`, `openssl`, `patch`, `less`, `make`, `tree`, `tar`, `gzip`, `unzip` and `file`.\n\n- **Python.** Use`uv pip install` : there is no`pip` command. There is no gcc or musl-dev, so packages that need a source build do not install ([Packages](#packages) ).`multiprocessing.Pool` works with no setup, also with functions, lambdas and closures defined in the REPL. With the default`cpu_cores=1` , CPU-bound work does not run faster: set`VmSpec(cpu_cores=N)` or use several sessions.\n- **JavaScript.** Bun is a Node.js-compatible runtime, not Node.js.`exec()` runs JavaScript and TypeScript with no build step, but not JSX: put JSX in a`.jsx` or`.tsx` file and run it with`bun run` . Use`bun install` for packages and`bun run` for scripts.\n- **Shell (raw).** The common tools only, for shell scripts and runtimes that you install yourself.`Language.RAW` runs GNU Bash. Code that starts with`#!` runs as a script file in its own process: its variables and working folder do not stay for the next call.\n\n`memory_mb` holds the kernel (about 20 MB), your processes and the tmpfs mounts. Files in tmpfs use RAM: 10 MB of files use about 10 MB. Each tmpfs mount has a quota per user (UID).\n\n| Mount | Size | Purpose | \n|---|---|---|\n| `/home/user` | `VmSpec.disk_mb` (host disk, not RAM) | Writable home folder (ext4, nosuid, nodev) | \n| `/tmp` | 50% of RAM | Temp files, package manager scratch space | \n| `/dev/shm` | 40% of RAM | POSIX shared memory (Python multiprocessing semaphores) | \n\nCompressed swap (zram, lz4) adds 40% of RAM: code can use about 40% more than `memory_mb` (about 269 MB at 192 MB). The compressed data takes at most 20% of RAM. Use timeouts, not memory limits, to stop code that allocates memory without end.\n\n1. The `Scheduler` runs in your Python process. It takes a VM from the warm pool, restores it from a snapshot, or boots it ([Snapshot cache](#snapshot-cache) ). It uses KVM on Linux and HVF on macOS, else software emulation (TCG): it also runs on cloud VMs and in containers without nested virtualization.\n2. In the VM, tiny-init (PID 1) starts the guest agent. The agent gets the code over virtio-serial, runs it in a REPL as a non-root user, and sends stdout and stderr back while the code runs.\n3. The root filesystem is read-only (EROFS). `/home/user` is a writable ext4 disk.\n\n- An L1 save keeps the VM RAM in `/dev/shm` . In a container, give`/dev/shm` room for the RAM of each L1 save that runs at the same time (`docker run --shm-size` ; the default is 64 MB). Else the save is slower.\n- For a service that runs as a regular user (uid 1000 or more), run `loginctl enable-linger <user>` , or set`RemoveIPC=no` in`/etc/systemd/logind.conf` . Else an L1 save can fail when the last login session of that user ends (the next save tries again).\n\nHome disks are sparse. Each VM reserves the part of its `disk_mb` that it has not written yet, divided by `disk_overcommit_ratio`, and `EXEC_SANDBOX_DISK_FREE_FLOOR_MB` (1024 MB) stays free. Admission counts only the home disks of its own `Scheduler`; of other processes, it sees only the space that they use now.\n\nWhen space is short, admission removes the L1 and L2 cache entries on the same filesystem that no running VM uses (oldest last use first), then waits for VMs to end. When neither makes room, the request fails at once with `TransientError` (`LOCAL_DISK`).\n\n- VMs restored from one L1 memory snapshot share its RAM pages (copy-on-write).\n- Linux: KSM page merging, when the host runs KSM (`echo 1 | sudo tee /sys/kernel/mm/ksm/run` ).\n- Linux with cgroups v2 and host zswap on: exec-sandbox moves the memory of idle warm-pool VMs to zswap.\n\nTo report a vulnerability, follow the [security policy](https://github.com/dualeai/exec-sandbox/blob/main/SECURITY.md).\n\n| Layer | Technology | Protection | \n|---|---|---|\n| 1 | Hardware virtualization | CPU isolation enforced by hardware (KVM or HVF; not TCG) | \n| 2 | Hardened guest kernel | No modules, no io_uring, no eBPF, slab and memory hardening, about 530 kernel options turned off | \n| 3 | Unprivileged QEMU | QEMU runs as your user, or as `qemu-vm` on Linux ([Linux host setup](#linux-host-setup) ) | \n| 4 | Non-root REPL (UID 1000) | Blocks mount, ptrace, raw sockets, kernel modules | \n| 5 | seccomp | Linux, when QEMU supports `-sandbox` : a filter on the QEMU process | \n| 6 | cgroups v2 | Linux, with a delegated cgroup ( [Linux host setup](#linux-host-setup) ): memory, CPU and process limits per VM | \n| 7 | Namespaces | Linux, when `unshare` works for your user ([Linux host setup](#linux-host-setup) ): PID, network, mount, UTS and IPC namespaces for QEMU | \n| 8 | Socket authentication (SO_PEERCRED/LOCAL_PEERCRED) | Checks the identity of the QEMU process | \n\nAll VMs of a host run as the same user (layer 3): run workloads that must not share files on separate hosts.\n\n- **One VM per session.** Sessions never share a VM, and`close()` destroys it.\n- **No network by default.** Set`allow_network=True` to turn it on.\n- **Domain allow-list.** DNS does not resolve the domains that are not allowed. On port 443, the TLS SNI is checked, also against the DNS answers.\n- **Package check.** By default, only the top 10,000 PyPI and npm packages.\n- **Different random output in each VM** , also in VMs that start from one memory snapshot or one saved instance:[Random values in copies](#random-values-in-copies) .\n\nexec-sandbox checks each download against its SHA-256 digest ([How images download](#how-images-download)). Release assets have [provenance attestations](https://docs.github.com/en/actions/security-guides/using-artifact-attestations-to-establish-provenance-for-builds): check a file with `gh attestation verify <file> --repo dualeai/exec-sandbox`. Each [release](https://github.com/dualeai/exec-sandbox/releases) includes an SBOM (CycloneDX and SPDX) of the Python dependencies, not of the VM images.\n\nTo contribute, open an issue first to discuss the change.\n\n```\nmake install      # Set up the environment\nmake test         # Run tests\nmake lint         # Format and lint\nmake test-sudo    # Tests that need sudo (Linux: qemu-vm user, cgroups); make test does not run them\n```\n\n", "url": "https://wpnews.pro/news/linux-sandboxes-from-s3-93-99-cheaper-than-always-on-for-ai-agents", "canonical_source": "https://github.com/dualeai/exec-sandbox", "published_at": "2026-11-17 20:50:23+00:00", "updated_at": "2026-11-17 21:21:28.434567+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "developer-tools", "ai-infrastructure", "ai-safety"], "entities": ["exec-sandbox", "QEMU", "S3", "KVM", "HVF", "Python"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/linux-sandboxes-from-s3-93-99-cheaper-than-always-on-for-ai-agents", "markdown": "https://wpnews.pro/news/linux-sandboxes-from-s3-93-99-cheaper-than-always-on-for-ai-agents.md", "text": "https://wpnews.pro/news/linux-sandboxes-from-s3-93-99-cheaper-than-always-on-for-ai-agents.txt", "jsonld": "https://wpnews.pro/news/linux-sandboxes-from-s3-93-99-cheaper-than-always-on-for-ai-agents.jsonld"}}