{"slug": "velarx-a-container-engine-with-a-chain-of-trust-from-signature-to-page-reads", "title": "Velarx: A container engine with a chain of trust from signature to page reads", "summary": "A developer writing as scrivano.org released Velarx, a container engine written in Rust that builds on composefs and stores every regular file by its fs-verity digest, aiming to verify containers from the image signature down to every page the kernel reads. Velarx keeps containers, images, layers, networks, ports, volumes, secrets and build cache in a single SQLite database, uses one EROFS image per layer instead of per image, and replaces Podman's conmon with a per-container velarx-monitor process; the author benchmarked it against Podman 5.8.9 on a Fedora 44 VM with a 7.4-rc2 kernel, 16 vCPUs and 4 GB of memory. The author said AI was used heavily to prototype the engine, which runs real images and was compared with Podman.", "body_md": "# \n[Velarx: a container engine with a chain of trust](https://www.scrivano.org/posts/2026-11-18-velarx/)\n\n## Table of Contents\n\nFor the last few weeks I’ve been working on Velarx, a container engine\nwritten in Rust that is built around composefs from the start.  I used\nAI heavily to prototype it, otherwise in a few weeks I wouldn’t have\ngotten to something that runs real images and that I could compare with\nPodman.  Like composefs, it stores every file by its fs-verity digest.\nThe name comes from the Latin *velox*, swift, and *arx*, citadel, and\nthose are the two goals: it has to be fast, and what it runs must be\nverified from the image signature down to every page the kernel reads.\n\n## The store[#](#the-store) \n\nEvery regular file of every layer is stored once under `objects/`,\nnamed by its fs-verity digest, as in a composefs repository.  A layer\nis never unpacked into a directory.  It is a table of contents and an\nEROFS image with only the metadata, where each regular file is a stub\nwith `trusted.overlay.redirect` and `trusted.overlay.metacopy` pointing\nto the object and carrying its digest.  It is the layout from the\n[composefs sealing post](https://www.scrivano.org/posts/2026-06-05-sealing-with-composefs/),\nwith one EROFS image per layer instead of one per image, so a layer\nshared by many images is built only once.\n\nThere isn’t a single fs-verity digest for a file: it depends on the hash algorithm, the Merkle tree block size and the salt.\n\nVelarx uses the same parameters as composefs, SHA-256 (or SHA-512) over\n4096-byte blocks and no salt, and the object names say it, as in\n`fsverity-sha256-12:<hex>` where 12 is the log2 of the block size.\n\nAll the per-file work, hashing, enabling fs-verity and writing the EROFS images, is done when the pull ends. Starting a container then costs the same whatever the size of the image, there is nothing to unpack or chown, just a few mounts.\n\nEverything else is in a single SQLite database: containers, images, layers, the tables of contents, every file and the layers referencing it, networks and IP allocations, ports, volumes, secrets, the build cache and the namespaces of the rootless user. Podman spreads the same information across the containers/storage JSON files and their lock files, the libpod database, the netavark configuration and its IPAM database and a secrets file. With one database, a change that touches several of these is one transaction, and garbage collection is just a query for the files that no layer references anymore.\n\n## Other differences from Podman[#](#other-differences-from-podman) \n\nUnlikely Podman there is no conmon external process to wait for the container process\nand call `podman container cleanup`.  Instead Each container has a `velarx-monitor`\nprocess that knows how to do the cleanup so it won’t spawn a different process with\nall the complexity of the locking Podman has.  The monitor also takes care to serve\nfiles to rootless through FUSE, emulating the EROFS kernel driver.\n\nThere is no pause process either. Rootless, all the processes of a user share one user namespace and one mount namespace for the boot, and nothing keeps them alive except the processes living in them. The database records the namespaces by the file handles of their nsfs files, which have a unique ID unlike inode numbers, so a command never joins a namespace that was recycled.\n\nA daemon is not needed, each command can run the engine in its own process, similarly to Podman. When there is one, the REST API that the Podman and Docker clients use and the CRI server are separate processes that only translate their protocol and have no access to the store. The registry client and the decompression run in their own processes too, under Landlock and seccomp, so the code parsing what comes from the network never touches the store.\n\n## How fast is it[#](#how-fast-is-it) \n\nI compared Velarx with Podman 5.8.9 on a Fedora 44 VM with a 7.4-rc2\nkernel, 16 vCPUs and 4 GB of memory, both as root and with crun 1.30,\npulling from a registry on localhost.  Podman doesn’t use fs-verity, so\nto compare the same work both stores were on the XFS root file system,\nwhere Velarx runs without it.  The small image is `alpine`, one layer\nof 3.9 MB (8.7 MB uncompressed), the big one `python:3.12`, 7 layers\nand 412 MB (1.13 GB uncompressed), both with gzip layers as they are on\nDocker Hub:\n\n| operation | Velarx | Podman | Velarx/Podman | \n|---|---|---|---|\n| pull, small image, empty store | 169.9 ms ± 21.8 ms | 213.0 ms ± 8.5 ms | 0.80 | \n| pull, big image, empty store | 2.850 s ± 159.2 ms | 3.852 s ± 177.1 ms | 0.74 | \n| `run --rm` small image | 52.1 ms ± 4.6 ms | 246.7 ms ± 6.9 ms | 0.21 | \n| `run --rm --network none` small image | 33.4 ms ± 2.9 ms | 148.1 ms ± 3.6 ms | 0.23 | \n| `run --rm` big image | 56.1 ms ± 7.1 ms | 252.2 ms ± 14.9 ms | 0.22 | \n| create, start and rm | 144.7 ms ± 29.0 ms | 319.0 ms ± 8.1 ms | 0.45 | \n| `rm` | 31.9 ms ± 1.7 ms | 62.9 ms ± 1.9 ms | 0.51 | \n| `rmi` | 25.0 ms ± 1.5 ms | 38.9 ms ± 1.9 ms | 0.64 | \n| `exec true` | 18.0 ms ± 0.7 ms | 123.9 ms ± 3.1 ms | 0.15 | \n\nRunning `python:3.12` costs the same as running `alpine`, and `exec` is\nabout 7 times faster.\n\n## The chain of trust[#](#the-chain-of-trust) \n\nThe signature in `policy.json` covers the manifest.  The manifest has\nthe digests of the config and of the layers, and for each layer an\n`io.containers.composefs.digest` annotation with the fs-verity digest\nof its EROFS image.  The EROFS image has the fs-verity digest of each\nfile, and the kernel checks each file against it:\n\n| \n\n```\n 1\n 2\n 3\n 4\n 5\n 6\n 7\n 8\n 9\n10\n11\n12\n13\n```\n\n | \n\n```\npolicy.json\n  |  signature over the manifest\n  v\nmanifest\n  |  io.containers.composefs.digest of each layer\n  v\nEROFS image of each layer\n  |  fs-verity digest of each file in the stubs\n  v\nobjects/<fs-verity digest>\n  |  overlayfs verity=on or verity=require\n  v\nfs-verity checks every page on read\n```\n\n | \n\nFor this to work the images are pushed in a modified zstd:chunked\nformat.  The table of contents of zstd:chunked already lists every file\nand its chunks.  Velarx adds an `fsverity` field with the fs-verity\ndigest of each regular file, written as `sha256-12:<hex>` so that the\nalgorithm and the block size are part of the value, and the\n`io.containers.composefs.digest` annotation to each layer in the\nmanifest.  With the digests in the\ntable of contents, a puller knows the name each file will have in the\nstore before downloading it, so it can skip what it already has, ask\nother nodes for it, and verify each file as it arrives.  With the EROFS\ndigest, the image the puller builds from the table of contents can be\ncompared with what the publisher signed.  Images pushed by other tools\nhave neither, so they are pulled eagerly and the digests are computed\nlocally.  They still run, but the chain stops at the manifest.\n\nAt every start the engine measures each layer image with\n`FS_IOC_MEASURE_VERITY`, which takes constant time since the kernel\nalready has the Merkle tree, and compares it with the manifest.  The\nsame open file is then passed to EROFS with `FSCONFIG_SET_FD`, so\nnothing can be swapped between the check and the mount.  At runtime\noverlayfs refuses with `EIO` any file whose digest doesn’t match the\nstub, and with `verity=require` also any file without one.\n\nThis doesn’t protect you from whoever owns the store, root or the rootless user can still edit the database and replace files. What they cannot do is change what a container runs without the start check or the kernel noticing, as long as the manifest is signed and the digests are required.\n\nThe cost of computing the Merkle trees is paid once, at pull time, when\nevery file pulled is sealed with fs-verity.  On an ext4 with fs-verity\nand `verity=require`, the cold pull of `python:3.12` takes 5.2 s\ninstead of the 4.1 s it takes on the same ext4 without it, while\nstarting a container costs the same.\n\n## Peer-to-peer[#](#peer-to-peer) \n\nAnother thing I wanted to experiment with is pulling images from peers.\n\nSince every file is identified by its digest, it doesn’t matter who\nsends it.  `velarx-p2p` is an optional service that lets nodes pull\nfrom each other before going to the registry:\n\n| \n\n```\n1\n2\n3\n4\n5\n6\n7\n```\n\n | \n\n```\n[[sources.external]]\nname = \"peers\"\nsocket = \"/run/velarx/services/p2p-fetcher.sock\"\npeer = true\n\n[pull]\nsources = [\"peers\", \"registry\"]\n```\n\n | \n\nWhat a peer sends goes through the same checks as what a registry sends, so a bad peer can make a pull slower but not wrong. Nodes are admitted with a pre-shared key or with certificates.\n\nEach node tells the others which layers it has with a Bloom filter, so a pull knows whom to ask without asking everyone first. The interesting case is a rollout, where all the nodes pull the same image at the same time. A node shares each file as soon as it has verified it, without waiting for the rest of the layer, and each node fetches the files in a different order, so they trade files with each other instead of waiting for the first one to finish. With 20 nodes on 1 Gbit/s links, pulling 300 MB went from 47 to 25 seconds.\n\n## Rootless and FUSE[#](#rootless-and-fuse) \n\nAs root, Velarx mounts the EROFS image of each layer, stacks them as\noverlayfs lower layers and adds `objects/` as a data-only layer with\n`datadir+`.  Rootless it can’t: EROFS cannot be mounted in a user\nnamespace, and with `userxattr` overlayfs reads `user.overlay.*`\ninstead of the `trusted.overlay.*` xattrs of the stubs.  So rootless\nVelarx serves the EROFS images through FUSE, one directory per layer,\ntranslating the owners to the ids of the container and the\n`trusted.overlay.*` xattrs to `user.overlay.*`:\n\n| \n\n```\n1\n2\n3\n```\n\n | \n\n```\nmount -t overlay overlay \\\n    -o userxattr,lowerdir+=layer3,lowerdir+=layer2,lowerdir+=layer1,datadir+=objects \\\n    merged\n```\n\n | \n\nFUSE is supposed to serve only the metadata. On open, overlayfs follows the stub to the object and reads it directly, so no read goes through userspace and the page cache of an object is shared among all the containers using it.\n\nIt worked with single layer images and broke as soon as there was more\nthan one layer: a stub in the lowest layer works, the same stub one\nlayer above it fails with `EPERM`, and the kernel logs `overlayfs: refusing to follow metacopy origin`.\n\n## The overlayfs fix[#](#the-overlayfs-fix) \n\nFollowing redirects from a lower layer into a data-only layer without\n`metacopy=on` was allowed in 6.16 by commit 5ef7bcdeecc9 (“ovl: relax\nredirect/metacopy requirements for lower -> data redirect”), exactly so\nthat data-only layers could be used with `userxattr`.  The problem is\nthat `ovl_lookup_layers()` keeps walking the lower layers after it\nfinds a metacopy file.  If the stub is in the lowest layer the loop\nends and the data is looked up in the data-only layers.  If it is in\nany other layer, the next iteration calls\n`ovl_check_follow_redirect()`:\n\n| \n\n```\n1\n2\n3\n4\n5\n6\n7\n8\n9\n```\n\n | \n\n```\nstatic bool ovl_check_follow_redirect(struct ovl_lookup_data *d)\n{\n\tstruct ovl_fs *ofs = OVL_FS(d->sb);\n\n\tif (d->metacopy && !ofs->config.metacopy) {\n\t\tpr_warn_ratelimited(\"refusing to follow metacopy origin for (%pd2)\\n\", d->dentry);\n\t\treturn false;\n\t}\n\t...\n```\n\n | \n\nand since `metacopy=on` can’t be used with `userxattr`, the lookup\nfails with `EPERM`.\n\nThe same commit also changed what `redirect_dir=nofollow` means without\nshowing it: with a data-only layer some redirects were followed while\nthe mount options said `redirect_dir=nofollow`, or nothing when that\nwas the default.\n\nThe fix,\n[cbf4ccabcbfd](https://git.kernel.org/torvalds/c/cbf4ccabcbfdda3da70c10b37e21e99cd2ee0946)\n(“ovl: follow lower to lowerdata redirects from any lower layer”), is\nin 7.4-rc3 and marked for stable.  The lookup now stops at a lower\nmetacopy file with an absolute redirect, when there are data-only\nlayers and no `metacopy=on`, and looks the data up only in the\ndata-only layers:\n\n| \n\n```\n 1\n 2\n 3\n 4\n 5\n 6\n 7\n 8\n 9\n10\n11\n12\n```\n\n | \n\n```\n/*\n * Without metacopy=on, the absolute redirect of a lower metacopy file is\n * followed only into the data-only layers.\n */\nstatic bool ovl_is_lowerdata_redirect(struct ovl_lookup_ctx *ctx,\n\t\t\t\t      struct ovl_lookup_data *d)\n{\n\tstruct ovl_fs *ofs = OVL_FS(d->sb);\n\n\treturn ofs->numdatalayer && !ofs->config.metacopy && ctx->ctr &&\n\t       d->metacopy && d->absolute_redirect;\n}\n```\n\n | \n\nIt also adds `redirect_dir=followdata`, which follows only redirects\nfrom a lower layer into a data-only layer.  It is selected\nautomatically when there are data-only layers and nothing more\npermissive was asked for, so the mount options now show what the\nkernel does, and `redirect_dir=nofollow` with data-only layers is\nrefused since those layers can only be reached through redirects.\n\n## Older kernels[#](#older-kernels) \n\nVelarx still has to run on kernels without the fix, so the first\nrootless container with more than one layer mounts a small overlay on a\ndetached tmpfs, with `redirect_dir=followdata` and a stub in a middle\nlayer, and tries to read it.  If it works, all the layers keep their\nstubs.  If not, the FUSE server removes the stubs from every layer but\nthe lowest and serves the files of the upper layers itself, from\nuserspace.  FUSE passthrough would help, but it needs `CAP_SYS_ADMIN`\nin the initial user namespace.\n\nTo see the cost, on a VM with the stock 7.3 kernel I made an image\nwith a 512 MB file `/low` in its only layer, another image on top of it\nwith a 512 MB file `/top`, and ran a rootless container from the second\none.  Its overlay looks like this (paths shortened):\n\n| \n\n```\n1\n```\n\n | \n\n```\noverlay / ... lowerdir+=.../layer2,lowerdir+=.../layer1,datadir+=.../objects,upperdir=...,workdir=...,redirect_dir=nofollow,uuid=on,userxattr\n```\n\n | \n\nThere is the `redirect_dir=nofollow` from before, while the stubs of\n`layer1` are followed into `objects`.  Reading both files with a cold\ncache:\n\n| \n\n```\n1\n2\n3\n4\n5\n6\n7\n```\n\n | \n\n```\nBenchmark 1: velarx exec fb /busybox cat /low\n  Time (mean ± σ):     215.7 ms ±   6.2 ms    [User: 49.4 ms, System: 103.3 ms]\n  Range (min … max):   206.7 ms … 227.3 ms    8 runs\n\nBenchmark 2: velarx exec fb /busybox cat /top\n  Time (mean ± σ):     314.1 ms ±  10.4 ms    [User: 54.7 ms, System: 101.3 ms]\n  Range (min … max):   304.5 ms … 335.8 ms    8 runs\n```\n\n | \n\nThe file served by FUSE is about 46% slower to read, and its page cache belongs to the FUSE inode, so it is not shared with other containers. With the fix both files are read by overlayfs directly from the store.\n\nThe source is at [github.com/giuseppe/velarx](https://github.com/giuseppe/velarx).", "url": "https://wpnews.pro/news/velarx-a-container-engine-with-a-chain-of-trust-from-signature-to-page-reads", "canonical_source": "https://scrivano.org/posts/2026-11-18-velarx/", "published_at": "2026-11-19 15:02:21+00:00", "updated_at": "2026-11-19 15:15:40.814240+00:00", "lang": "en", "topics": ["developer-tools", "ai-tools"], "entities": ["Velarx", "composefs", "Podman", "Rust", "Fedora", "scrivano.org"], "also_reported_by": [], "takeaway": "Velarx stores each file once by fs-verity digest (SHA-256 over 4096-byte blocks, no salt) and builds one EROFS metadata image per layer, so container start cost is independent of image size.", "recommendation": null, "alternates": {"html": "https://wpnews.pro/news/velarx-a-container-engine-with-a-chain-of-trust-from-signature-to-page-reads", "markdown": "https://wpnews.pro/news/velarx-a-container-engine-with-a-chain-of-trust-from-signature-to-page-reads.md", "text": "https://wpnews.pro/news/velarx-a-container-engine-with-a-chain-of-trust-from-signature-to-page-reads.txt", "jsonld": "https://wpnews.pro/news/velarx-a-container-engine-with-a-chain-of-trust-from-signature-to-page-reads.jsonld"}}