cd /news/developer-tools/velarx-a-container-engine-with-a-cha… · home › topics › developer-tools › article
[ARTICLE · art-172619] src=scrivano.org ↗ pub= topic=developer-tools verified=true sentiment=↑ positive

Velarx: A container engine with a chain of trust from signature to page reads

A developer writing as scrivano.org released Velarx, a container engine written in Rust that builds on composefs and stores every regular file by its fs-verity digest, aiming to verify containers from the image signature down to every page the kernel reads. Velarx keeps containers, images, layers, networks, ports, volumes, secrets and build cache in a single SQLite database, uses one EROFS image per layer instead of per image, and replaces Podman's conmon with a per-container velarx-monitor process; the author benchmarked it against Podman 5.8.9 on a Fedora 44 VM with a 7.4-rc2 kernel, 16 vCPUs and 4 GB of memory. The author said AI was used heavily to prototype the engine, which runs real images and was compared with Podman.

key takeaway

Velarx stores each file once by fs-verity digest (SHA-256 over 4096-byte blocks, no salt) and builds one EROFS metadata image per layer, so container start cost is independent of image size.

by read12 min views1 publishedNov 19, 2026
Velarx: A container engine with a chain of trust from signature to page reads
Image: source

#

Velarx: a container engine with a chain of trust

Table of Contents #

For the last few weeks I’ve been working on Velarx, a container engine written in Rust that is built around composefs from the start. I used AI heavily to prototype it, otherwise in a few weeks I wouldn’t have gotten to something that runs real images and that I could compare with Podman. Like composefs, it stores every file by its fs-verity digest. The name comes from the Latin velox, swift, and arx, citadel, and those are the two goals: it has to be fast, and what it runs must be verified from the image signature down to every page the kernel reads.

The store# #

Every regular file of every layer is stored once under objects/, named by its fs-verity digest, as in a composefs repository. A layer is never unpacked into a directory. It is a table of contents and an EROFS image with only the metadata, where each regular file is a stub with trusted.overlay.redirect and trusted.overlay.metacopy pointing to the object and carrying its digest. It is the layout from the composefs sealing post, with one EROFS image per layer instead of one per image, so a layer shared by many images is built only once.

There isn’t a single fs-verity digest for a file: it depends on the hash algorithm, the Merkle tree block size and the salt.

Velarx uses the same parameters as composefs, SHA-256 (or SHA-512) over 4096-byte blocks and no salt, and the object names say it, as in fsverity-sha256-12:<hex> where 12 is the log2 of the block size.

All the per-file work, hashing, enabling fs-verity and writing the EROFS images, is done when the pull ends. Starting a container then costs the same whatever the size of the image, there is nothing to unpack or chown, just a few mounts.

Everything else is in a single SQLite database: containers, images, layers, the tables of contents, every file and the layers referencing it, networks and IP allocations, ports, volumes, secrets, the build cache and the namespaces of the rootless user. Podman spreads the same information across the containers/storage JSON files and their lock files, the libpod database, the netavark configuration and its IPAM database and a secrets file. With one database, a change that touches several of these is one transaction, and garbage collection is just a query for the files that no layer references anymore.

Other differences from Podman# #

Unlikely Podman there is no conmon external process to wait for the container process and call podman container cleanup. Instead Each container has a velarx-monitor process that knows how to do the cleanup so it won’t spawn a different process with all the complexity of the locking Podman has. The monitor also takes care to serve files to rootless through FUSE, emulating the EROFS kernel driver.

There is no process either. Rootless, all the processes of a user share one user namespace and one mount namespace for the boot, and nothing keeps them alive except the processes living in them. The database records the namespaces by the file handles of their nsfs files, which have a unique ID unlike inode numbers, so a command never joins a namespace that was recycled.

A daemon is not needed, each command can run the engine in its own process, similarly to Podman. When there is one, the REST API that the Podman and Docker clients use and the CRI server are separate processes that only translate their protocol and have no access to the store. The registry client and the decompression run in their own processes too, under Landlock and seccomp, so the code parsing what comes from the network never touches the store.

How fast is it# #

I compared Velarx with Podman 5.8.9 on a Fedora 44 VM with a 7.4-rc2 kernel, 16 vCPUs and 4 GB of memory, both as root and with crun 1.30, pulling from a registry on localhost. Podman doesn’t use fs-verity, so to compare the same work both stores were on the XFS root file system, where Velarx runs without it. The small image is alpine, one layer of 3.9 MB (8.7 MB uncompressed), the big one python:3.12, 7 layers and 412 MB (1.13 GB uncompressed), both with gzip layers as they are on Docker Hub:

operation Velarx Podman Velarx/Podman
pull, small image, empty store 169.9 ms ± 21.8 ms 213.0 ms ± 8.5 ms 0.80
pull, big image, empty store 2.850 s ± 159.2 ms 3.852 s ± 177.1 ms 0.74
run --rm small image 52.1 ms ± 4.6 ms 246.7 ms ± 6.9 ms 0.21
run --rm --network none small image 33.4 ms ± 2.9 ms 148.1 ms ± 3.6 ms 0.23
run --rm big image 56.1 ms ± 7.1 ms 252.2 ms ± 14.9 ms 0.22
create, start and rm 144.7 ms ± 29.0 ms 319.0 ms ± 8.1 ms 0.45
rm 31.9 ms ± 1.7 ms 62.9 ms ± 1.9 ms 0.51
rmi 25.0 ms ± 1.5 ms 38.9 ms ± 1.9 ms 0.64
exec true 18.0 ms ± 0.7 ms 123.9 ms ± 3.1 ms 0.15

Running python:3.12 costs the same as running alpine, and exec is about 7 times faster.

The chain of trust# #

The signature in policy.json covers the manifest. The manifest has the digests of the config and of the layers, and for each layer an io.containers.composefs.digest annotation with the fs-verity digest of its EROFS image. The EROFS image has the fs-verity digest of each file, and the kernel checks each file against it:

|

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

|

policy.json
  |  signature over the manifest
  v
manifest
  |  io.containers.composefs.digest of each layer
  v
EROFS image of each layer
  |  fs-verity digest of each file in the stubs
  v
objects/<fs-verity digest>
  |  overlayfs verity=on or verity=require
  v
fs-verity checks every page on read

|

For this to work the images are pushed in a modified zstd:chunked format. The table of contents of zstd:chunked already lists every file and its chunks. Velarx adds an fsverity field with the fs-verity digest of each regular file, written as sha256-12:<hex> so that the algorithm and the block size are part of the value, and the io.containers.composefs.digest annotation to each layer in the manifest. With the digests in the table of contents, a puller knows the name each file will have in the store before down it, so it can skip what it already has, ask other nodes for it, and verify each file as it arrives. With the EROFS digest, the image the puller builds from the table of contents can be compared with what the publisher signed. Images pushed by other tools have neither, so they are pulled eagerly and the digests are computed locally. They still run, but the chain stops at the manifest.

At every start the engine measures each layer image with FS_IOC_MEASURE_VERITY, which takes constant time since the kernel already has the Merkle tree, and compares it with the manifest. The same open file is then passed to EROFS with FSCONFIG_SET_FD, so nothing can be swapped between the check and the mount. At runtime overlayfs refuses with EIO any file whose digest doesn’t match the stub, and with verity=require also any file without one.

This doesn’t protect you from whoever owns the store, root or the rootless user can still edit the database and replace files. What they cannot do is change what a container runs without the start check or the kernel noticing, as long as the manifest is signed and the digests are required.

The cost of computing the Merkle trees is paid once, at pull time, when every file pulled is sealed with fs-verity. On an ext4 with fs-verity and verity=require, the cold pull of python:3.12 takes 5.2 s instead of the 4.1 s it takes on the same ext4 without it, while starting a container costs the same.

Peer-to-peer# #

Another thing I wanted to experiment with is pulling images from peers.

Since every file is identified by its digest, it doesn’t matter who sends it. velarx-p2p is an optional service that lets nodes pull from each other before going to the registry:

|

1
2
3
4
5
6
7

|

[[sources.external]]
name = "peers"
socket = "/run/velarx/services/p2p-fetcher.sock"
peer = true

[pull]
sources = ["peers", "registry"]

|

What a peer sends goes through the same checks as what a registry sends, so a bad peer can make a pull slower but not wrong. Nodes are admitted with a pre-shared key or with certificates.

Each node tells the others which layers it has with a Bloom filter, so a pull knows whom to ask without asking everyone first. The interesting case is a rollout, where all the nodes pull the same image at the same time. A node shares each file as soon as it has verified it, without waiting for the rest of the layer, and each node fetches the files in a different order, so they trade files with each other instead of waiting for the first one to finish. With 20 nodes on 1 Gbit/s links, pulling 300 MB went from 47 to 25 seconds.

Rootless and FUSE# #

As root, Velarx mounts the EROFS image of each layer, stacks them as overlayfs lower layers and adds objects/ as a data-only layer with datadir+. Rootless it can’t: EROFS cannot be mounted in a user namespace, and with userxattr overlayfs reads user.overlay.* instead of the trusted.overlay.* xattrs of the stubs. So rootless Velarx serves the EROFS images through FUSE, one directory per layer, translating the owners to the ids of the container and the trusted.overlay.* xattrs to user.overlay.*:

|

1
2
3

|

mount -t overlay overlay \
    -o userxattr,lowerdir+=layer3,lowerdir+=layer2,lowerdir+=layer1,datadir+=objects \
    merged

|

FUSE is supposed to serve only the metadata. On open, overlayfs follows the stub to the object and reads it directly, so no read goes through userspace and the page cache of an object is shared among all the containers using it.

It worked with single layer images and broke as soon as there was more than one layer: a stub in the lowest layer works, the same stub one layer above it fails with EPERM, and the kernel logs overlayfs: refusing to follow metacopy origin.

The overlayfs fix# #

Following redirects from a lower layer into a data-only layer without metacopy=on was allowed in 6.16 by commit 5ef7bcdeecc9 (“ovl: relax redirect/metacopy requirements for lower -> data redirect”), exactly so that data-only layers could be used with userxattr. The problem is that ovl_lookup_layers() keeps walking the lower layers after it finds a metacopy file. If the stub is in the lowest layer the loop ends and the data is looked up in the data-only layers. If it is in any other layer, the next iteration calls ovl_check_follow_redirect():

|

1
2
3
4
5
6
7
8
9

|

static bool ovl_check_follow_redirect(struct ovl_lookup_data *d)
{
	struct ovl_fs *ofs = OVL_FS(d->sb);

	if (d->metacopy && !ofs->config.metacopy) {
		pr_warn_ratelimited("refusing to follow metacopy origin for (%pd2)\n", d->dentry);
		return false;
	}
	...

|

and since metacopy=on can’t be used with userxattr, the lookup fails with EPERM.

The same commit also changed what redirect_dir=nofollow means without showing it: with a data-only layer some redirects were followed while the mount options said redirect_dir=nofollow, or nothing when that was the default.

The fix, cbf4ccabcbfd (“ovl: follow lower to lowerdata redirects from any lower layer”), is in 7.4-rc3 and marked for stable. The lookup now stops at a lower metacopy file with an absolute redirect, when there are data-only layers and no metacopy=on, and looks the data up only in the data-only layers:

|

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

|

/*
 * Without metacopy=on, the absolute redirect of a lower metacopy file is
 * followed only into the data-only layers.
 */
static bool ovl_is_lowerdata_redirect(struct ovl_lookup_ctx *ctx,
				      struct ovl_lookup_data *d)
{
	struct ovl_fs *ofs = OVL_FS(d->sb);

	return ofs->numdatalayer && !ofs->config.metacopy && ctx->ctr &&
	       d->metacopy && d->absolute_redirect;
}

|

It also adds redirect_dir=followdata, which follows only redirects from a lower layer into a data-only layer. It is selected automatically when there are data-only layers and nothing more permissive was asked for, so the mount options now show what the kernel does, and redirect_dir=nofollow with data-only layers is refused since those layers can only be reached through redirects.

Older kernels# #

Velarx still has to run on kernels without the fix, so the first rootless container with more than one layer mounts a small overlay on a detached tmpfs, with redirect_dir=followdata and a stub in a middle layer, and tries to read it. If it works, all the layers keep their stubs. If not, the FUSE server removes the stubs from every layer but the lowest and serves the files of the upper layers itself, from userspace. FUSE passthrough would help, but it needs CAP_SYS_ADMIN in the initial user namespace.

To see the cost, on a VM with the stock 7.3 kernel I made an image with a 512 MB file /low in its only layer, another image on top of it with a 512 MB file /top, and ran a rootless container from the second one. Its overlay looks like this (paths shortened):

|

1

|

overlay / ... lowerdir+=.../layer2,lowerdir+=.../layer1,datadir+=.../objects,upperdir=...,workdir=...,redirect_dir=nofollow,uuid=on,userxattr

|

There is the redirect_dir=nofollow from before, while the stubs of layer1 are followed into objects. Reading both files with a cold cache:

|

1
2
3
4
5
6
7

|

Benchmark 1: velarx exec fb /busybox cat /low
  Time (mean ± σ):     215.7 ms ±   6.2 ms    [User: 49.4 ms, System: 103.3 ms]
  Range (min … max):   206.7 ms … 227.3 ms    8 runs

Benchmark 2: velarx exec fb /busybox cat /top
  Time (mean ± σ):     314.1 ms ±  10.4 ms    [User: 54.7 ms, System: 101.3 ms]
  Range (min … max):   304.5 ms … 335.8 ms    8 runs

|

The file served by FUSE is about 46% slower to read, and its page cache belongs to the FUSE inode, so it is not shared with other containers. With the fix both files are read by overlayfs directly from the store.

The source is at github.com/giuseppe/velarx.

── more in #developer-tools 4 stories · sorted by recency
── more on @velarx 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/velarx-a-container-e…] indexed:0 read:12min 2026-11-19 · —