OpenZL v0.3.0, compression with a Neural Network

Letting a neural network build the compression graph

OpenZL compresses data by chaining processing layers in any order. This is powerful, but requires careful configuration: choosing the right combination of codecs for a given input has always been its greatest challenge. The Compression Transformer now makes that choice automatically, building the compression graph on the fly, one decision at a time. It needs no per-source training, no manual tuning, and no change on the decompression side. It is available from both the API and the CLI.

The Power — and Complexity — of Graph Compression #

OpenZL losslessly compresses data using a collection of codecs that can be freely combined into graphs. A graph can push one input stream through several stages, split it into child streams, and choose a different codec for each one. This flexibility is powerful: in theory, searching the full space of possible graphs would find the best compression strategy for every input. In practice, that search is far too expensive to perform while data is being compressed, so production systems must settle for simpler strategies that are broadly effective.

One can tune a graph by hand, but it is time-consuming and does not scale well. That is why OpenZL comes with a trainer, zli train, which analyzes representative samples from a data source and generates a graph optimized for that traffic. ACE produces multiple graphs spanning different compression-ratio and speed trade-offs, letting the user choose which one best corresponds to their constraint.

This approach depends on two assumptions: that the samples represent future traffic, and that the traffic is reasonably homogeneous. Those assumptions often hold. When they do not, the trainer has to settle on a middle ground — a graph that works adequately across the different shapes in the sample, but is optimal for none of them.

The natural next step is to adapt the graph to each individual input, in real time. Choosing the graph per input could handle heterogeneous traffic, respond to outliers, and absorb unannounced changes in the data, all without manual intervention or source-specific training. This is the capability that the Compression Transformer brings to OpenZL.

How the Compression Transformer Works #

The Compression Transformer generates a compression graph from a data stream, one decision at a time. There is a useful analogy to a large language model generating text, hence the name: at each step, the model emits a token conditioned on its current context. In OpenZL, that token determines a codec and its set of parameters, that is, the next node in the graph. The produced output is not a linear sequence, however. Multi-stage codecs produce child streams, and the model generates decisions for each child, recursively building a graph.

For each new stream, a specialized scorer — a small multilayer perceptron (MLP), one per context (for now, one per numeric element width) — evaluates every candidate codec. The selector picks the highest-scoring valid codec and applies it. If that codec produces child streams, they are fed back into the selector. In this way, the compression graph assembles itself one node at a time.

A few design choices make this practical and robust:

• Skip stage: trivial inputs, such as empty or constant streams, are handled directly without invoking the neural network.
• Score guards: deterministic rules reject impossible or nonsensical operations after the scorer has ranked the candidates. They also drop codecs that are unavailable in the requested format version, so the produced graph is valid for target decompressors.
• Depth guard and static fallback: recursion is bounded, and low-confidence decisions fall back to a deterministic decision tree.
• Standard integration: the Transformer plugs into OpenZL as a regular selector. It needs no special execution framework.

The model only runs at compression time. The produced result is an ordinary OpenZL frame, read back by the same universal decompressor as usual.

In its first implementation, the Transformer is focused on numeric types, and ships four scorers, one per numeric element width (1, 2, 4 and 8 bytes).

Results Overview #

We have evaluated the numeric Transformer against 868 families of numeric streams, totalling 34,737 files and 17.9 GB, each family representing a different data shape. In the results presented below, the Transformer is compared against OpenZL, untrained and trained, and against the strongest settings of two widely used general-purpose compressors, zstd -19 and xz -9. These are compression-ratio references, not speed ones: the Transformer compresses much faster than zstd -19, at speeds comparable to zstd -8 to -12 depending on data width.

Figures are weighted geometric means of compression ratios. "OpenZL untrained" is OpenZL’s default numeric path at the default compression level (6), which compresses with Field LZ; "OpenZL trained" uses a graph trained with ACE separately for each family.

The Transformer lands within 1.1% of OpenZL trained overall, without needing any per-family training.

For details per data type, please consult the following tables:

num8 — best, median and worst of 151 families #

num16 — best, median and worst of 226 families #

num32 — best, median and worst of 249 families #

num64 — best, median and worst of 242 families #

At the time of writing, the Transformer does well on a broad range of numeric data shapes, but its gains are uneven, and it still makes mistakes: it trails zstd -19 on 95 of the 868 families, by up to 2x in the worst case. This is still early days, and we expect these gaps to shrink as the model improves.

Trying It Out #

The Transformer ships with OpenZL v0.3.0, and is currently opt-in. From the CLI, you can see the Transformer in action by requesting compression level 7 or above. It works for direct numeric streams, or for numeric child streams, for example extracted from a parsing operation (like SDDL).

./zli compress --profile le-i32 --level 7 examples/getting_started/sample_inputs/era5_ints.bin --output era5_ints.zl

On this sample, the compression ratio goes from 18.95x at the default level to 29.41x at level 7.

From the C API, select ZL_GRAPH_TRANSFORMER_NUMERIC as the starting graph, or ZL_GRAPH_NUMERIC with compression level 7 or above:

#include "openzl/codecs/zl_transformer.h"

ZL_Report r = ZL_Compressor_selectStartingGraphID(
        compressor, ZL_GRAPH_TRANSFORMER_NUMERIC);

From C++, the same graph is available as openzl::graphs::TransformerNumeric:

#include "openzl/cpp/codecs/Transformer.hpp"

compressor.selectStartingGraph(openzl::graphs::TransformerNumeric::graph);

It accepts one or more numeric streams of width 1, 2, 4 or 8 bytes.

If you want to see what the model decided, add --trace era5_ints.cbor to the command above and open that file in the graph visualizer — the Transformer’s choices show up as ordinary graph nodes, since that is what they are.

What’s Next #

Two directions are already clear.

First, we want to take the Transformer beyond numeric streams. Other stream types, such as string, will be covered in future expansions.

Second, we want to make speed an explicit target, so the selector can find the best compression ratio within a stated performance budget, instead of just aiming at the strongest option.

Those are subjects for a future post. The larger point is already here: a compression graph no longer has to be designed or trained ahead of time. It can be built on the fly, for each input, by a model that reads the data as it goes, so when the data changes, the graph changes with it. And since the result is an ordinary OpenZL frame, that model can keep getting smarter over time, without the decompressor having to change or be redeployed.