VaeZip is a PyTorch-based library that implements several latent quantization codecs and stores the compressed tensors in common Safetensors files that are much smaller then if you store these tensors in fp32.
The main use cases of this are
The codecs are designed specifically for VAE latents: some of them exploit both spatial redundancy and that the VAE latent distribution is usually normal. That allows to store latent tensor with about 3 bits/value precision (10x smaller files) while keeping pixel MSE at 0.0003 after decoding.
For example, one of the best codecs uses a combination of global range scaling, local range scaling with square spatial blocks and so called “Lloyd-Max table” which is essentially codebook that is tuned for the sample distribution. After that it quantizes values to 2-8 bits and packs them. The codecs were selected from the benchmark that compared ~50 algorithms on 8 image and video VAEs by the reconstruction quality, fidelity, space usage and I/O latency.
I have also measured the impact on pre-training a small T2I flow matching DiT with latents compressed with different codecs. In one run the loss curve is indistinguishable from the fp32 baseline with just 6 bit storage.