Requential Coding <1 bit compression with better generalization

Researchers introduced Requential Coding, a method that compresses generative models by coding a training process using self-generated data, achieving under 1 bit per parameter compression with better generalization. The technique's code length depends only on the cumulative teacher-student KL divergence, independent of parameter count or data entropy. The open-source implementation supports JAX-based training on GPUs and TPUs.

Code for the paper "Requential Coding: Pushing the Limits of Model Compression with Self-Generated Training Data" . Requential coding compresses a generative model by coding a training process built from self-generated data. A student model samples its own training batches, a teacher training on real data selects them via relative entropy coding, and the code length is the cumulative teacher-student KL, independent of both parameter count and data entropy. train.py : the requential training loop teacher on real data, student on teacher-sampled data, per-step MC KL, optional iso-loss projection and PAC-Bayes bound logging model.py : GPT-2-style transformer flax NNX bounds.py : PAC-Bayes bound quantities data.py : dataset loaders ckpt.py : checkpoint save/resume configs/default.yaml : all hyperparameters with the paper defaults data prep/ : dataset preparation scripts sweeps/ : one wandb sweep per paper experiment notebooks/ : one self-contained notebook per paper figure conda create -n requential python=3.10 -y conda activate requential pip install -U "jax cuda12 " flax optax hydra-core omegaconf wandb tqdm numpy pip install pandas matplotlib seaborn jupyter huggingface hub datasets data prep + figures Everything runs on GPUs and, with jax tpu , on TPUs. The batch is sharded over all visible devices, and both single- and multi-GPU setups work. Use CUDA VISIBLE DEVICES to pick GPUs. The paper's experiments ran on TPU v6e-8 hosts, and the accumulation factors in the sweeps are calibrated for that memory. The FineWeb sweeps use A=8 , and only the largest width-2752 model needs A=16 . If you run out of memory, raise the gradient-accumulation factor A the per-step microbatch is B/A or set model.gradient checkpointing=true . Note that the FineWeb loader shuffles data in microbatch-sized blocks, so changing A changes the data order. The run is statistically equivalent but not step-for-step identical. python data prep/prepare fineweb.py 201 FineWeb, GPT-2 tokens 201 chunks = 20B python data prep/prepare open.py OpenWebText, character-level vocab 96 python data prep/prepare cifar5m.py '/path/to/cifar5m part{}.npz' CIFAR-5M pixels Datasets land under datasets/<name / . The synthetic noise dataset fresh uniform-random tokens, unlearnable and repeat dataset one token repeated, trivially learnable need no preparation. A single requential run, with the teacher on real data and the student distilled on teacher-sampled data, both with EMA smoothing: CUDA VISIBLE DEVICES=0 python main.py ds path=open model.width=256 T=100 000 000 \ B=256 opt.warmup tokens=16 384 000 wandb project=requential Sampling from the teacher dominates the runtime, since every step autoregressively generates a full synthetic batch. To maximize sampling throughput, use a large batch size B and train on multiple devices. Generation is parallelized across all visible devices, with each device sampling its own shard of the batch. Key flags see configs/default.yaml for all of them : | flag | meaning | |---|---| T / tpp | real-token budget D, absolute or in tokens per parameter | model.width | model size axis depth fixed at 8 in the paper | isoloss proj | iso-loss projection: periodically reset the teacher to the student, then retrain it on real data student paused until its loss recovers | teacher ema / student ema | EMA smoothing timescales teacher smoothing off at 0 | ensemble size | E students sharing one teacher and one synthetic stream | unique tokens | cap on unique real tokens multi-epoch regime | log bound | log PAC-Bayes bound quantities req bound/ , ptq bound/ each eval | train student | false = plain LM training prequential/PTQ baseline only | ckpt dir | local directory to save/auto-resume full training state | Every run logs the per-token teacher-student kl nats , the code lengths L req requential, bits and L preq prequential, bits , and ema {student,teacher} eval loss on held-out data. With log bound=true it also logs ema {student,teacher} train loss and the bound families req bound/{loss,c,sigma,bounded loss} the student, priced with the requential code and ptq bound/{...} the teacher, priced as an idealized lossless 4-bit quantization . The bound's empirical risk and variance statistic are measured on a fixed train prefix, and the token denominator is D = unique tokens when multi-epoching. Each experiment is a wandb sweep. Launch it and point one agent per node at it: wandb sweep -p requential sweeps/scaling fineweb.yaml CUDA VISIBLE DEVICES=0,1,2,3 wandb agent <entity /requential/<sweep-id | sweep | figure | experiment | |---|---|---| scaling {open,cifar5m,fineweb}.yaml | Figs 5, 7, 12 | width sweep × iso-loss projection, bounds on | ensemble fineweb.yaml | Fig 6 | E ∈ {1,2,4,8} ensembles, shared teacher | overfit fineweb.yaml | Fig 8 | multi-epoch over 1e8 unique tokens | synthetic controls.yaml | Fig 9 | noise / repeat controls | hundredM {open,cifar5m,fineweb}.yaml | Figs 4, 9 | ~100M-param matrix | After the sweeps finish, run the matching notebook in notebooks/ . Each is self-contained, fetches your runs from wandb by sweep tag set PROJECT / ENTITY at the top , caches locally, and writes figures/ .pdf : fig5 scaling model.ipynb : larger models compress to smaller sizes fig6 ensemble.ipynb : larger ensembles are more compressible fig7 generalization bound.ipynb : PAC-Bayes bounds vs idealized PTQ also draws the Fig 12 OWT panels fig8 overfitting.ipynb : U-shaped bound under data repetition fig9 learnable info.ipynb : learnable information across datasets @article{qiu2026requential, title={Requential Coding: Pushing the Limits of Model Compression with Self-Generated Training Data}, author={Qiu, Shikai and Finzi, Marc and Zheng, Yujia and Zhang, Kun and Wilson, Andrew Gordon}, year={2026} }