{"slug": "requential-coding-1-bit-compression-with-better-generalization", "title": "Requential Coding <1 bit compression with better generalization", "summary": "Researchers introduced Requential Coding, a method that compresses generative models by coding a training process using self-generated data, achieving under 1 bit per parameter compression with better generalization. The technique's code length depends only on the cumulative teacher-student KL divergence, independent of parameter count or data entropy. The open-source implementation supports JAX-based training on GPUs and TPUs.", "body_md": "Code for the paper **\"Requential Coding: Pushing the Limits of Model Compression with Self-Generated Training Data\"**.\n\nRequential coding compresses a generative model by coding a training process built from self-generated data. A student model samples its own training batches, a teacher training on real data selects them via relative entropy coding, and the code length is the cumulative teacher-student KL, independent of both parameter count and data entropy.\n\n`train.py`\n\n: the requential training loop (teacher on real data, student on teacher-sampled data, per-step MC KL, optional iso-loss projection and PAC-Bayes bound logging)`model.py`\n\n: GPT-2-style transformer (flax NNX)`bounds.py`\n\n: PAC-Bayes bound quantities`data.py`\n\n: dataset loaders`ckpt.py`\n\n: checkpoint save/resume`configs/default.yaml`\n\n: all hyperparameters with the paper defaults`data_prep/`\n\n: dataset preparation scripts`sweeps/`\n\n: one wandb sweep per paper experiment`notebooks/`\n\n: one self-contained notebook per paper figure\n\n```\nconda create -n requential python=3.10 -y\nconda activate requential\npip install -U \"jax[cuda12]\" flax optax hydra-core omegaconf wandb tqdm numpy\npip install pandas matplotlib seaborn jupyter huggingface_hub datasets  # data prep + figures\n```\n\nEverything runs on GPUs and, with `jax[tpu]`\n\n, on TPUs. The batch is sharded\nover all visible devices, and both single- and multi-GPU setups work. Use\n`CUDA_VISIBLE_DEVICES`\n\nto pick GPUs. The paper's experiments ran on TPU v6e-8\nhosts, and the accumulation factors in the sweeps are calibrated for that\nmemory. The FineWeb sweeps use `A=8`\n\n, and only the largest width-2752 model\nneeds `A=16`\n\n. If you run out of memory, raise the gradient-accumulation\nfactor `A`\n\n(the per-step microbatch is `B/A`\n\n) or set\n`model.gradient_checkpointing=true`\n\n. Note that the FineWeb loader shuffles\ndata in microbatch-sized blocks, so changing `A`\n\nchanges the data order. The\nrun is statistically equivalent but not step-for-step identical.\n\n```\npython data_prep/prepare_fineweb.py 201        # FineWeb, GPT-2 tokens (201 chunks = 20B)\npython data_prep/prepare_open.py               # OpenWebText, character-level (vocab 96)\npython data_prep/prepare_cifar5m.py '/path/to/cifar5m_part{}.npz'  # CIFAR-5M pixels\n```\n\nDatasets land under `datasets/<name>/`\n\n. The synthetic `noise`\n\ndataset (fresh\nuniform-random tokens, unlearnable) and `repeat`\n\ndataset (one token repeated,\ntrivially learnable) need no preparation.\n\nA single requential run, with the teacher on real data and the student distilled on teacher-sampled data, both with EMA smoothing:\n\n```\nCUDA_VISIBLE_DEVICES=0 python main.py ds_path=open model.width=256 T=100_000_000 \\\n    B=256 opt.warmup_tokens=16_384_000 wandb_project=requential\n```\n\nSampling from the teacher dominates the runtime, since every step\nautoregressively generates a full synthetic batch. To maximize sampling\nthroughput, use a large batch size `B`\n\nand train on multiple devices.\nGeneration is parallelized across all visible devices, with each device\nsampling its own shard of the batch.\n\nKey flags (see `configs/default.yaml`\n\nfor all of them):\n\n| flag | meaning |\n|---|---|\n`T` / `tpp` |\nreal-token budget D, absolute or in tokens per parameter |\n`model.width` |\nmodel size axis (depth fixed at 8 in the paper) |\n`isoloss_proj` |\niso-loss projection: periodically reset the teacher to the student, then retrain it on real data (student paused) until its loss recovers |\n`teacher_ema` / `student_ema` |\nEMA smoothing timescales (teacher smoothing off at 0) |\n`ensemble_size` |\nE students sharing one teacher and one synthetic stream |\n`unique_tokens` |\ncap on unique real tokens (multi-epoch regime) |\n`log_bound` |\nlog PAC-Bayes bound quantities (`req_bound/*` , `ptq_bound/*` ) each eval |\n`train_student` |\n`false` = plain LM training (prequential/PTQ baseline only) |\n`ckpt_dir` |\nlocal directory to save/auto-resume full training state |\n\nEvery run logs the per-token teacher-student `kl`\n\n(nats), the code lengths\n`L_req`\n\n(requential, bits) and `L_preq`\n\n(prequential, bits), and\n`ema_{student,teacher}_eval_loss`\n\non held-out data. With `log_bound=true`\n\nit\nalso logs `ema_{student,teacher}_train_loss`\n\nand the bound families\n`req_bound/{loss,c,sigma,bounded_loss}`\n\n(the student, priced with the\nrequential code) and `ptq_bound/{...}`\n\n(the teacher, priced as an idealized\nlossless 4-bit quantization). The bound's empirical risk and variance\nstatistic are measured on a fixed train prefix, and the token denominator is\nD = `unique_tokens`\n\nwhen multi-epoching.\n\nEach experiment is a wandb sweep. Launch it and point one agent per node at it:\n\n```\nwandb sweep -p requential sweeps/scaling_fineweb.yaml\nCUDA_VISIBLE_DEVICES=0,1,2,3 wandb agent <entity>/requential/<sweep-id>\n```\n\n| sweep | figure | experiment |\n|---|---|---|\n`scaling_{open,cifar5m,fineweb}.yaml` |\nFigs 5, 7, 12 | width sweep × iso-loss projection, bounds on |\n`ensemble_fineweb.yaml` |\nFig 6 | E ∈ {1,2,4,8} ensembles, shared teacher |\n`overfit_fineweb.yaml` |\nFig 8 | multi-epoch over 1e8 unique tokens |\n`synthetic_controls.yaml` |\nFig 9 | noise / repeat controls |\n`hundredM_{open,cifar5m,fineweb}.yaml` |\nFigs 4, 9 | ~100M-param matrix |\n\nAfter the sweeps finish, run the matching notebook in `notebooks/`\n\n. Each is\nself-contained, fetches your runs from wandb by sweep tag (set `PROJECT`\n\n/\n`ENTITY`\n\nat the top), caches locally, and writes `figures/*.pdf`\n\n:\n\n`fig5_scaling_model.ipynb`\n\n: larger models compress to smaller sizes`fig6_ensemble.ipynb`\n\n: larger ensembles are more compressible`fig7_generalization_bound.ipynb`\n\n: PAC-Bayes bounds vs idealized PTQ (also draws the Fig 12 OWT panels)`fig8_overfitting.ipynb`\n\n: U-shaped bound under data repetition`fig9_learnable_info.ipynb`\n\n: learnable information across datasets\n\n```\n@article{qiu2026requential,\n  title={Requential Coding: Pushing the Limits of Model Compression with Self-Generated Training Data},\n  author={Qiu, Shikai and Finzi, Marc and Zheng, Yujia and Zhang, Kun and Wilson, Andrew Gordon},\n  year={2026}\n}\n```\n\n", "url": "https://wpnews.pro/news/requential-coding-1-bit-compression-with-better-generalization", "canonical_source": "https://github.com/shikaiqiu/requential-coding", "published_at": "2026-07-17 13:01:39+00:00", "updated_at": "2026-07-17 13:21:19.107168+00:00", "lang": "en", "topics": ["machine-learning", "artificial-intelligence", "generative-ai", "ai-research"], "entities": ["Requential Coding", "JAX", "Flax", "GPT-2", "FineWeb", "OpenWebText", "CIFAR-5M", "TPU v6e-8"], "alternates": {"html": "https://wpnews.pro/news/requential-coding-1-bit-compression-with-better-generalization", "markdown": "https://wpnews.pro/news/requential-coding-1-bit-compression-with-better-generalization.md", "text": "https://wpnews.pro/news/requential-coding-1-bit-compression-with-better-generalization.txt", "jsonld": "https://wpnews.pro/news/requential-coding-1-bit-compression-with-better-generalization.jsonld"}}