Learning to Compress a Billion Bytes of Wikipedia Tufalabs' tufazip took #2 on Matt Mahoney's Large Text Compression Benchmark as of its Sept. 7, 2026 submission, compressing one billion bytes of Wikipedia into 105,388,353 bytes (105.39 MB) including offline trained weights and decoder. The system uses a 12-layer, 163M-parameter Transformer initialized from a 34M-parameter offline model quantized to 4.33 MB, with two offline trained layers copied into twelve independent layers that keep learning during compression so the decoder repeats the same process and the full model's weights never need to be transferred. The entry is the best-performing on the 20+ year-old benchmark to use a neural network as its sole prediction model. ← Research https://tufalabs.ai/research/ tufazip: Learning to Compress a Billion Bytes of Wikipedia Summary I took 2 on the Large Text Compression Benchmark , Large Text Compression Benchmark https://www.mattmahoney.net/dc/text.html Matt Mahoney . at the time of submission: Sept. 7, 2026 a benchmark that has been running for 20+ years, with tufazip https://github.com/Tufalabs/tufazip , the best-performing entry using a neural network as its sole prediction model. My final result compressed one billion bytes of Wikipedia into 105,388,353 bytes 105.39 MB , including the offline trained weights and decoder. The final system uses a 12-layer, 163M-parameter Transformer, initialized from a 34M-parameter offline model quantized to 4.33 MB. Two offline trained layers are copied into twelve independent layers, which continue learning as the file is compressed. The decoder repeats the same learning process, so the full model’s learned weights never need to be transferred. Introduction What is intelligence? Broadly speaking, intelligence is typically linked to capability of abstraction, logic, reasoning, problem solving, and adaptability; in daily use, we often use it to just mean crystallized intelligence. But it’s not so clear…it’s an age-old question that remains difficult to answer. Let’s look at a few concrete perspectives focused on machine intelligence: - Achieving goals: Shane Legg and Marcus Hutter define intelligence as “an agent’s ability to achieve goals in a wide range of environments.” Universal Intelligence: A Definition of Machine Intelligence https://arxiv.org/abs/0712.3329 Legg and Hutter, 2007 . Here, Legg and Hutter emphasize the breadth of environments the agent can handle and its ability to accomplish tasks within them. - Learning new skills efficiently: François Chollet defines intelligence as skill-acquisition efficiency: how effectively a system turns its prior knowledge and experience into new abilities. On the Measure of Intelligence https://arxiv.org/abs/1911.01547 Chollet, 2019 . Think of picking up an unfamiliar game from a few examples. Learnability is still important but with an additional focus on efficiency. - Finding useful representations: One perspective inspired by Kenneth Stanley’s work is that intelligence depends on finding abstractions that make learning and problem solving easier. Think of discovering a reusable rule that connects several seemingly unrelated problems. Interestingly, Stanley and colleagues argue here that how knowledge is organized inside a system, despite being much more ambiguous to measure, may matter for its ability to generalize and keep learning. Questioning Representational Optimism in Deep Learning: The Fractured Entangled Representation Hypothesis https://arxiv.org/abs/2505.11581 Kumar, Clune, Lehman, and Stanley, 2025 . - Compressing information: Matt Mahoney argues for understanding intelligence through compression: the ability to capture the structure in observations so they can be described more compactly. For text, learning spelling, grammar, and relationships makes the next word more predictable—and predictable information takes fewer bits to encode. From this perspective, compression gives us a concrete way to measure how much structure a system has captured. Rationale for a Large Text Compression Benchmark https://mattmahoney.net/dc/rationale.html Mahoney, 2009 , section “Compression is Equivalent to General Intelligence.” Inspired by the compression perspective, we focus on lossless compression as our task. For text, the connection between prediction and compression is especially direct. If a model assigns probability $p$ to the next token, its ideal encoding cost is $-\log 2 p$ bits. Doubling the probability assigned to the token that actually occurs saves one bit. For a fixed tokenization, next-token prediction and compression have the same objective : the summed cross-entropy loss in bits is exactly the ideal compressed length. Here, $x t$ is the token at position $t$ and $x {