{"slug": "cartesia-h-net-byte-level-model-matches-smollm3-3b-with-10x-less-compute-and", "title": "Cartesia H-Net: Byte-Level Model Matches SmolLM3-3B With 10x Less Compute and Data, Claimed", "summary": "Cartesia published two research posts on October 28, 2026 describing hierarchical networks (H-Nets) that learn from raw bytes instead of tokens, claiming an internal 2-stage H-Net matches SmolLM3-3B's final average score after about 1T tokens versus 11T, a 10x reduction in compute and data. The controlled scaling ladder is more modest: converting BPE Transformer rungs from 221M to 2.62B total parameters into 1-stage byte-level H-Nets yielded 1.28x compute efficiency on validation bits-per-byte (about 22% less compute), 1.83x on Paloma and 2.06x on OLMES, with code gains of a median 2.04x across 15 in-mix languages and 3.10x on ten unseen languages. The approach builds on the July 2025 paper by Sukjun Hwang, Brandon Wang and Albert Gu, whose MIT-licensed code is public, while Cartesia says newer architectures will be released \"over time.", "body_md": "Cartesia, best known for its voice models, has published a research series arguing that language models should learn from raw bytes instead of tokens. Two posts dated October 28, 2026, [From Tokens to Bytes I](https://www.cartesia.ai/blog/from-tokens-to-bytes-part-1?ref=explainx) and [II](https://www.cartesia.ai/blog/from-tokens-to-bytes-part-2?ref=explainx), describe a newer generation of hierarchical networks, or H-Nets, that the company says \"matches an open-source 3B Transformer with 10x less compute and data.\"\n\nThis post separates the 10x figure from the more modest numbers in the controlled experiments. Short version: the idea is credible and the authors are candid about limits, but the 10x comes from one internal model against one baseline, and the careful ladder shows gains of about 1.3x to 2x on text.\n\n## TL;DR: claim versus measurement\n\n| Claim | What Cartesia's posts show | \n|---|---|\n| \"10x less compute and data\" | An internal 2-stage H-Net reaches SmolLM3-3B's final average score after about 1T tokens versus 11T; compute saving is proportional because FLOPs per byte are matched within 2% | \n| \"3-5x efficiency\" in controlled scaling | Posts say up to 3-5x purely from learned tokenization; the measured ladder shows 1.28x on validation loss, 1.83x on Paloma and 2.06x on OLMES | \n| Better on code | 1.49x (Markdown) to 2.40x (Go) in-distribution, median 3.10x on ten unseen languages, up to 5.37x on Prolog | \n| Beyond text | Previews only: DNA (about 6x) and vision (no numbers) | \n| Release | Some newer architectures promised \"over time\"; original code is public | \n\n## What is an H-Net, and why replace the tokenizer?\n\nLanguage models do not read text directly. A tokenizer, usually Byte Pair Encoding, first merges frequent byte pairs into a fixed vocabulary (128k entries for Llama-3). On Cartesia's data mix each BPE token covers about 4.57 bytes. That step is hand-built and frozen before training, which Cartesia calls one of the last unlearned parts of the pipeline.\n\nAn H-Net moves that step inside the model. Per the post, five modules do the work: an encoder reads bytes, a learned chunking step compresses them, a main network works on the coarse chunks like tokens, the output is expanded back to byte resolution, and a decoder finishes. The main network can itself be an H-Net, so chunking nests: bytes into words, words into phrases. The original idea came from [the July 2025 paper](https://arxiv.org/abs/2507.07955?ref=explainx) by Sukjun Hwang, Brandon Wang and Albert Gu, with [code on GitHub](https://github.com/goombalab/hnet?ref=explainx) under an MIT license.\n\nWhy does this matter to practitioners? Tokenizers cause real costs: non-English text needs more tokens, as we covered in the [tokenizer tax analysis](https://explainx.ai/blog/tokenizer-tax-non-english-languages-pay-more-tokens-17x-gap-rest-of-world-2026), and per-token prices hide those differences, as in [why price per token misleads](https://explainx.ai/blog/price-per-token-misleading-tokenizer-efficiency-2026). A model that chunks adaptively could narrow those gaps.\n\n## The controlled result: 1-stage H-Net versus a BPE ladder\n\nCartesia built a scaling ladder of BPE Transformers (from 221M to 2.62B total parameters in the examples shown) trained on a mix of 70% English, 20% code, 5% math and 5% multilingual text, measured in bytes. It then converted each rung into a 1-stage byte-level H-Net, keeping the Transformer backbone and hyperparameters the same and matching the outer networks' parameters and FLOPs to the tokenization layers. A strong auxiliary loss held the compression at the BPE rate of 4.57 bytes.\n\nResults, from fitted compute-optimal scaling laws:\n\n- **Validation bits-per-byte:** 1.28x compute efficiency, meaning about 22% less compute for the same loss.\n- **Downstream suites:** about 1.83x on Paloma and 2.06x on OLMES.\n- **Code:** a median of 2.04x across 15 in-mix languages, rising to a median of 3.10x on ten languages outside the mix, including OCaml, Prolog, Lean and Fortran.\n\nCartesia also trained its own BPE tokenizer on the corpus for a fair baseline, which it says is worth about 1.1x over an off-the-shelf tokenizer. So against a standard tokenizer the H-Net advantage would be somewhat larger.\n\n## The 10x result: 2-stage H-Net versus SmolLM3-3B\n\nThe headline experiment pits a larger 2-stage H-Net against SmolLM3-3B, Hugging Face's 3B model with a documented recipe covering 11.1T training tokens. The H-Net has 3.67B parameters versus 3.08B, and FLOPs per byte are 4.44G versus 4.36G. Its main network sees about one position per 8 bytes, roughly 40% fewer than SmolLM3 on the same text, with byte-level stages built from Mamba-2 layers and sliding-window attention.\n\nPer the post, the H-Net reaches SmolLM3's final average benchmark score, measured on six tasks from SmolLM3's published suites, after about 1T tokens versus about 11T. That is roughly 11x on data, and compute scales with it because the models are matched per byte.\n\nRead this carefully:\n\n- It is a single comparison against one baseline.\n- The model is internal; Cartesia says it plans to release newer architectures \"over time.\"\n- Exact scores appear only in a figure, not in text.\n- The authors describe their ladder matching to BPE as restrictive and favorable to the baseline, and say fair matching of data and compute is unresolved, since H-Nets compress code more and so spend less compute on it.\n\nFor perspective on similar claims, see our checks of [Microsoft's MAI-Base-2 efficiency claim](https://explainx.ai/blog/microsoft-ai-mai-base-2-2-8t-moe-pre-training-4-12x-efficiency-claim-2026) and [Q Labs' 4000x claim](https://explainx.ai/blog/q-labs-dust-noise-pretrain-transformers-without-backpropagation-4000x-claim-verified-2026): headline multipliers usually depend on a specific baseline and metric.\n\n## Part II: why bytes may \"understand\" data better\n\nThe second post asks why H-Nets help. It compares five setups: a byte Transformer, BPE without pretokenization, BPE with pretokenization, an H-Net and a BLT-like model, using two kinds of scaling law: compute-to-loss and loss-to-loss, which measures how pretraining loss transfers to downstream tasks.\n\nFindings reported by Cartesia:\n\n1. **Compression buys efficiency.** BPE without pretokenization is nearly 5x more compute-efficient than a byte model at reaching the same validation loss (about 6.03x compression), and BPE with pretokenization reaches about 7.2x.\n2. **Compression costs transfer.** At equal validation loss, the byte model does better downstream. BPE with pretokenization lags the byte model by about 1.5x in compute terms on loss-to-loss, and BPE without pretokenization by over 1.9x.\n3. **H-Nets get both.** They match the byte model's loss-to-loss transfer while keeping compression gains; versus BPE the loss-to-loss gap is 1.61x.\n4. **A hidden prior.** Pretokenization regexes (GPT-2, cl100k, o200k, DeepSeek-V3) are hand-engineered, which cuts against the \"bitter lesson\" philosophy the authors invoke.\n5. **A known flaw.** BPE without pretokenization suffers from the prompt boundary problem, where a prompt ends mid-token; the evaluation needed a correction to be sensible.\n\nThe authors argue that tokenization is about finding semantics, not only compression: BPE without pretokenization compresses about 32% more than BPE with it yet performs worse.\n\n## What H-Nets learn: in-context tokenization\n\nThe posts show behaviors absent from tokenized models. New words are first split into several chunks and then grouped into fewer as they repeat; repeated formats form very large chunks; and recurring motifs and digit blocks are picked out as units.\n\n## Beyond text: DNA and images\n\nCartesia previews two other domains. On prokaryote genomes (about 274B nucleotides) nucleotide-level H-Nets with 6x target compression show compute gains of 5.95x on validation and up to 7.06x on a test set, with chunks that align to codons. For vision, a 2D H-Net replaces the backbone of a pixel-space flow-matching baseline and learns chunks without supervision, but no numbers are given. Treat both as early signals. Our post on a [pixel-space image model with no VAE](https://explainx.ai/blog/sperid-iris-3b-pixel-space-image-model-no-vae-2026) shows the same appetite for dropping hand-built encoders.\n\n## What this means for builders\n\n- **Nothing to deploy yet.** No new checkpoint is announced; the existing public code and checkpoints are from the 2025 version.\n- **Watch code and low-resource languages.** Those are where the reported gains are largest, and where tokenizers hurt most.\n- **Expect inference trade-offs.** Byte-level models process longer sequences; the design relies on compressing inside the model, and Cartesia reports the outer stages take about 43% of H-Net FLOPs versus 8% for embedding and head in the baseline.\n- **Compare with other byte-model work.** See our[fact check of byte-model distillation](https://explainx.ai/blog/breaking-token-ceiling-byte-model-distillation-fact-check-2026) and[Hugging Face Tokenizers v1](https://explainx.ai/blog/huggingface-tokenizers-v1-rc-30x-faster-2026) for the token-based status quo.\n\nCartesia's day job is voice: its [Sonic-3.6 text-to-speech results](https://explainx.ai/blog/cartesia-sonic-3-6-tts-artificial-analysis-august-2026) are a separate line of work, and these posts come from its research team.\n\n## Open questions\n\n- Will Cartesia release the newer 2-stage model or weights, and under what license?\n- Do the gains hold at 30B+ parameters and with reinforcement learning post-training?\n- How does serving cost compare once longer byte sequences and the outer-network FLOPs are counted?\n- Can independent groups reproduce the 11T-to-1T result?\n\n## How to read a claim like this\n\nEfficiency multipliers in pretraining research are only as good as the baseline and the metric behind them. Here, three choices shape the result: which tokenizer the baseline uses, whether compute is counted per byte or per token, and which benchmarks are averaged. A reader who wants to judge the 10x figure should check each of those three, then compare it with the smaller, better-controlled 1.28x and roughly 2x numbers from the scaling ladder. The gap between the headline and the ladder is not a contradiction, because the two experiments answer different questions, but only the ladder isolates the effect of learned chunking on its own.\n\nWe will add an update when code or weights appear.\n\n## Related reading\n\n*Accurate as of October 28, 2026, based on Cartesia's posts. Figures are the authors' own and unreplicated.*", "url": "https://wpnews.pro/news/cartesia-h-net-byte-level-model-matches-smollm3-3b-with-10x-less-compute-and", "canonical_source": "https://explainx.ai/blog/cartesia-h-net-bytes-10x-less-compute-smollm3-3b-claimed-vs-verified-2026?utm_source=rss&utm_medium=feed&utm_campaign=blog", "published_at": "2026-10-28 00:00:00+00:00", "updated_at": "2026-10-28 22:12:26.228375+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "large-language-models", "ai-research"], "entities": ["Cartesia", "H-Net", "SmolLM3-3B", "Sukjun Hwang", "Brandon Wang", "Albert Gu", "Paloma", "OLMES"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/cartesia-h-net-byte-level-model-matches-smollm3-3b-with-10x-less-compute-and", "markdown": "https://wpnews.pro/news/cartesia-h-net-byte-level-model-matches-smollm3-3b-with-10x-less-compute-and.md", "text": "https://wpnews.pro/news/cartesia-h-net-byte-level-model-matches-smollm3-3b-with-10x-less-compute-and.txt", "jsonld": "https://wpnews.pro/news/cartesia-h-net-byte-level-model-matches-smollm3-3b-with-10x-less-compute-and.jsonld"}}