# Entropy Is Spare Capacity > Source: > Published: 2026-09-21 22:36:47+00:00 What if you want to exchange a secret in the open? You might use [steganography](https://en.wikipedia.org/wiki/Steganography) to exchange a note that is hiding in plain sight. [LLM watermarks](https://www.anthropic.com/news/claude-text-watermark) do exactly that - the text looks plausible, but it carries information alongside your text. ## LLMs are *just* next token predictors LLMs take a stream of text, break them into [tokens](https://fffej.substack.com/p/whats-a-token), turn it into [maths](https://fffej.substack.com/p/playing-with-transformers) and give you the next one based on a probability distribution. For example, if you give the text “the quick brown fox jumps over the lazy” the next completion is highly likely to be “dog” (certainly that’s more likely than asparagus). Given this phrase we’ve got low entropy (e.g. there isn’t much signal as there’s only one choice). But sometimes there is choice, and within that choice we can encode information. This is how [Claude’s text watermarks](https://www.anthropic.com/news/claude-text-watermark) work. Look for an opportunity to replace one word with another where the probability means that word is likely enough to not notice the difference. Let’s try and do the same to make an LLM (well at least an SLM) encode an arbitrary binary as text. ## The plan We’ll ask the LLM to write us a story (doesn’t really matter what the prompt is here, just something that spews text). At each text prefix, the encoder asks the LLM for the 20 most likely tokens and partitions them into approximately equal groups. - If there’s a 50/50 chance, then we can encode 1 bit of information. - If there’s a 33/33/33 chance, then we can encode 1.585 bits ( `log2(3)` ). - If there’s a 25/25/25/25 chance, then we can encode 2 bits and so on. If the model only strongly prefers one continuation, then our encoder emits the best usable token without carrying data. To avoid getting stuck, after N skips we permit an unbalanced choice (if I don’t do this, I get repeating data). Decoding is similar, we can inspect those 20 most likely tokens and apply the same steps in reverse. ## Building it This was a good chance to put GPT6-Astra to the test. I just used the low thinking effort, gave this prompt with more of less the text above, and it just worked. The nice thing about this particular problem is that the model has an oracle (encode/decode should produce the original). I pondered whether I should share the code, but I think that’s the least interesting part these days, but shout if it’s interesting. As of the morning I’m writing this, my current vibes-based leader board of “models that solve my problems with the least effort on my part” is Astra > Fable > Sol > Opus. ## Does it work? Sure does! I can now roundtrip an arbitrary file into a story. Here’s an example. If I have a file with this context ``` alan ate an asparagus. ``` Then I can turn that into a series of bits and feed it into the encoder, primed with “a seaside village at dawn”. We generate text one token at a time, find opportunities to encode bits. When we run out of response, we set a new scene (”a hillside town in autumn”, “a garden behind a library”) and off it goes again. This explains the rather unhinged set of text below! ``` The salt air hit Silas’s face like a cool bruise, stinging his cheeks while the first slivers of dawn dripped orange across the slate gray stone path. A thick fog had swallowed the far edge of the harbor, leaving only a skeletal outline of fishing boats bobbing with ghostly grace. Mist clung to the pilings of the aging lighthouse, turning them into twisted fingers reaching towards the unseen horizon. Rain, a persistent whisper against the corrugated iron roof of the inn, had settled, lending the cobblestones a sullen, almost melancholic hue. The sun, a sticky orange smear against the dust, beat down on the cobblestones of the Marrakech market. The air hung thick with a thousand scents – sweet dates and roasting coriander, wet dust mingling with jasmine and something distinctly earthy, like turned earth and old stone. A chaotic tapestry of color and noise enveloped Karim, swallowed by the relentless press of merchants hawking saffron, leather goods, and intricately embroidered cushions. A donkey laden with pottery bumped against him, momentarily disrupting his careful pace, forcing him to weave through a whirlwind of silver bracelets, sapphire jewelry, and stacks of vibrant, hand-painted pottery. The rain had stopped only an hour ago, leaving behind a thick, cloying perfume of ozone and something almost earthy, like wet stone. Sunlight pierced the canopy through fissures in the leaves, illuminating swirling motes like fallen stars. Beneath a canopy so old the moss clung like emerald armor, Elias knelt, tracing the outline of a weathered, gnarled root with the tip of his boot. The ground, now softly draped in a tapestry of vibrant mushrooms and fallen ferns, swallowed sound; the forest held its breath. The light faded, not with a bloom of a sunset, but with a slow bleed, staining the grey stone of the abandoned station building in shades of bruised plum and faded indigo. The rusted doors, thick as a shield, swung open with a groan that scraped across the cracked pavement like a restless ghost, inviting me into a silent cavern. Cobwebs, spun with silver droplets, clung to the decaying sign that declared “Willow Creek Railway – For Sale,” its once vibrant lettering blurred and weeping. The air, surprisingly, smelled of burnt sugar and damp stone, clinging to the narrow streets of Blackwood, a tiny hillside town clinging stubbornly to the face of the Dragon's Tooth Mountains. Ruska, laden with a battered canvas filled with sketches, navigated the uneven cobblestones, the stones slick with a perpetual layer of autumn’s decay. ``` And yes, it does round trip backwards! Is it useful? Absolutely not. But if you ever want to send a secret message to someone in the form of a story, it’s quite fun!