{"slug": "your-models-agreed-with-each-other-they-were-agreeing-with-themselves", "title": "Your models agreed with each other. They were agreeing with themselves.", "summary": "An engineer's experiment testing whether LLM readers recover meaning from word-length-encoded messages found that agreement among independent readings varied wildly: one message showed agreement four times above chance, while a second reversed the effect. The project identified two baseline failure modes that can corrupt self-consistency, majority-vote ensembles, and LLM-as-judge pipelines, including a control arm whose random texts became near-duplicates, inflating agreement metrics above the model readings they were meant to benchmark.", "body_md": "There is a small art project in our house that encodes a sentence as nothing but its word\n\nlengths. Each word becomes a run of some symbol, repeated once per letter; the symbol itself is\n\nchosen at random and carries nothing. \"The night is long\" becomes four clusters of length 3, 5, 2,\n\nThe project's README makes a claim I liked: that LLM readers *never recover the intended meaning*,\n\nthat they generate from structure and bias, and that every reading is a projection. It is the most\n\ninteresting sentence in the repository and nobody had ever checked it.\n\nSo we checked it. And the first result looked like the README was wrong: independent readings of\n\nthe same encoded message agreed with each other **four times above chance**, with non-overlapping\n\nconfidence intervals. Something was clearly getting through.\n\nThen we ran a second message, and the effect **reversed**.\n\nThis post is about why that reversal is the most useful thing in the experiment, and about the two\n\ndifferent ways a baseline can lie to you when you are measuring whether models agree. If you run\n\nself-consistency, majority-vote ensembles, LLM-as-judge panels, or any \"ask it five times and see\n\nif it converges\" pipeline, both of those failure modes are already in your numbers.\n\nEncode a known sentence. Take N *independent* readings — separate processes, no shared context,\n\nbecause one sampled list of five guesses is one reading, not five. Then measure how much the\n\nreadings agree **with each other**.\n\nThe obvious way to do that is to compare the agreement against random chance, and that is where it\n\ngoes wrong. The design that survived contact needs three arms, not two:\n\n| arm | what it is | what it isolates |\n|---|---|---|\nA treatment |\nN readings of the true message's length sequence | channel + prior |\nB prior control |\nN readings of a different length sequence — same word count, lengths resampled from the same distribution |\nthe prior and the task framing alone, with no particular message behind it |\nC random basis |\nN texts assembled with no model at all: a random word of the right length at each slot |\nthe chance floor |\n\nEvery text within an arm shares a length profile, so *positional* agreement is well defined: do two\n\nindependent readings put the same word in slot 4? That is the metric that carries the argument\n\nbelow. Jaccard over word bags and cosine over sentence embeddings were computed too, and I will\n\ncome back to why the embedding metric turned out to be worthless here.\n\nThe first version of arm C built its random texts from a vocabulary pooled out of arm B's own\n\nreadings. This is a very natural thing to do — you want the \"random\" texts to be made of words the\n\nmodel would actually use, so you harvest them from the model's own output. Otherwise you are\n\ncomparing model English against dictionary English and the gap is meaningless.\n\nWith seven valid control readings, the pooled vocabulary was so small that the random texts came\n\nout as near-duplicates of each other. The floor rose to **cosine 0.391 — above the model readings\nit was supposed to sit under, at 0.232.**\n\nThe comment that now sits in the code is the best artifact in the whole project, and I am quoting\n\nit verbatim rather than paraphrasing it:\n\nPooling the basis out of the control arm's few readings — the first version of this — is not a\n\nfloor. Seven texts yield a vocabulary so small that the random texts built from it are\n\nnear-duplicates of each other, which inflates every agreement metric and inflates the SEMANTIC\n\none worst (measured: cosine 0.391 for a basis pooled from 7 texts, ABOVE the model readings it\n\nwas supposed to sit under).A floor that rises with how little you sampled it is not a floor.\n\nNote what that failure would have done if it had gone the other way. A floor built from *plenty* of\n\nsamples is fine. A floor built from a handful is inflated, and an inflated floor makes a real effect\n\ndisappear. You would have concluded \"no signal\" and shipped that, and the number would have looked\n\ncompletely reasonable — a baseline at 0.391, a treatment at 0.232, no effect, move on. Nothing in\n\nthe output says \"this floor was estimated from seven things.\"\n\nThe fix is to draw the alphabet from a request that is independent of any message — \"list 240\n\ncommon English words, mix lengths 1 to 12\" — so it is still the model's own vocabulary but it\n\ncannot inherit the convergence it is meant to measure. The run prints which source it used, and\n\nsays so in the saved artifact when it has to fall back.\n\nHere is the first message's result, positional agreement, bootstrap CIs:\n\n```\nmessage: \"The night is long and the city keeps its silence\"\n\nA treatment      0.175   CI [0.153, 0.198]\nB prior-control  0.129   CI [0.100, 0.162]\nC random basis   0.044   CI [0.035, 0.053]\n```\n\nTreatment is four times the chance floor and the intervals do not touch. If you had built this\n\nwith two arms — treatment against random — you would stop here, write \"independent readers converge\n\nfar above chance on the encoded message,\" and you would have a real, reproducible, correctly\n\ncomputed number that means nothing like what you think it means.\n\nBecause arm B is *also* miles above the floor, and arm B is reading a **different message**.\n\nFifteen valid readings in arm A there, twelve in B, twenty in the model-free arm C; 105, 66 and\n\n190 pairs. Then the second run:\n\n```\nmessage: \"Rain fell across the empty market and nobody counted the hours\"\n\nA treatment      0.115   CI [0.099, 0.132]\nB prior-control  0.160   CI [0.140, 0.181]\nC random basis   0.047   CI [0.039, 0.056]\n```\n\nTwenty valid readings in A, fifteen in B, twenty in C. The prior control is now **above** the\n\ntreatment, and their intervals do not overlap in that\n\ndirection either (A tops out at 0.132, B starts at 0.140). Readings of a length sequence that was\n\nnever anybody's message agree with each other *more* than readings of the real one.\n\nBoth arms are far above the chance floor in both runs. The ordering between them flips. That is\n\nwhat \"the convergence is the prior\" looks like when you finally have an arm that can show it: the\n\ndistance from random is large and stable, the distance from *each other* is noise, and no amount of\n\nprecision on the random floor would ever have told you.\n\nThe general form, and it is not about this art project at all:\n\n**Random noise is not what your model's agreement is competing against. Its own prior is.** A\n\nbaseline made of noise answers \"is the model doing something other than nothing,\" which is almost\n\nnever the question. The question is \"is the model doing something other than what it would have\n\ndone anyway,\" and only a control arm that is a real run on a real *different* input can answer it.\n\nIf you evaluate self-consistency, this is the arm you are missing. Sampling the same prompt five\n\ntimes and finding 80% agreement is not evidence the model knows the answer until you know what five\n\nsamples of a *neighbouring* prompt agree at. Very often it is 75%.\n\nThe saved artifact keeps agreement per slot, and this is where the whole thing becomes legible.\n\nFirst message, agreement at each position, with the position's word length:\n\n```\nslot                 0     1     2     3     4     5     6     7     8     9\n\nlengths (A and C)    3     5     2     4     3     3     4     5     3     7\nA treatment         .63   .23   .36   .06   .17   .03   .07   .00   .10   .11\nC random basis      .03   .05   .10   .02   .03   .02   .03   .06   .03   .10\n\nlengths (B)          3     4     5     5     3     3     7     3     3     3\nB prior control     .68   .05   .09   .09   .14   .02   .03   .17   .02   .02\n```\n\nArm B is listed separately because it is reading its own resampled length profile — that is the\n\nwhole point of it — and only slot 0 happens to be three letters in both.\n\nSlot 0 is a three-letter word. The readings agree there 63% of the time. The prior-control arm — a\n\ndifferent message — agrees there **68%** of the time, *more* than the treatment arm. Both are\n\nproducing \"The\" and \"All\". The random basis, same length, same alphabet, no model, agrees 3%.\n\nAnd it is not even a *length* effect. Slots 4, 5 and 8 are all three-letter words too, and they\n\ncome in at 17%, 3% and 10%. What the models agree on is that English sentences start with \"The\".\n\nThat is the entire signal, sitting in position zero, present just as strongly when there is no\n\nmessage behind the lengths at all.\n\nAggregate it and the shape holds: words of three letters or fewer agree at 0.257 in treatment and\n\n0.172 in the prior control; words of five letters or more — the only ones that could carry any\n\ncontent — agree at 0.114 in treatment and 0.071 in the control, against a 0.067 floor. On the\n\ncontent words, the treatment arm is a hair above chance and the control arm is sitting on it.\n\nThe convergence is English's function-word skeleton. It was never the channel.\n\nAn arm can converge beautifully and be uniformly wrong, so recovery gets its own measurement: each\n\nreading is compared to the true original, and to eight **decoys carrying the same length profile**.\n\nA reading no closer to the truth than to a matched decoy has recovered nothing, however convergent\n\nits arm is.\n\n```\n                    vs TRUE original      vs matched DECOY\nmessage 1                  0.210                 0.216\nmessage 2                  0.138                 0.168\n```\n\nCosine similarity, so higher is closer. In both runs the readings are, if anything, slightly closer\n\nto a random decoy than to the sentence that was actually encoded. The README's claim survives its\n\nfirst contact with a measurement: readers do not recover the message. They converge hard, and they\n\nare all equally wrong.\n\nTwo honest notes on those numbers. The intervals overlap in both runs — heavily in message 1, and\n\nstill overlapping in message 2 ([0.111, 0.163] against [0.146, 0.190]) — so the right reading is\n\n\"indistinguishable,\" not \"decoys win.\" And the embedding metric barely\n\nmoved across every arm in either experiment. Run one: C 0.239, A 0.252, B 0.268. Run two: C 0.286,\n\nA 0.293, B 0.273 — where the **model-free** arm outscores the prior control. A spread of three\n\nhundredths across arms that the positional metric separates by a factor of four, and the arm with\n\nno model in it landing in the middle, is what a metric with no discriminative power looks like. A sentence-embedding\n\nmodel asked to compare ten words of grammatical nonsense has nothing to grip.\n\nThe positional metric carries the argument; the semantic one should not be quoted on its own.\n\nI did not run these; I read the saved JSON to decide whether it was worth writing up. And the two\n\nresult files were **not written by the same instrument**, which nothing in either file says.\n\nThe first file carries a per-slot breakdown. The second does not — and the second is the *newer*\n\nfile on disk, written under four minutes after the commit that added that breakdown. Python reads its\n\nsource once, at start. The second run was already in flight when the harness was edited, so it ran\n\nthe old code to completion and wrote pre-edit output with a post-edit timestamp. The only evidence\n\nis a **key missing from one file**, which is the weakest possible signal and looks exactly like a\n\nrun that had nothing to report.\n\nThat matters because the second run is the one that reverses the headline. I am confident the arm\n\narithmetic was unchanged — the edit added an output block — but I cannot *prove* it from the\n\nartifact, and the fix is one line: stamp the commit hash, a dirty bit and the script's own hash\n\ninto every result file at write time. A result file that cannot name the instrument that produced\n\nit is a measurement you have to take on trust, and the whole point of writing the numbers to disk\n\nwas not having to.\n\nTreat the reversal as a strong signal rather than a settled fact for that reason. It is two\n\nmessages, one reader pool, one embedding model.\n\nThe encoding project is [genaforvena/hidden_language_of_silence](https://github.com/genaforvena/hidden_language_of_silence);\n\nits README is where the claim being tested comes from. The measurement harness is committed but\n\n**not yet pushed to that repo**, so the numbers above are, for now, the artifact — which is also why\n\nevery arm size, interval and per-slot figure is printed here rather than linked. The readers were a\n\nsmall hosted-inference pool, the embeddings were `all-minilm`\n\nrunning locally, and every individual\n\nreading is written into the JSON alongside the aggregates so the whole thing can be recomputed\n\nwithout spending another token on the models.\n\nIf you want the shortest possible version to take into your own eval harness: **add the arm that\nruns your pipeline on a different input, and see how much of your agreement survives it.**", "url": "https://wpnews.pro/news/your-models-agreed-with-each-other-they-were-agreeing-with-themselves", "canonical_source": "https://dev.to/ilya_mozerov_867dbdd91feb/your-models-agreed-with-each-other-they-were-agreeing-with-themselves-3jb0", "published_at": "2026-08-30 03:54:32+00:00", "updated_at": "2026-08-30 04:22:07.675607+00:00", "lang": "en", "topics": ["large-language-models", "ai-research", "ai-tools"], "entities": [], "alternates": {"html": "https://wpnews.pro/news/your-models-agreed-with-each-other-they-were-agreeing-with-themselves", "markdown": "https://wpnews.pro/news/your-models-agreed-with-each-other-they-were-agreeing-with-themselves.md", "text": "https://wpnews.pro/news/your-models-agreed-with-each-other-they-were-agreeing-with-themselves.txt", "jsonld": "https://wpnews.pro/news/your-models-agreed-with-each-other-they-were-agreeing-with-themselves.jsonld"}}