{"slug": "zero-weights-deterministic-graph-language-model-mse-glm", "title": "Zero Weights Deterministic Graph Language Model (MSE-GLM)", "summary": "Researchers introduced MSE-GLM, a language model that operates without neural weights, embeddings, or probability distributions, relying instead on deterministic matrix structures built from training text. The model uses a custom BPE tokenizer and sparse matrices to enable fully traceable text generation and inference.", "body_md": "Most language models today are a pile of weights nobody can fully read.\nMSE-GLM (**Matrix-Structured Edge — Graph Language Model**)\nis an attempt at the opposite: a language model with **no neural\nweights, no embeddings, and no probability distributions**. Every\ndecision it makes during generation can be traced back to a specific row\nin a specific matrix, built from the training text itself. Nothing is\nlearned in the gradient-descent sense — everything is *counted,\ndeduplicated, and indexed*.\n\nThis post walks through every stage of the pipeline in the order it\nactually runs — tokenizing, building the graph, generating text,\ninferring beyond what was literally seen, and finally, a newer piece:\nfiguring out what a cluster of interchangeable tokens actually\n*means*.\n\n`tokenizer.py`\n\n, `graph.py`\n\n,\n`train.py`\n\n, `inference.py`\n\n, `experience.py`\n\n,\n`analyse.py`\n\n, `interpret.py`\n\n, `model.py`\n\n`tokenizer.py`\n\n)\nEverything starts with a from-scratch Byte Pair Encoding (BPE) tokenizer.\nNo external tokenizer library, no pretrained vocabulary. Training text is\nlowercased, stripped of anything that isn't a letter, digit, or space, and\nsplit into sentences on `. ! ?`\n\nand newlines.\n\nFour reserved ids anchor the vocabulary before any merges happen:\n\n```\n<PAD> = 0   reserved\n<UNK> = 1   unknown character fallback\n<BOS> = 2   prepended to every encoded sequence\n<EOS> = 3   appended only during training\n```\n\nFrom there it's classic BPE: every character in the corpus becomes a\none-character token, then the tokenizer repeatedly finds the most frequent\nadjacent pair of symbols across all words and merges it into a new token,\nuntil `vocab_size`\n\nis reached or there are no pairs left to\nmerge. The merge list is stored in order and replayed identically at\nencode time, so a word is always split the same way it would have been\nsplit during training.\n\nTwo details matter for everything downstream:\n\n`encode()`\n\nalways prepends `<BOS>`\n\n—\nthis matters later because BOS is treated as a real, if artificial,\n\"previous token\" for the very first generation step.`encode_for_training()`\n\nadditionally appends\n`<EOS>`\n\n, so every training sentence has an explicit,\nlearnable end.`graph.py`\n\n)\nOnce every training sentence is a sequence of token ids, MSE-GLM builds\nthree matrices from those sequences. All three are array-backed\n(Python's `array('i')`\n\n) and CSR-indexed — the same\ncompressed-sparse-row layout used in sparse linear algebra — so\nevery lookup (\"what comes after token X?\") is O(1) rather than a scan.\n\nThe simplest structure: every distinct bigram ```\n(token[i],\ntoken[i+1])\n```\n\nseen anywhere in training, deduplicated. Given a token,\n`successors()`\n\nreturns every token that has ever directly\nfollowed it. This is the model's notion of \"grammatically legal\nnext token\" — nothing more.\n\nThis is the heart of the system. For every 3-token window\n`(source, bridge, target)`\n\nin the training sequences, the\nBridge Matrix stores a deduplicated triple. So for *\"the cat sat\"*,\nthe triple is `(source=the, bridge=cat, target=sat)`\n\n.\n\nEvery triple additionally gets a `cluster_id`\n\n, assigned by a\nsimple dual-axis rule:\n\n`(source, target)`\n\npair, their `bridge`\n\ntokens are grouped into one cluster. Example: `(the, sat)`\n\n, so `cat`\n\nand `dog`\n\nget the same cluster_id — they're\ninterchangeable in that slot.`(source, bridge)`\n\npair, their `target`\n\ntokens are\ngrouped instead. Example: `(cat, is)`\n\n, so `animal`\n\nand `pet`\n\nget grouped as things \"cat\" can be.`cluster_id = 0`\n\n— the \"unclustered\" bucket. (More on why that bucket turns out to\nmatter later in this post.)\nAlongside the triples, the Bridge Matrix keeps a `t_index`\n\n:\na reverse map from every token to the (non-zero) cluster_ids it\nparticipates in as a bridge or target anywhere in the graph. This is\nwhat lets the system ask \"does token A show up in the same kind of slot\nas token B, anywhere in the corpus?\" without rescanning everything.\n\nThe Relationship Matrix is deliberately thin: it stores only pairs of\n`(triple_id, relationship_id)`\n\n, where a `relationship_id`\n\nis just the index of the training sentence a triple came from. It doesn't\nduplicate any triple content — it's a foreign key back into the\nBridge Matrix's row order. A single triple can belong to several\nrelationship_ids if the exact same 3-token pattern showed up in more than\none training sentence.\n\nThis sounds like a small bookkeeping detail, but it's what makes generation deterministic rather than a popularity contest (see Step 3).\n\n`inference.py`\n\n)\nThis is the part that replaces what a neural model would do with a\nsoftmax over a hidden state. MSE-GLM's `InferenceEngine`\n\npicks\nthe next token through two ranked stages, both gated by a\nconcept called **lineage** — the set of\n`relationship_id`\n\ns (training sentences) the generation path\nhas stayed consistent with so far.\n\nGiven the current token, collect every legal successor from the Edge\nMatrix. For each candidate `C`\n\n, check whether a Bridge triple\n`(source = current, bridge = C)`\n\nexists whose\n`relationship_id`\n\nintersects the currently active lineage.\nRead as: *\"the current token only votes for a candidate it recognizes\nas its own bridge, and only if that recognition traces back to a\nsentence we're still consistent with.\"*\n\nFirst, the active lineage is narrowed to whatever `relationship_id`\n\ns\nthe exact `(previous, current)`\n\npair actually shares —\nthis keeps the lineage tied to what those two tokens were literally\ntrained on together, rather than a broader set inherited from earlier in\ngeneration. Then, among only the Stage 1 survivors, vote for candidate\n`C`\n\nif the *exact* triple\n`(source = previous, bridge = current, target = C)`\n\nexists\nwith a matching `relationship_id`\n\n.\n\nOne rule holds throughout the whole generation: **active_rels only\never narrows by intersection, never widens**. A later, looser\nmatch can't undo an earlier, more specific one. If Stage 1 ever finds\nliterally no legal successors, the model emits `<EOS>`\n\nimmediately — it doesn't try to guess.\n\n```\nEvery one of these decisions is inspectable.\n```` step()`\n\nreturns\nnot just the chosen token but which stage decided it, which rule fired,\nand what the active lineage was at that moment — this is what makes\nthe \"explainable\" half of \"deterministic, explainable, zero-weight\" true\nin practice, not just in the README.\n\n`experience.py`\n\n)\nStrict Mode only ever says what it directly saw. Open Mode adds a second,\nclearly separated set of matrices — the Experience Edge, Bridge,\nand Relationship matrices — built by a structural inference pass\nover the *already-trained* Bridge Matrix, using cluster\nsubstitutability rather than anything new observed in text.\n\nTwo expansion rules generate new, never-literally-seen triples:\n\n`X`\n\nshares\na cluster with bridge member `B`\n\nin slot `(S, T)`\n\n,\nand `B`\n\nalso bridges some other slot `(S2, T2)`\n\n,\nthen `X`\n\nis inferred to be able to bridge `(S2, T2)`\n\ntoo.`X`\n\nshares a cluster with target member `T`\n\nin\nslot `(S, B)`\n\n, and `T`\n\nalso targets some other\nslot `(S2, B2)`\n\n, `X`\n\ncan target\n`(S2, B2)`\n\ntoo.\nBoth rules are pure substitution logic over clusters already discovered\nin Step 2 — no new information is invented, only recombined. The\nthree resulting matrices are saved separately\n(`experience_edges.json`\n\n, `experience_bridges.json`\n\n,\n`experience_relationships.json`\n\n) so Strict Mode never touches\nthem, and Open Mode inference simply unions both sets of matrices at\nevery lookup. Passing `None`\n\nfor the experience matrices\ndegrades an `InferenceEngine`\n\nstraight back to Strict Mode\nwith no special-casing required.\n\n`analyse.py`\n\n)Because everything is stored as explicit, inspectable matrices, MSE-GLM ships a whole read-only analysis layer on top — it never touches generation, it only reads. Highlights:\n\n`cluster_report`\n\n— lists every non-zero cluster_id,\nits axis (bridge or target), and its members.`token_similarity`\n\n/ `infer_shared_role`\n\n— given two or more tokens, intersects their `t_index`\n\ncluster memberships to find slots they're all interchangeable in, and\npredicts what would fill that shared slot.`trace`\n\n— runs generation for a prompt and prints\nthe stage/rule/lineage behind every single token, exactly as\n`step()`\n\nreturned it.`interpret.py`\n\n)\nA `cluster_id`\n\nis just an integer. Knowing that\n`cat`\n\n, `dog`\n\n, and `pig`\n\nshare cluster 1\ndoesn't tell you *why*. The Cluster Interpreter layer is a\nread-only pass — same category as `analyse.py`\n\n, never\ntouches inference — that proposes a human-readable label for a\ncluster (e.g. naming `{cat, dog, pig}`\n\nas\n**\"animal\"**), gathering evidence from all three matrices\nbuilt in Step 2:\n\n`(bridge, target)`\n\n, let `source`\n\nvary: if cluster\nmembers are each the `(bridge, target)`\n\npair — e.g. each of them is the\nsource of an \"is animal\" triple — that target token becomes a\ncandidate label. `coverage`\n\nis simply the fraction of\nmembers that reach it this way.`source→target`\n\nbigram. So checking whether the corpus\nalso states the fact a shorter way (skipping the bridge word) is\ngenuinely separate evidence, not a restatement of signal 1.`t_index`\n\n, as an\ninterchangeable member of some\nEvery candidate carries an `evidence_mask`\n\nlisting exactly\nwhich of these fired — deliberately not collapsed into one\nconfidence float, because the signals are correlated (they're all read\noff the same training sequences to different degrees), not independent\nprobabilistic votes.\n\nNothing requires exactly one interpreter per cluster. ```\n{cat, dog,\npig}\n```\n\ncan be **\"animal\"** (full coverage, all three)\n*and*, separately, the `{cat, dog}`\n\nsubset within it\ncan independently support **\"pet\"** (partial coverage,\nsince the corpus never calls a pig a pet) — both survive as long\nas each clears its own thresholds. `build_interpreter_matrix`\n\nreturns one row per `(cluster_id, interpreter)`\n\npair, not one\nbest guess per cluster.\n\nThe dual-axis rule from Step 2 only clusters by *fixed source*.\nThere's no rule for \"fix bridge and target, let source vary\" — so a\nset of tokens that are each the source of one categorical statement, but\nnever otherwise share a context, gets **no cluster_id at all**.\nThey sit in the `cluster_id = 0`\n\nbucket individually, invisible\nto every tool in Step 5 and Step 6 so far.\n\n`discover_zero_cluster_groups`\n\nmines that bucket directly,\napplying exactly the missing rule to the rows the first two rules left\nbehind. Tested against a corpus engineered so `cat`\n\n,\n`dog`\n\n, and `pig`\n\nshared nothing else —\ndifferent verbs, different lead-in words — it still recovered\n`{cat, dog, pig} → \"animal\"`\n\npurely from the unclustered\nbucket. It's deliberately scoped as a targeted recovery tool for that one\ngap, not a general quality upgrade: most of what sits at\n`cluster_id = 0`\n\nis unclustered for good reason (one-off\nphrasing, low-frequency noise), and the candidate space it searches is\nmuch larger than the regular interpreter matrix.\n\n| Stage | File | What it does |\n|---|---|---|\n| Tokenize | `tokenizer.py` | From-scratch BPE, no pretrained vocab |\n| Build graph | `graph.py` | Edge, Bridge (+ dual-axis clusters), Relationship matrices |\n| Train | `train.py` | Orchestrates tokenizer + graph construction, persists artifacts |\n| Generate | `inference.py` | Two-stage lineage-vote pipeline, fully traceable |\n| Go beyond training | `experience.py` | Cluster-substitution rules → Open Mode |\n| Analyze | `analyse.py` | Read-only reporting: clusters, similarity, generation traces |\n| Interpret | `interpret.py` | Names clusters using Edge + Bridge + Relationship evidence |\n\nNone of this involves a single learned weight. Every output — a generated token, a cluster label, an inferred Open Mode triple — can be traced back to specific rows in specific matrices, built directly from the training text. That's the whole bet MSE-GLM is making: that a useful amount of language modeling can be done with counting, indexing, and explicit structure, instead of a black box.", "url": "https://wpnews.pro/news/zero-weights-deterministic-graph-language-model-mse-glm", "canonical_source": "https://tonlexianert.com/pages/blog.php", "published_at": "2026-07-17 11:35:04+00:00", "updated_at": "2026-07-17 11:51:15.535736+00:00", "lang": "en", "topics": ["large-language-models", "natural-language-processing", "ai-research"], "entities": ["MSE-GLM"], "alternates": {"html": "https://wpnews.pro/news/zero-weights-deterministic-graph-language-model-mse-glm", "markdown": "https://wpnews.pro/news/zero-weights-deterministic-graph-language-model-mse-glm.md", "text": "https://wpnews.pro/news/zero-weights-deterministic-graph-language-model-mse-glm.txt", "jsonld": "https://wpnews.pro/news/zero-weights-deterministic-graph-language-model-mse-glm.jsonld"}}