Zero Weights Deterministic Graph Language Model (MSE-GLM)

Researchers introduced MSE-GLM, a language model that operates without neural weights, embeddings, or probability distributions, relying instead on deterministic matrix structures built from training text. The model uses a custom BPE tokenizer and sparse matrices to enable fully traceable text generation and inference.

Most language models today are a pile of weights nobody can fully read. MSE-GLM Matrix-Structured Edge — Graph Language Model is an attempt at the opposite: a language model with no neural weights, no embeddings, and no probability distributions . Every decision it makes during generation can be traced back to a specific row in a specific matrix, built from the training text itself. Nothing is learned in the gradient-descent sense — everything is counted, deduplicated, and indexed . This post walks through every stage of the pipeline in the order it actually runs — tokenizing, building the graph, generating text, inferring beyond what was literally seen, and finally, a newer piece: figuring out what a cluster of interchangeable tokens actually means . tokenizer.py , graph.py , train.py , inference.py , experience.py , analyse.py , interpret.py , model.py tokenizer.py Everything starts with a from-scratch Byte Pair Encoding BPE tokenizer. No external tokenizer library, no pretrained vocabulary. Training text is lowercased, stripped of anything that isn't a letter, digit, or space, and split into sentences on . ? and newlines. Four reserved ids anchor the vocabulary before any merges happen: <PAD = 0 reserved <UNK = 1 unknown character fallback <BOS = 2 prepended to every encoded sequence <EOS = 3 appended only during training From there it's classic BPE: every character in the corpus becomes a one-character token, then the tokenizer repeatedly finds the most frequent adjacent pair of symbols across all words and merges it into a new token, until vocab size is reached or there are no pairs left to merge. The merge list is stored in order and replayed identically at encode time, so a word is always split the same way it would have been split during training. Two details matter for everything downstream: encode always prepends <BOS — this matters later because BOS is treated as a real, if artificial, "previous token" for the very first generation step. encode for training additionally appends <EOS , so every training sentence has an explicit, learnable end. graph.py Once every training sentence is a sequence of token ids, MSE-GLM builds three matrices from those sequences. All three are array-backed Python's array 'i' and CSR-indexed — the same compressed-sparse-row layout used in sparse linear algebra — so every lookup "what comes after token X?" is O 1 rather than a scan. The simplest structure: every distinct bigram token i , token i+1 seen anywhere in training, deduplicated. Given a token, successors returns every token that has ever directly followed it. This is the model's notion of "grammatically legal next token" — nothing more. This is the heart of the system. For every 3-token window source, bridge, target in the training sequences, the Bridge Matrix stores a deduplicated triple. So for "the cat sat" , the triple is source=the, bridge=cat, target=sat . Every triple additionally gets a cluster id , assigned by a simple dual-axis rule: source, target pair, their bridge tokens are grouped into one cluster. Example: the, sat , so cat and dog get the same cluster id — they're interchangeable in that slot. source, bridge pair, their target tokens are grouped instead. Example: cat, is , so animal and pet get grouped as things "cat" can be. cluster id = 0 — the "unclustered" bucket. More on why that bucket turns out to matter later in this post. Alongside the triples, the Bridge Matrix keeps a t index : a reverse map from every token to the non-zero cluster ids it participates in as a bridge or target anywhere in the graph. This is what lets the system ask "does token A show up in the same kind of slot as token B, anywhere in the corpus?" without rescanning everything. The Relationship Matrix is deliberately thin: it stores only pairs of triple id, relationship id , where a relationship id is just the index of the training sentence a triple came from. It doesn't duplicate any triple content — it's a foreign key back into the Bridge Matrix's row order. A single triple can belong to several relationship ids if the exact same 3-token pattern showed up in more than one training sentence. This sounds like a small bookkeeping detail, but it's what makes generation deterministic rather than a popularity contest see Step 3 . inference.py This is the part that replaces what a neural model would do with a softmax over a hidden state. MSE-GLM's InferenceEngine picks the next token through two ranked stages, both gated by a concept called lineage — the set of relationship id s training sentences the generation path has stayed consistent with so far. Given the current token, collect every legal successor from the Edge Matrix. For each candidate C , check whether a Bridge triple source = current, bridge = C exists whose relationship id intersects the currently active lineage. Read as: "the current token only votes for a candidate it recognizes as its own bridge, and only if that recognition traces back to a sentence we're still consistent with." First, the active lineage is narrowed to whatever relationship id s the exact previous, current pair actually shares — this keeps the lineage tied to what those two tokens were literally trained on together, rather than a broader set inherited from earlier in generation. Then, among only the Stage 1 survivors, vote for candidate C if the exact triple source = previous, bridge = current, target = C exists with a matching relationship id . One rule holds throughout the whole generation: active rels only ever narrows by intersection, never widens . A later, looser match can't undo an earlier, more specific one. If Stage 1 ever finds literally no legal successors, the model emits <EOS immediately — it doesn't try to guess. Every one of these decisions is inspectable. step returns not just the chosen token but which stage decided it, which rule fired, and what the active lineage was at that moment — this is what makes the "explainable" half of "deterministic, explainable, zero-weight" true in practice, not just in the README. experience.py Strict Mode only ever says what it directly saw. Open Mode adds a second, clearly separated set of matrices — the Experience Edge, Bridge, and Relationship matrices — built by a structural inference pass over the already-trained Bridge Matrix, using cluster substitutability rather than anything new observed in text. Two expansion rules generate new, never-literally-seen triples: X shares a cluster with bridge member B in slot S, T , and B also bridges some other slot S2, T2 , then X is inferred to be able to bridge S2, T2 too. X shares a cluster with target member T in slot S, B , and T also targets some other slot S2, B2 , X can target S2, B2 too. Both rules are pure substitution logic over clusters already discovered in Step 2 — no new information is invented, only recombined. The three resulting matrices are saved separately experience edges.json , experience bridges.json , experience relationships.json so Strict Mode never touches them, and Open Mode inference simply unions both sets of matrices at every lookup. Passing None for the experience matrices degrades an InferenceEngine straight back to Strict Mode with no special-casing required. analyse.py Because everything is stored as explicit, inspectable matrices, MSE-GLM ships a whole read-only analysis layer on top — it never touches generation, it only reads. Highlights: cluster report — lists every non-zero cluster id, its axis bridge or target , and its members. token similarity / infer shared role — given two or more tokens, intersects their t index cluster memberships to find slots they're all interchangeable in, and predicts what would fill that shared slot. trace — runs generation for a prompt and prints the stage/rule/lineage behind every single token, exactly as step returned it. interpret.py A cluster id is just an integer. Knowing that cat , dog , and pig share cluster 1 doesn't tell you why . The Cluster Interpreter layer is a read-only pass — same category as analyse.py , never touches inference — that proposes a human-readable label for a cluster e.g. naming {cat, dog, pig} as "animal" , gathering evidence from all three matrices built in Step 2: bridge, target , let source vary: if cluster members are each the bridge, target pair — e.g. each of them is the source of an "is animal" triple — that target token becomes a candidate label. coverage is simply the fraction of members that reach it this way. source→target bigram. So checking whether the corpus also states the fact a shorter way skipping the bridge word is genuinely separate evidence, not a restatement of signal 1. t index , as an interchangeable member of some Every candidate carries an evidence mask listing exactly which of these fired — deliberately not collapsed into one confidence float, because the signals are correlated they're all read off the same training sequences to different degrees , not independent probabilistic votes. Nothing requires exactly one interpreter per cluster. {cat, dog, pig} can be "animal" full coverage, all three and , separately, the {cat, dog} subset within it can independently support "pet" partial coverage, since the corpus never calls a pig a pet — both survive as long as each clears its own thresholds. build interpreter matrix returns one row per cluster id, interpreter pair, not one best guess per cluster. The dual-axis rule from Step 2 only clusters by fixed source . There's no rule for "fix bridge and target, let source vary" — so a set of tokens that are each the source of one categorical statement, but never otherwise share a context, gets no cluster id at all . They sit in the cluster id = 0 bucket individually, invisible to every tool in Step 5 and Step 6 so far. discover zero cluster groups mines that bucket directly, applying exactly the missing rule to the rows the first two rules left behind. Tested against a corpus engineered so cat , dog , and pig shared nothing else — different verbs, different lead-in words — it still recovered {cat, dog, pig} → "animal" purely from the unclustered bucket. It's deliberately scoped as a targeted recovery tool for that one gap, not a general quality upgrade: most of what sits at cluster id = 0 is unclustered for good reason one-off phrasing, low-frequency noise , and the candidate space it searches is much larger than the regular interpreter matrix. | Stage | File | What it does | |---|---|---| | Tokenize | tokenizer.py | From-scratch BPE, no pretrained vocab | | Build graph | graph.py | Edge, Bridge + dual-axis clusters , Relationship matrices | | Train | train.py | Orchestrates tokenizer + graph construction, persists artifacts | | Generate | inference.py | Two-stage lineage-vote pipeline, fully traceable | | Go beyond training | experience.py | Cluster-substitution rules → Open Mode | | Analyze | analyse.py | Read-only reporting: clusters, similarity, generation traces | | Interpret | interpret.py | Names clusters using Edge + Bridge + Relationship evidence | None of this involves a single learned weight. Every output — a generated token, a cluster label, an inferred Open Mode triple — can be traced back to specific rows in specific matrices, built directly from the training text. That's the whole bet MSE-GLM is making: that a useful amount of language modeling can be done with counting, indexing, and explicit structure, instead of a black box.