Most language models today are a pile of weights nobody can fully read. MSE-GLM (Matrix-Structured Edge — Graph Language Model) is an attempt at the opposite: a language model with no neural weights, no embeddings, and no probability distributions. Every decision it makes during generation can be traced back to a specific row in a specific matrix, built from the training text itself. Nothing is learned in the gradient-descent sense — everything is counted, deduplicated, and indexed.
This post walks through every stage of the pipeline in the order it actually runs — tokenizing, building the graph, generating text, inferring beyond what was literally seen, and finally, a newer piece: figuring out what a cluster of interchangeable tokens actually means.
tokenizer.py
, graph.py
,
train.py
, inference.py
, experience.py
,
analyse.py
, interpret.py
, model.py
tokenizer.py
)
Everything starts with a from-scratch Byte Pair Encoding (BPE) tokenizer.
No external tokenizer library, no pretrained vocabulary. Training text is
lowercased, stripped of anything that isn't a letter, digit, or space, and
split into sentences on . ! ?
and newlines.
Four reserved ids anchor the vocabulary before any merges happen:
<PAD> = 0 reserved
<UNK> = 1 unknown character fallback
<BOS> = 2 prepended to every encoded sequence
<EOS> = 3 appended only during training
From there it's classic BPE: every character in the corpus becomes a
one-character token, then the tokenizer repeatedly finds the most frequent
adjacent pair of symbols across all words and merges it into a new token,
until vocab_size
is reached or there are no pairs left to merge. The merge list is stored in order and replayed identically at encode time, so a word is always split the same way it would have been split during training.
Two details matter for everything downstream:
encode()
always prepends <BOS>
—
this matters later because BOS is treated as a real, if artificial,
"previous token" for the very first generation step.encode_for_training()
additionally appends
<EOS>
, so every training sentence has an explicit,
learnable end.graph.py
)
Once every training sentence is a sequence of token ids, MSE-GLM builds
three matrices from those sequences. All three are array-backed
(Python's array('i')
) and CSR-indexed — the same compressed-sparse-row layout used in sparse linear algebra — so every lookup ("what comes after token X?") is O(1) rather than a scan.
The simplest structure: every distinct bigram ``` (token[i], token[i+1])
seen anywhere in training, deduplicated. Given a token,
`successors()`
returns every token that has ever directly
followed it. This is the model's notion of "grammatically legal
next token" — nothing more.
This is the heart of the system. For every 3-token window
`(source, bridge, target)`
in the training sequences, the
Bridge Matrix stores a deduplicated triple. So for *"the cat sat"*,
the triple is `(source=the, bridge=cat, target=sat)`
.
Every triple additionally gets a `cluster_id`
, assigned by a
simple dual-axis rule:
`(source, target)`
pair, their `bridge`
tokens are grouped into one cluster. Example: `(the, sat)`
, so `cat`
and `dog`
get the same cluster_id — they're
interchangeable in that slot.`(source, bridge)`
pair, their `target`
tokens are
grouped instead. Example: `(cat, is)`
, so `animal`
and `pet`
get grouped as things "cat" can be.`cluster_id = 0`
— the "unclustered" bucket. (More on why that bucket turns out to
matter later in this post.)
Alongside the triples, the Bridge Matrix keeps a `t_index`
:
a reverse map from every token to the (non-zero) cluster_ids it
participates in as a bridge or target anywhere in the graph. This is
what lets the system ask "does token A show up in the same kind of slot
as token B, anywhere in the corpus?" without rescanning everything.
The Relationship Matrix is deliberately thin: it stores only pairs of
`(triple_id, relationship_id)`
, where a `relationship_id`
is just the index of the training sentence a triple came from. It doesn't
duplicate any triple content — it's a foreign key back into the
Bridge Matrix's row order. A single triple can belong to several
relationship_ids if the exact same 3-token pattern showed up in more than
one training sentence.
This sounds like a small bookkeeping detail, but it's what makes generation deterministic rather than a popularity contest (see Step 3).
`inference.py`
)
This is the part that replaces what a neural model would do with a
softmax over a hidden state. MSE-GLM's `InferenceEngine`
picks
the next token through two ranked stages, both gated by a
concept called **lineage** — the set of
`relationship_id`
s (training sentences) the generation path
has stayed consistent with so far.
Given the current token, collect every legal successor from the Edge
Matrix. For each candidate `C`
, check whether a Bridge triple
`(source = current, bridge = C)`
exists whose
`relationship_id`
intersects the currently active lineage.
Read as: *"the current token only votes for a candidate it recognizes
as its own bridge, and only if that recognition traces back to a
sentence we're still consistent with."*
First, the active lineage is narrowed to whatever `relationship_id`
s
the exact `(previous, current)`
pair actually shares —
this keeps the lineage tied to what those two tokens were literally
trained on together, rather than a broader set inherited from earlier in
generation. Then, among only the Stage 1 survivors, vote for candidate
`C`
if the *exact* triple
`(source = previous, bridge = current, target = C)`
exists
with a matching `relationship_id`
.
One rule holds throughout the whole generation: **active_rels only
ever narrows by intersection, never widens**. A later, looser
match can't undo an earlier, more specific one. If Stage 1 ever finds
literally no legal successors, the model emits `<EOS>`
immediately — it doesn't try to guess.
Every one of these decisions is inspectable. ```` step()`
returns not just the chosen token but which stage decided it, which rule fired, and what the active lineage was at that moment — this is what makes the "explainable" half of "deterministic, explainable, zero-weight" true in practice, not just in the README.
experience.py
) Strict Mode only ever says what it directly saw. Open Mode adds a second, clearly separated set of matrices — the Experience Edge, Bridge, and Relationship matrices — built by a structural inference pass over the already-trained Bridge Matrix, using cluster substitutability rather than anything new observed in text.
Two expansion rules generate new, never-literally-seen triples:
X
shares
a cluster with bridge member B
in slot (S, T)
,
and B
also bridges some other slot (S2, T2)
,
then X
is inferred to be able to bridge (S2, T2)
too.X
shares a cluster with target member T
in
slot (S, B)
, and T
also targets some other
slot (S2, B2)
, X
can target
(S2, B2)
too.
Both rules are pure substitution logic over clusters already discovered
in Step 2 — no new information is invented, only recombined. The
three resulting matrices are saved separately
(experience_edges.json
, experience_bridges.json
,
experience_relationships.json
) so Strict Mode never touches
them, and Open Mode inference simply unions both sets of matrices at
every lookup. Passing None
for the experience matrices
degrades an InferenceEngine
straight back to Strict Mode with no special-casing required.
analyse.py
)Because everything is stored as explicit, inspectable matrices, MSE-GLM ships a whole read-only analysis layer on top — it never touches generation, it only reads. Highlights:
cluster_report
— lists every non-zero cluster_id,
its axis (bridge or target), and its members.token_similarity
/ infer_shared_role
— given two or more tokens, intersects their t_index
cluster memberships to find slots they're all interchangeable in, and
predicts what would fill that shared slot.trace
— runs generation for a prompt and prints
the stage/rule/lineage behind every single token, exactly as
step()
returned it.interpret.py
)
A cluster_id
is just an integer. Knowing that
cat
, dog
, and pig
share cluster 1
doesn't tell you why. The Cluster Interpreter layer is a
read-only pass — same category as analyse.py
, never
touches inference — that proposes a human-readable label for a
cluster (e.g. naming {cat, dog, pig}
as "animal"), gathering evidence from all three matrices built in Step 2:
(bridge, target)
, let source
vary: if cluster
members are each the (bridge, target)
pair — e.g. each of them is the
source of an "is animal" triple — that target token becomes a
candidate label. coverage
is simply the fraction of
members that reach it this way.source→target
bigram. So checking whether the corpus
also states the fact a shorter way (skipping the bridge word) is
genuinely separate evidence, not a restatement of signal 1.t_index
, as an
interchangeable member of some
Every candidate carries an evidence_mask
listing exactly which of these fired — deliberately not collapsed into one confidence float, because the signals are correlated (they're all read off the same training sequences to different degrees), not independent probabilistic votes.
Nothing requires exactly one interpreter per cluster. ``` {cat, dog, pig}
can be **"animal"** (full coverage, all three)
*and*, separately, the `{cat, dog}`
subset within it
can independently support **"pet"** (partial coverage,
since the corpus never calls a pig a pet) — both survive as long
as each clears its own thresholds. `build_interpreter_matrix`
returns one row per `(cluster_id, interpreter)`
pair, not one
best guess per cluster.
The dual-axis rule from Step 2 only clusters by *fixed source*.
There's no rule for "fix bridge and target, let source vary" — so a
set of tokens that are each the source of one categorical statement, but
never otherwise share a context, gets **no cluster_id at all**.
They sit in the `cluster_id = 0`
bucket individually, invisible
to every tool in Step 5 and Step 6 so far.
`discover_zero_cluster_groups`
mines that bucket directly,
applying exactly the missing rule to the rows the first two rules left
behind. Tested against a corpus engineered so `cat`
,
`dog`
, and `pig`
shared nothing else —
different verbs, different lead-in words — it still recovered
`{cat, dog, pig} → "animal"`
purely from the unclustered
bucket. It's deliberately scoped as a targeted recovery tool for that one
gap, not a general quality upgrade: most of what sits at
`cluster_id = 0`
is unclustered for good reason (one-off
phrasing, low-frequency noise), and the candidate space it searches is
much larger than the regular interpreter matrix.
| Stage | File | What it does |
|---|---|---|
| Tokenize | `tokenizer.py` | From-scratch BPE, no pretrained vocab |
| Build graph | `graph.py` | Edge, Bridge (+ dual-axis clusters), Relationship matrices |
| Train | `train.py` | Orchestrates tokenizer + graph construction, persists artifacts |
| Generate | `inference.py` | Two-stage lineage-vote pipeline, fully traceable |
| Go beyond training | `experience.py` | Cluster-substitution rules → Open Mode |
| Analyze | `analyse.py` | Read-only reporting: clusters, similarity, generation traces |
| Interpret | `interpret.py` | Names clusters using Edge + Bridge + Relationship evidence |
None of this involves a single learned weight. Every output — a generated token, a cluster label, an inferred Open Mode triple — can be traced back to specific rows in specific matrices, built directly from the training text. That's the whole bet MSE-GLM is making: that a useful amount of language modeling can be done with counting, indexing, and explicit structure, instead of a black box.