Hi everyone,
I have just published the first public prototype of LIMEN Runtime Audit, a small open-source toolkit for inspecting layer-wise activation trajectories in open-weight language models.
GitHub repository:
This is my first public GitHub repository, so feedback on the code, methodology, terminology and documentation would be genuinely valuable.
Most LLM evaluations focus on the final output:
LIMEN adds another descriptive layer: it examines how the model’s hidden activation vector changes as information passes through successive layers.
For each token, the toolkit treats the sequence of hidden states across layers as a measurable trajectory. The objective is not to claim that these trajectories directly represent reasoning, semantic concepts or cognitive states. The objective is to provide a reproducible observability layer for comparing internal runtime behaviour.
Version 0.1 accepts exported NumPy arrays with:
hidden_states
: [tokens, layers, hidden_dim]
logits
: [tokens, vocabulary]
It currently measures:
| Metric | Plain-language interpretation |
|---|---|
| Path length | Total movement across layers |
| Displacement | Direct distance between the first and final layer |
| Tortuosity | How indirect the layer-wise route is |
| Mean speed | Average change between adjacent layers |
| Speed variability | How regularly or irregularly the representation changes |
| Mean acceleration | How much the layer-to-layer movement itself changes |
| Turning angle | How much the direction of movement changes |
| Entropy | Uncertainty in the output distribution |
| Top-1/top-2 margin | Separation between the two leading token probabilities |
The toolkit also:
With proper validation and controls, this kind of instrumentation could potentially help with:
These are possible applications, not validated product claims.
This prototype is currently a descriptive audit and observability tool.
Its measurements do not, by themselves, establish:
A geometrical difference is an observation. Explaining what that difference means requires separate experiments, controls and held-out validation.
The next important milestone is therefore predictive validation: testing whether LIMEN trajectory features detect or predict held-out behavioural regressions beyond conventional baselines and shuffled controls.
The project grew from an independent empirical investigation of runtime activation dynamics. Earlier experiments used broader dynamical-systems terminology. As controls accumulated, I decided that the public engineering implementation should use narrower and more cautious language.
Instead of waiting for every research question to be resolved, I am releasing the minimal audit layer so that other people can inspect it, challenge it and potentially reproduce or improve it.
The repository is released under the Apache 2.0 licence.
GitHub:
Thanks in advance for any critical feedback, failed reproductions, issues or pull requests. At this stage, a rigorous objection is more useful to me than enthusiastic agreement.