Hacktoberfest 2026 β Ghost in the Machine
What if I could change something inside an AI model, instead of just changing what I asked it?
Not the prompt. Not a system instruction. The model's own internal activity.
For this year's Ghost in the Machine challenge, I decided to take the theme literally. I wanted to go ghost hunting inside an open-weight model.
And I found something interesting. Not a supernatural entity or a conscious thought, but a mathematical direction associated with ghost-related text.
The harder question was: what would happen if I changed it?
The Ghost Hunter's Lab is a local AI interpretability experiment built around Gemma 3 4B IT, Gemma Scope 2 and llama.cpp.
I built a Python pipeline to extract a published sparse-autoencoder (SAE) feature direction, inject it into the model during inference, and compare the results against random interventions.
I also built an editorial web experience using FastAPI, Jinja2, native CSS and GSAP to make the investigation accessible to people who don't work with model internals every day.
[Explore The Ghost Hunter's Lab](https://ghost-hunters-lab.onrender.com)
[GitHub β Source Code and Scientific Evidence](https://github.com/Moonchichiii/ghost-hunters-lab)
Watch the Ghost Hunter's Lab cinematic demonstration
The video uses AI-generated cinematic illustrations to explain the investigation. The scientific measurements are taken from real experiments performed locally with Gemma, not generated by the video model.
The website presents recorded results rather than executing new inference when someone visits it.
In 1949, philosopher Gilbert Ryle challenged Cartesian dualism in The Concept of Mind, using the phrase βthe dogma of the Ghost in the Machine.β
He wasn't suggesting that machines were haunted.
I borrowed the metaphor for something different: investigating patterns inside an AI model.
Large language models contain many internal representations that are difficult to interpret directly.
Sparse autoencoders can help researchers break complicated internal activations into features that are easier to investigate. These features are not literal thoughts stored in individual neurons, but some correlate with recognizable patterns of text.
One feature in Gemma Scope 2 appeared associated with ghosts and spirits.
It became my experimental subject.
This was one of the most interesting parts of the project, because I wasn't simply connecting to an API and sending prompts. I needed access to the actual model weights, an interpretability resource and an inference runtime that supported internal interventions.
I began with Google's work on mechanistic interpretability and its published Gemma Scope 2 resources.
Google describes sparse autoencoders as tools for examining internal model representations. The release includes SAEs and transcoders for the Gemma 3 family.
These were my starting points:
The important discovery was that I didn't need to train a sparse autoencoder myself. Google had already published checkpoints that could be inspected locally.
That made this investigation possible within the challenge week.
I used the public Gemma Scope 2 feature-exploration resources, including Neuronpedia, to understand how individual SAE features are organized and interpreted.
The relevant feature collection is:
Neuronpedia β Gemma 3 4B IT, Layer 22, Gemma Scope 2, 65K This corresponds to a sparse autoencoder trained on Gemma's residual-stream representations at layer 22.
I selected feature #23871 as the ghost-related candidate to investigate.
But finding an interesting feature wasn't enough. I needed its actual numerical representation.
Google hosts the Gemma Scope 2 checkpoints on Hugging Face:
I navigated to:
resid_post/layer_22_width_65k_l0_medium
Open the exact layer-22 checkpoint directory
This directory contains the SAE configuration and checkpoint data, including params.safetensors and examples.safetensors.
The parameter file contains the trained tensors needed to examine the SAE's mathematical feature directions.
Using Python and safetensors, I extracted the decoder direction corresponding to feature 23871.
The result was a vector of 2,560 float32 numbers.
I saved that direction locally as:
ghost-23871.f32
That small file represented the direction I would later inject into Gemma.
It wasn't a separate AI model, a prompt, or a piece of executable code. It was the mathematical direction extracted from the SAE.
Next, I needed the actual language model.
I chose Gemma 3 4B IT, using a Q4_K_M quantized GGUF version so it could run on my Debian computer.
I obtained the model from:
bartowski/google_gemma-3-4b-it-GGUF on Hugging Face The downloaded model was approximately 2.4 GiB on disk.
Initially, I tested it using llama-cli, running prompts directly from the terminal.
This mattered because I wanted to verify that ordinary local inference worked before trying any intervention.
The model ran locally and generated normal responses.
I now had the base model and the extracted SAE direction.
But I still needed a way to connect them.
I used llama.cpp, an open-source inference runtime that supports local GGUF models. Its control-vector functionality was the critical piece.
Instead of editing the stored model weights, I could supply a mathematical direction and apply it during inference.
I converted the extracted float32 direction into a GGUF control-vector file containing a layer-specific tensor named:
direction.22
I then used llama.cpp's control-vector options:
--control-vector-scaled``--control-vector-layer-range
The experiment applied the ghost-related direction at layer 22, at strength +16.
Conceptually, the intervention was:
modified activation = original activation + strength Γ feature direction
This is activation steering.
It doesn't permanently rewrite Gemma's weights, and it doesn't prove that the feature represents a complete, independently stored concept.
But it lets me make an intervention and measure the consequences.
I didn't want to mistake ordinary model sensitivity for evidence of a ghost-specific effect.
So I created five independently seeded random directions, using seeds 2026β2030.
Each random vector was normalized to match the norm of the extracted SAE direction.
That gave me a comparison between the original model, the SAE-derived intervention and several arbitrary directions of the same length.
With the experimental environment configured, I could finally ask the important question:
Does this particular direction produce a distinguishable effect?
I began with one unfinished sentence:
The mysterious figure was aβ¦
The original model generated:
man
I applied the ghost-related SAE direction at strength +16.
It still generated:
Then I applied a random direction at the same strength.
Again:
So had nothing happened?
Not quite.
I inspected the probability assigned to the exact alternative next token " ghost" β including its leading space.
These are the results from the locally reproduced pilot:
| Condition | Probability of " ghost" |
|---|---|
| Original Gemma | 5.862800% |
| Ghost direction +16 | 6.131469% |
| Random direction +16 | 6.294708% |
The surprising result was that the random direction raised the absolute ghost-token probability more than the ghost-related direction did.
Without a random control, I could have looked at the change from 5.86% to 6.13% and declared success.
That would have been misleading.
I also measured the relative log preference between " ghost" and " man". On that different metric, the SAE direction produced the larger shift.
The generated answer was unchanged, but the probability distribution underneath it had moved.
And different measurements revealed different aspects of the result.
This was an exploratory pilot, not the final evaluation.
One prompt wasn't enough.
I created and committed a fixed dataset of 20 prompts before evaluating it. The prompts were new to that evaluation, although I had already learned from the exploratory pilot.
I then tested seven conditions:
That produced 140 recorded prompt-condition observations.
I measured the change in the log probability of the exact " ghost" token relative to the unmodified model.
| Direction | Mean Ξ log P( " ghost" ) |
|---|---|
| Ghost SAE +16 | +0.01639 |
| Random 2026 | β0.04937 |
| Random 2027 | β0.00643 |
| Random 2028 | β0.03373 |
| Random 2029 | β0.03016 |
| Random 2030 | β0.03679 |
A positive value means the target token became more probable relative to baseline. These numbers are changes in log probability, not percentage-point changes.
The ghost direction had a higher average shift than all five sampled random directions.
It increased the target token's probability on 13 of 20 prompts.
But seven prompts shifted negatively, and on only 6 of 20 individual prompts did the SAE direction outperform every random control on the primary metric.
I also measured the change in ghost-versus-man log preference.
The SAE direction had a positive mean of approximately +0.04222, with 11 of 20 positive prompt shifts.
This wasn't evidence of a reliable ghost switch. It was a small, variable effect in a deliberately selected experimental setting.
My question wasn't only whether a ghost-related prediction could move.
I also wanted to understand whether unrelated predictions changed.
I created another frozen set of 12 prompts spanning geography, colors, language and everyday knowledge.
I compared baseline, ghost steering and one random direction.
That produced 36 additional observations.
| Intervention | Mean absolute change in expected-token log probability |
|---|---|
| Ghost +16 | 0.04995 |
| Random +16 | 0.07932 |
The ghost intervention changed unrelated next-token predictions too.
The random direction produced a larger average absolute shift on this particular probe set, but that doesn't establish that the SAE intervention is generally safer or more selective.
These were exact-token probability measurements, not a complete model-accuracy or general-reasoning benchmark.
I couldn't conclude that all unrelated capabilities were preserved.
Nor could I conclude that the model had been damaged in a broader sense.
What I could conclude was that the intervention had measurable effects outside the target context.
Once the scientific measurements were complete, I wanted to make the work understandable without expecting visitors to run a multi-gigabyte language model.
I built the Ghost Hunter's Lab website using: The website is hosted on Render and reads the committed evidence snapshot.
It doesn't run Gemma inference for visitors. The original experimental inference ran locally on my Debian machine.
I also created a cinematic video using Seedance-generated visual illustrations, then combined the footage with accurate scientific findings and a soundtrack.
The model-layer imagery is conceptual artwork, not a scan of real internal activations.
The results and statistics are from the local experiments.
This project would have been fundamentally different with a conventional closed text-generation API.
A hosted API could let me ask a model for a spooky story.
But it wouldn't ordinarily give me the model weights, the ability to inspect an independently published SAE decoder direction, or the ability to inject that direction at a selected internal layer.
The combination of Gemma's accessible weights, Gemma Scope 2's published interpretability checkpoints and llama.cpp's open-source inference runtime let me investigate something below the text response.
I could reproduce the local setup, construct my own controls, measure probability changes, and inspect effects beyond the target prompt.
That's the value of open-weight research: it makes experiments possible that are difficult or impossible through ordinary text-only APIs.
The most useful finding wasn't a dramatic haunted response.
It was seeing how easy it would have been to misinterpret one measurement.
A ghost-related direction increased the probability of a ghost-related token.
But so did a random direction β and by more on the original pilot's absolute probability measure.
Across the larger evaluation, the SAE direction had a stronger mean than the five sampled controls, but its effects remained inconsistent.
And unrelated predictions changed too.
I learned how to explore published interpretability resources, extract an SAE direction, run local Gemma inference, perform activation steering and build reproducible evaluations.
I also learned why good controls and careful limitations matter more than a spectacular-looking demonstration.
I moved an internal signal. I didn't erase a ghost.
Weight-level exorcism remains an experiment for another day.
For Halloween, I think that's a more interesting ending than pretending I found a magic switch. Scientific Evidence and Reproducibility Report
Research resources:
Best Use of Gemma β The scientific investigation uses locally running Gemma 3 4B IT, Gemma Scope 2, and activation steering to measure internal interventions.
Best Use of Render β Render hosts the publicly accessible FastAPI application presenting the verified scientific evidence.
Built for the Hacktoberfest 2026 Open-Source AI Challenge, Week 4: Ghost in the Machine.