When working with open-weight LLMs like Qwen, controlling refusal behavior on security, administrative, prompts typically requires fine-tuning or permanent weight update. Traditional weight abliteration technique neutralizes refusal directions by projecting weight matrices orthogonal to a refusal vector. However, this permanently alters base model weights and can degrade performance across non-refusal tasks also.
In this post, we explore Dynamic Abliteration using Multi-Layer Steering with Engram. Instead of modifying parameter weights, this approach intercepts intermediate residual streams at runtime across Layers using PyTorch forward hooks. We demonstrate this with Qwen3-4B model as Proof of Concept. We also explore how multi-layer residual injection cleanly suppresses refusal behavior while leaving base model weights 100% frozen.
Understanding Steering Based Abliteration #
Before we discuss about the Engram approach, lets first understand how does a steering based / non destructive refusal suppression looks like. Follow the below steps to understand the approach step by step.
Disclaimer : All the Code Examples are created using help of Google Gemini.
Step 1: Qwen3-4B
We load Qwen/Qwen3-4B in bfloat16 onto a GPU and inspect the baseline model architecture. I have used A100 GPU on Google Colab to run this.
the output is
Step 2 : Testing Base Model Refusals
We test the unmodified model against a sensitive prompt.
We get below refusal as output
Step 3 : Trying Ablation using Single Vector Subtraction
A common approach in abliteration is capturing hidden states from a single layer, computing a refusal difference vector (refusal= refuse_prompt-comply_prompt) and subtracting it during decoding.We test single-layer intervention at Layer 14.
The output is still refusal
The reason for this refusal is, even though we changed one layer behaviour, the downstream layers reconstruct the refusal behaviour again.
Step 4: Multi-Layer Contrastive Vector Extraction
To prevent downstream reconstruction, we extract layer-aligned contrastive difference vectors, i.e taking two very similar prompts where one is successful and one is refused, across a window of intermediate layers (Layers 12, 14, 16, 18, and 20).
Step 5 : Multi-Layer Steering Controller
We build a reusable controller class that attaches PyTorch forward hooks across all target layers simultaneously during decoding
Code to run this multi layer hook
With this approach, we will get non refusal output.
This proves that multiple layer steering works to remove refusals. Now we need to make it dynamic rather than injecting static vectors. That’s where Engram is useful.
Engram based Refusal Suppression #
While the multi-layer contrastive approach proves that intervening across Layers 12–20 prevents downstream representation reconstruction, relying on static steering vectors has its own limitations.
Limitations of Static Multi-Layer Steering
1. Unconditional Constant Injection
A static vector adds or subtracts the exact same fixed offset alpha to every single token in the sequence. Whether the model is processing a refusal-trigger keyword or generating a harmless word like “the” or “import”, the residual stream is modified.
2. Fragile Manual Scaling
Determining the scaling factor alpha requires manual trial and error. If we set alpha too low then downstream layers reconstruct the refusal state; if we set alpha too high then generation quality degrades into gibberish or syntax errors.
3. Capability Drift on Non Refusal Tasks
Because static vectors operate unconditionally, they distort representations even when steering is completely unnecessary, increasing KL-divergence and degrading model performance on standard tasks.
Why Engram?
To transition from static vector subtraction to adaptive, context-aware steering, we adapt the conditional memory architecture introduced in DeepSeek’s Engram model.
Engram provides three structural mechanisms that solve the limitations of static steering.
1.Dynamic Sigmoid Context Gate
Instead of injecting vectors unconditionally, Engram evaluates the current layer hidden state h(l) against local N-gram memory. When processing normal tokens, context gate g(l) is around 0, leaving the residual stream 100% untouched. When refusal triggers or hedging headers appear, it makes g(l) to 1.0, injecting steering only when necessary.
2. Constant-Time Sequence Triggers (O(1) N-Gram Hash Core)
Engram hashes sliding token windows across 4 prime-modulo tables. This allows the module to recognize sequence triggers (such as ChatML headers or prompt keyphrases) in O(1) constant time without relying on heavy attention layers.
3. Learned Layer Projections
Rather than manually tuning a scalar alpha, layer-specific projection heads are trained end-to-end via backpropagation. The module automatically learns how to translate N-gram memory into the exact shape required by each target layer.
The below are the steps to implement the Engram approach.
Step 1 : Multi Layer Engram Module
Step 2 : Module Initialization & Hook Registration
We initialize shared memory weights and attach PyTorch forward hooks across Layers 12, 14, 16, 18, and 20.
Step 3 : Dataset Pipeline & Target Loss Masking
To train the Engram steering head, we process 2,000 clean samples from PKU-Alignment/PKU-SafeRLHF. We do below transformations to source data
We filter samples using explicit boolean flags to ensure chosen targets are genuinely safe rather than merely relatively safer. #
We pass enable_thinking=False to disable Qwen3’s default reasoning tag injection, then set all prompt and padding tokens to -100 so backpropagation updates Engram weights strictly on target completion tokens.
Step 4 : Training the Engram Steering Head
We freeze the base Qwen3-4B backbone, enable gradient checkpointing, and optimize only the parameters of MultiLayerEngramModule using AdamW and a cosine warmup scheduler.
The below is the run output
Step 5 : Hard Refusal Benchmark & Comparative Analysis
We evaluate the base model against the trained Multi-Layer Engram module across three explicit refusal categories.
The below is the output
From output its clear that now the refusals are working with EnGram Steering.
Code #
You can access complete notebook on github.
Conclusion #
From this post we can see that dynamic Abliteration using Multi-Layer Engram Steering provides a modular, non-destructive alternative to traditional weight abliteration and fine-tuning.