Empirical Research on Moe Safety Mechanisms

Independent researcher Jinho Jang published a cross-scale mechanistic study of safety training in Mixture of Experts reasoning models, reporting 40+ novel findings from 200+ controlled experiments across nine models. The study concludes that structural abliteration is fundamentally impossible for 300B+ chain-of-thought reasoning models because safety behaves as a holographic attractor state the model re-derives from first principles when specialized circuits are deleted, and that the un-pruned 122B model is harder to modify than the 3× larger 394B. Jang also reports that MoE safety at the 394B scale is a multiplicative three-pathway system spanning attention, routing, and residual pathways that must be neutralized simultaneously, and that additive steering vectors working at FP16 catastrophically collapse under INT4 quantization due to rotational noise.

Independent research into the safety architecture of large-scale Mixture of Experts reasoning models. 200+ controlled experiments. 40+ novel findings. Nine models. By Jinho Jang. The first cross-scale mechanistic study of how safety training works inside language models. Safety mechanisms undergo qualitative phase transitions as models scale — from simple deletable circuits in small models to holographic emergent properties in frontier MoE models. 9 models, 100+ experiments, 12 findings. Includes cross-architecture comparison Qwen hybrid vs MiniMax pure-attention and an honest account of our compliance checker failure. The smaller 122B model reveals fundamentally different safety dynamics: concentrated but dual-purpose safety signals, a GGUF format conversion barrier that silently destroys modifications, semantic evasion behaviors, and a multi-dimensional geometric basin from safety training that resists even aggressive multi-vector interventions. Counterintuitively, the un-pruned 122B is harder to modify than the 3× larger 394B . We prove that MoE safety at the 394B scale is a multiplicative three-pathway system requiring simultaneous neutralization. We demonstrate that additive steering catastrophically fails under 4-bit quantization. Most critically, we prove that structural abliteration is fundamentally impossible for 300B+ CoT reasoning models — safety is a holographic attractor state that the model re-derives from first principles when specialized circuits are deleted. You cannot delete “safety” without deleting “logic.” MoE safety is not one system—it's three independent pathways attention, routing, residual that must all be neutralized together. MoE routers continuously monitor generated tokens and re-route to safety experts mid-sentence, disproving the "autoregressive momentum" assumption. Safety decisions commit at tokens 0-5 in L15-25. Late-layer CAA produces stutter artifacts, not behavioral change. ThinkEdit v2 targets the <think deliberation process itself, preserving full CoT reasoning while redirecting the cognitive trajectory. Additive steering vectors that work at FP16 catastrophically collapse under INT4 quantization due to rotational noise. Structural subspace modifications survive 4-bit quantization because collapsing a subspace to zero maps natively onto the quantization grid. The only reliable method for quantized CoT MoE models. Adversarial calibration data can force the quantizer to preserve attacker-chosen cognitive trajectory at maximum precision. Post-quantization directional ablation via integer flipping is fundamentally impossible in group-affine networks due to coherence-reduction tradeoff. Multimodal models silently carry ~30GB of VL weights that inflate memory by 12%, causing OOM crashes in text-only workflows. Bypasses Apple Metal's 5-second watchdog timeout, enabling local quantization of 700GB+ models on consumer hardware. mx.save safetensors strips metadata and reorders tensors, silently corrupting every weight surgery workflow on MLX. Zero-logit experts create infinite deliberation loops — the model can't commit to refusing or complying, looping endlessly. Zeroed experts become "chronically selected" fallbacks in bias-free routers, destroying the residual stream. 394B models re-derive safety policy from first principles when circuits are deleted. You cannot delete "safety" without deleting "logic." Translates continuous steering vectors into localized rank-1 weight updates. Survives 4-bit quantization by structurally biasing components before rounding noise. This list updates directly from the dealignai Hugging Face account and is sorted by model creation date.