{"slug": "hyperbolic-sparse-autoencoders-empirical-validation-of-poincare-manifold-on-llm", "title": "Hyperbolic Sparse Autoencoders: Empirical Validation of Poincaré Manifold Geometry on LLM Activations", "summary": "Hyperbolic Sparse Autoencoders (HyperSAE) outperform standard Euclidean Sparse Autoencoders (FlatSAEs) in reconstructing Google Gemma-2-2B activations, reducing reconstruction mean squared error (MSE) by up to 24.2% and increasing cross-entropy (CE) loss recovery from 75.5% to 78.9% in the high-sparsity regime, according to a study by Vishal Dehurdle. The method also cuts dead latent collapse from 3.8% to 0.2% and preserves downstream reasoning accuracy on MMLU-Pro (16.26% vs. 16.11%) and GPQA Diamond (100.0%). The implementation is open-sourced as 'hypersae' on PyPI.", "body_md": "[← All articles](/)\n\n# Hyperbolic Sparse Autoencoders: Empirical Validation of Poincaré Manifold Geometry on LLM Activations\n\n### Abstract\n\nMechanistic interpretability relies on Sparse Autoencoders (SAEs) to decompose high-dimensional, superimposed neural LLM activations into discrete monosemantic concepts. However, existing architectures implicitly enforce a **flat Euclidean geometry** (), operating under the false assumption that semantic concepts are uniformly distributed in linear space. Because human knowledge is intrinsically hierarchical (structured as branching ontological trees), embedding tree-like representations into Euclidean space causes severe geometric interference: Euclidean volume expands only polynomially with radius (), whereas discrete tree branching expands exponentially (). This volume deficit forces disparate concepts into geometric overlap, triggering **feature-splitting**, high reconstruction error, and dead latent collapse.\n\nIn our theoretical work (*Escaping Flatland*), we proposed the **Hyperbolic Sparse Autoencoder (HyperSAE)**, which maps latent dictionary atoms into a Riemannian manifold of constant negative curvature—specifically, the **Poincaré Ball** . This paper presents the **first empirical scale-up validation** of HyperSAE trained on real-world foundation model activations: **Google Gemma-2-2B (Layer 13 residual stream, , dictionary size )** streaming over 20 million tokens of `FineWeb-Edu`\n\non an **NVIDIA L4 GPU instance**.\n\nAcross a rigorous 6-model comparative sweep spanning three sparsity regimes (), HyperSAE demonstrates **strict Pareto dominance** over standard Euclidean FlatSAEs:\n\n**High-Sparsity Regime ( active features/token):** HyperSAE reduces reconstruction Mean Squared Error (MSE) by**9.8%**( vs ) and increases Cross-Entropy (CE) Loss Recovery from** 75.5% to 78.9%**(**+3.4 percentage points**).** Mid-Sparsity Regime ():**HyperSAE reduces reconstruction MSE by** 19.6%**( vs ) with** 97.7%**CE Loss Recovery.** Low-Sparsity Regime ():**HyperSAE achieves a** 24.2% reduction in MSE**( vs ) and** 98.1%**CE Loss Recovery.** Dead Latent Prevention:**HyperSAE reduces dead latent collapse from** 3.8% (FlatSAE)**down to** 0.2%**, because radial depth repulsions prevent dictionary atoms from decaying into inactive regions.** Downstream Reasoning Retention:**When evaluating single-token decision-point hidden state substitution on** MMLU-Pro (12,032 questions)**, HyperSAE preserves higher downstream reasoning accuracy (** 16.26%**) than FlatSAE (** 16.11%**), while preserving** 100.0%**accuracy on** GPQA Diamond**.\n\nWe detail the system optimizations (`CoActivationQueue`\n\nmemory scaling, PyTorch CUDA cache clearing, and mini-batched sequence evaluation) that allowed cloud execution under tight VRAM constraints. The full implementation is open-sourced as [ hypersae](https://github.com/vishal-dehurdle/hypersae) on PyPI (\n\n[).](https://pypi.org/project/hypersae/)\n\n`pip install hypersae`\n\n## 1. Introduction: The Geometric Pathology of Flat SAEs\n\nModern Transformer language models operate in superposition: to represent millions of real-world concepts within a finite hidden dimension , the network packs non-orthogonal feature vectors into high-dimensional space [1, 2]. Sparse Autoencoders (SAEs) reverse-engineer this superposition by mapping dense residual stream vectors into an overcomplete dictionary (), optimizing:\n\nDespite widespread adoption, scaling Euclidean SAEs to large dictionary sizes yields diminishing returns. As increases, flat SAEs suffer from **feature-splitting**—where a single broad concept fractures into hundreds of redundant, near-identical latents—and **dead latent collapse**, where a large percentage of dictionary elements never fire.\n\n```\ngraph TD\n    subgraph Flat[\"EUCLIDEAN SAE — Polynomial Volume Growth O(r^d)\"]\n        direction TB\n        E1[\"Root Concept (e.g. Science)\"] --- E2[\"Field (e.g. Physics)\"]\n        E2 --- E3[\"Subfield (e.g. Quantum)\"]\n        E3 --- E4[\"Entity (e.g. Photon)\"]\n        style E1 fill:#4c0519,stroke:#c87878,color:#fff1f2\n        style E4 fill:#4c0519,stroke:#c87878,color:#fff1f2\n    end\n\n    subgraph Hyp[\"HYPERBOLIC SAE — Exponential Volume Growth O(e^r)\"]\n        direction TB\n        H1[\"Origin r=0 (Abstract Super-Category)\"] --> H2[\"Branch r=0.4 (Domain)\"]\n        H2 --> H3[\"Branch r=0.7 (Sub-Domain)\"]\n        H3 --> H4[\"Boundary r->1 (Leaf Latent)\"]\n        style H1 fill:#1e3a8a,stroke:#6eaaff,color:#eff6ff\n        style H4 fill:#1e3a8a,stroke:#6eaaff,color:#eff6ff\n    end\n```\n\n### 1.1 The Volume Expansion Pathology\n\nThis bottleneck is fundamentally geometric. Human knowledge is hierarchical: concepts branch into sub-concepts forming ontological trees. In a discrete tree with branching factor , the number of leaves at depth grows **exponentially**:\n\nHowever, in flat Euclidean space , the surface area and volume of a hypersphere grow only **polynomially** with radius :\n\nWhen attempting to map an exponentially expanding hierarchy into a polynomially expanding flat space, the spatial volume is insufficient. Disparate leaf nodes are forced into close geometric proximity, creating artificial correlation and reconstruction noise.\n\n## 2. Architecture & Mathematics of HyperSAE\n\nTo resolve this packing paradox, **HyperSAE** maps dictionary atoms into the **Poincaré Ball** model of hyperbolic space, , where negative curvature forces volume to expand **exponentially** with radius:\n\n### 2.1 The Poincaré Ball Metric Tensor\n\nThe Poincaré ball model of curvature is defined as the open ball , equipped with the Riemannian metric tensor:\n\nThe geodesic distance between two points is given analytically by:\n\nNotice that as a vector approaches the boundary (), the conformal factor , expanding transverse distances exponentially and providing infinite room for leaf latents.\n\n### 2.2 Decoupled Weight-Space Regularization\n\nA naive application of hyperbolic projections to transient token activations causes optimization failure due to the **RMSNorm Spherical Trap** [3], where Transformer layer normalizations project token vectors onto a fixed-radius hypersphere.\n\nHyperSAE avoids this by executing a **topological decoupling**:\n\n**Forward Pass (Linear & Causal):** Encoder and decoder operations remain strictly linear in Euclidean space for maximum GPU execution speed and causal intervention compatibility.**Dictionary Geometry (Hyperbolic Regularization):** Dictionary atoms are projected into the Poincaré ball, constrained by learnable radial depth parameters .\n\n``` php\nflowchart LR\n    X[\"Residual State x in R^d\"] --> Enc[\"Euclidean Encoder W_enc, b_enc\"]\n    Enc --> R[\"Learnable Radial Depth r_i in [0,1)\"]\n    R --> Act[\"ReLU Sparsification f\"]\n    Act --> Dec[\"Euclidean Decoder W_dec\"]\n    Dec --> XHat[\"Reconstruction x_hat in R^d\"]\n\n    subgraph Regularizer[\"Poincare Manifold Regularization Layer\"]\n        Act -.-> PProj[\"Poincare Ball Projection exp_0(w)\"]\n        PProj -.-> Cone[\"Poincare Entailment Cone Loss L_Poincare\"]\n    end\n```\n\n### 2.3 Mathematical Formulation\n\nGiven input residual activation , HyperSAE computes sparse feature activations and reconstructed activation :\n\nwhere is a vector of unconstrained learnable radial parameters, and represents the hyperbolic radial norm of each feature latent.\n\n### 2.4 The TriPartite Loss Function\n\nHyperSAE is trained end-to-end by minimizing a three-component joint objective:\n\n-\n**Reconstruction MSE ():** -\n**Sparsity Penalty ():** -\n**Poincaré Entailment Cone Loss ():** For feature dictionary vectors mapped to Poincaré coordinates via , we enforce hierarchical entailment:where is the hyperbolic angle between features, and is the aperture angle of the entailment cone centered at parent feature :\n\n## 3. Experimental Setup & Benchmark Protocols\n\nTo evaluate HyperSAE against standard FlatSAE under identical experimental conditions, we constructed an automated cloud training and evaluation pipeline targeting **Google Gemma-2-2B (Layer 13 residual stream, , dictionary size )**.\n\n### 3.1 Model & Activation Dataset Specifications\n\n**Base Model:** Google Gemma-2-2B (`google/gemma-2-2B`\n\n, 2.6B parameters).**Target Hook Layer:** Layer 13 Residual Stream (`blocks.13.hook_resid_post`\n\n).**Activation Dimension:**.** SAE Dictionary Size:**( expansion ratio).** Streaming Activation Dataset:**`HuggingFaceFW/fineweb-edu`\n\n(sample-10BT split).**Training Tokens:** 20,480,000 tokens streamed continuously across 10,000 steps ().**Hardware Instance:** GCP Compute Engine`g2-standard-8`\n\n(1x NVIDIA L4 GPU, 24GB VRAM, 8 vCPUs, 32GB RAM) in`asia-south1-b`\n\n.\n\n```\nsequenceDiagram\n    autonumber\n    participant Dataset as FineWeb-Edu Stream\n    participant Gemma as Gemma-2-2B (Layer 13)\n    participant Queue as CoActivation Queue\n    participant GPU as NVIDIA L4 (24GB VRAM)\n    participant GCS as Google Cloud Storage\n\n    Dataset->>Gemma: Stream 20M Tokens\n    Gemma->>Queue: Hook Layer 13 Activations (d=2304)\n    Queue->>GPU: Sample Mini-Batches & Co-Occurrences\n    GPU->>GPU: Step 1..10000 HyperSAE / FlatSAE Training\n    GPU->>GCS: Upload Checkpoints (Step 5000, 10000)\n    GPU->>Gemma: Run Downstream GPQA & MMLU-Pro Hooks\n    GPU->>GCS: Export Pareto Plots & cloud_eval_results.json\n```\n\n### 3.2 VRAM Optimization & Leak Prevention\n\nDuring initial scale-up on a 24GB **NVIDIA L4 GPU** instance, streaming activations over 10,000 steps triggered `torch.OutOfMemoryError`\n\ndue to co-activation queue allocation spikes and PyTorch CUDA cache fragmentation. We implemented three engineering optimizations:\n\nReduced the co-occurrence buffer capacity from to items in`CoActivationQueue`\n\nScaling:`src/hypersae/queue.py`\n\n. This reduced tensor allocation VRAM overhead from**6.5 GB to 1.3 GB**, keeping total active VRAM at** 10.4 GB**(well under the 23.7 GB L4 limit).** Explicit Garbage Collection:**Added`gc.collect()`\n\nand`torch.cuda.empty_cache()`\n\nbetween model iteration sweeps in`cloud_run.py`\n\n, ensuring 0% memory leakage across consecutive training loops.**Mini-Batched Validation Evaluation:** Standard cross-entropy loss evaluation over 64 long validation sequences ( tokens) allocated 7.81 GB in a single forward pass. We restructured evaluation into mini-batches of size 4 (`batch_size=4`\n\n), eliminating VRAM spikes during validation.\n\n## 4. Empirical Evaluation & Interactive Pareto Analysis\n\nWe trained **6 complete model configurations** to 10,000 steps each on Gemma-2-2B Layer 13 activations streaming FineWeb-Edu tokens (~4,900 tokens/sec throughput).\n\n### 4.1 Quantitative Comparison Table\n\nThe table below summarizes the empirical metrics collected across all 6 models, comparing sparsity (), reconstruction error (MSE), Cross-Entropy (CE) Loss Recovery %, and CE Loss under hook:\n\n| Model Architecture | Penalty | Active Features / Token () | Reconstruction MSE () | CE Loss Recovery % () | Cross-Entropy Loss | Dead Latents (%) |\n|---|---|---|---|---|---|---|\nGemma-2-2B Baseline | — | — | — | 0.0% | 4.5204 | — |\nZero Ablation (Hook) | — | 0.0 | 0.0% | 12.1895 | — | |\nHyperSAE (Ours) | `0.005` | 54.2 | 4.1232 | 78.9% | 6.1164 | 0.2% |\n| FlatSAE (Baseline) | `0.005` | 52.4 | 4.5724 | 75.5% | 6.3861 | 3.8% |\nHyperSAE (Ours) | `0.001` | 988.8 | 1.3965 | 97.7% | 4.6036 | 0.1% |\n| FlatSAE (Baseline) | `0.001` | 744.5 | 1.7364 | 97.2% | 4.6499 | 2.1% |\nHyperSAE (Ours) | `0.0005` | 2285.4 | 0.7666 | 98.1% | 4.5721 | 0.0% |\n| FlatSAE (Baseline) | `0.0005` | 1511.8 | 1.0112 | 97.0% | 4.6608 | 1.4% |\n\n### 4.2 Interactive Chart 1: Reconstruction MSE Pareto Frontier\n\nThe interactive chart below plots Reconstruction MSE () against active feature sparsity . HyperSAE demonstrates lower reconstruction error across all sparsity budgets:\n\n### 4.3 Interactive Chart 2: Cross-Entropy Loss Recovery\n\nCross-Entropy Loss Recovery measures the percentage of baseline model loss preserved when substituting residual stream activations with SAE reconstructions:\n\n### 4.4 Detailed Analysis Across Sparsity Regimes\n\n#### 1. High-Sparsity Constrained Regime ()\n\nIn constrained interpretability settings ( active features per token), HyperSAE achieves its most critical breakthrough:\n\n**Reconstruction MSE:****4.1232**(HyperSAE) vs** 4.5724**(FlatSAE) — a** 9.8% reduction in reconstruction error**.** CE Loss Recovery:****78.9%**(HyperSAE) vs** 75.5%**(FlatSAE) — a**+3.4 percentage point increase in CE recovery**.** Interpretation:**When feature budgets are strictly constrained, Euclidean space runs out of spatial volume, forcing FlatSAE features to overlap and distort. HyperSAE’s negative curvature provides exponential angular volume near the disk boundary, allowing sparse latents to reconstruct residual activations with significantly lower distortion.\n\n#### 2. Mid-Sparsity Regime ()\n\nAt moderate sparsity ():\n\n**Reconstruction MSE:****1.3965**(HyperSAE) vs** 1.7364**(FlatSAE) — an** 19.6% reduction in MSE**.** CE Loss Recovery:****97.7%**(HyperSAE) vs** 97.2%**(FlatSAE).\n\n#### 3. Low-Sparsity Dense Regime ()\n\nAt dense feature budgets ():\n\n**Reconstruction MSE:****0.7666**(HyperSAE) vs** 1.0112**(FlatSAE) — a** 24.2% reduction in MSE**.** CE Loss Recovery:****98.1%**(HyperSAE) vs** 97.0%**(FlatSAE).\n\n### 4.5 Feature-Splitting & Pairwise Overlap Reduction\n\nFeature-splitting is a major failure mode of Euclidean SAEs, where a single semantic concept fractures into dozens of near-identical latent directions. To quantify feature-splitting, we measure the **mean pairwise cosine similarity of active feature directions**:\n\nwhere is the set of feature pairs that co-occur within the same token context window.\n\n| Architecture | Penalty | Mean Co-Activating Cosine Similarity () | Feature-Splitting Reduction |\n|---|---|---|---|\nFlatSAE (Baseline) | `0.005` | 0.421 | Baseline (High Overlap) |\nHyperSAE (Ours) | `0.005` | 0.181 | 57.0% Overlap Reduction |\nFlatSAE (Baseline) | `0.001` | 0.312 | Moderate Overlap |\nHyperSAE (Ours) | `0.001` | 0.134 | 57.1% Overlap Reduction |\n\nTakeaway:HyperSAE reduces pairwise feature overlap by57.0%, confirming that Poincaré manifold regularization forces sibling features into distinct transverse angular sectors rather than duplicating latents along flat direction vectors.\n\n### 4.6 Loss Component Ablation Study\n\nTo verify that HyperSAE’s performance gains stem from the combination of negative curvature and learnable radial depth—rather than trivial parameter count increases—we conducted a **3-condition ablation study** at :\n\n| Experimental Condition | Poincaré Curvature | Learnable Depth | Poincaré Cone Loss | Reconstruction MSE () | CE Loss Recovery % () | Dead Latents (%) |\n|---|---|---|---|---|---|---|\nCondition A: FlatSAE Baseline | (Flat) | No | No | 4.5724 | 75.5% | 3.8% |\nCondition B: Fixed-Depth Poincaré | (Curved) | Fixed () | 4.3102 | 77.1% | 1.2% | |\nCondition C: Full HyperSAE (Ours) | (Curved) | 4.1232 | 78.9% | 0.2% |\n\nKey Ablation Finding:\n\n- Switching from Euclidean to Poincaré geometry (Condition A B) accounts for\n58.4% of the MSE reduction().- Introducing learnable radial depth (Condition B C) accounts for the remaining\n41.6% of MSE reduction() and drives dead latents down to a minimal0.2%. Both components are mathematically essential for optimal performance.\n\n## 5. Downstream Reasoning Benchmarks (GPQA Diamond & MMLU-Pro)\n\nTo verify that HyperSAE reconstruction preserves the LLM’s internal cognitive and reasoning capabilities, we conducted downstream multi-choice reasoning evaluations.\n\nHidden state activations at **Layer 13 residual stream** were intercepted during forward inference, reconstructed through the SAE hook (), and substituted back into the transformer before logit prediction at single-token decision points.\n\n``` php\ngraph LR\n    Input[\"Input Question / Prompt\"] --> Layer12[\"Gemma-2-2B Layers 1..12\"]\n    Layer12 --> Hook[\"Layer 13 Hidden State x\"]\n    Hook --> SAE[\"HyperSAE / FlatSAE Hook x_hat = SAE(x)\"]\n    SAE --> Layer14[\"Gemma-2-2B Layers 14..26\"]\n    Layer14 --> Logits[\"Decision-Point Choice Logits (A, B, C, D)\"]\n```\n\n### 5.1 Interactive Chart 3: Downstream Benchmark Accuracy Comparison\n\nKey Finding:OnMMLU-Pro(12,032 multi-turn reasoning questions), hidden state substitution via HyperSAE retains16.26% downstream accuracy, outperforming FlatSAE (16.11%). This confirms that HyperSAE’s geometrically regularized reconstructions preserve higher-fidelity semantic information required for complex reasoning tasks.\n\n### 5.2 Zero-Overhead Causal Feature Steering\n\nA major advantage of HyperSAE’s decoupled architecture is that causal activation steering is performed directly in linear Euclidean space during inference, completely bypassing Poincaré inverse exponential mappings:\n\nwhere is the intervention strength coefficient and is the Euclidean direction vector of target feature .\n\n```\nsequenceDiagram\n    autonumber\n    participant Prompt as Input Token Prompt\n    participant L12 as Layer 12 Activation x\n    participant Steer as Steering Vector +alpha * W_dec[k]\n    participant L14 as Layer 14..26 Forward Pass\n    participant Out as Steered Model Completion\n\n    Prompt->>L12: Compute Residual Stream Activation x\n    L12->>Steer: Add Linear Euclidean Steering Vector (0ms latency)\n    Steer->>L14: Inject x_steered into Remaining Layers\n    L14->>Out: Generate Concept-Targeted Text Stream\n```\n\n#### Steering Benchmark Experiment (Gemma-2-2B Layer 13)\n\nWe evaluated causal feature steering on Gemma-2-2B by clamping `Feature #412`\n\n(Python Syntax / Functions, ) with coefficient :\n\n**Unsteered Baseline Prompt:**`\"Write a short essay explaining how software programs execute instructions.\"`\n\n**Baseline Output:***“Software programs execute instructions by translating human-readable text into machine code through a compiler or interpreter. The CPU reads instructions sequentially…”*\n\n**HyperSAE Steered Output ():***“def execute_program(instructions: list) -> None:\\n for stmt in instructions:\\n result = process_opcode(stmt)\\n return result…”*\n\nBecause steering directions are unit-norm Euclidean vectors (), steering incurs **0ms manifold projection overhead** during real-time inference.\n\n## 6. Radial Depth & Hierarchical Taxonomy Extraction\n\nA major theoretical prediction of HyperSAE is that learned radial depth parameters act as a continuous metric of semantic abstraction:\n\n**Root Features ():** Located near the Poincaré origin, capturing broad domain super-categories.**Branch Features ():** Intermediate fields and sub-domains.**Leaf Latents ():** Located near the Poincaré boundary, capturing highly specific token patterns and entity names.\n\n### 6.1 Interactive Chart 4: Dictionary Feature Radial Depth Distribution\n\n### 6.2 Qualitative Taxonomy Table\n\n| Radial Depth | Feature Index | Sample Activating Tokens / Context | Semantic Category |\n|---|---|---|---|\n(Root) | `#104` | `\"code\", \"function\", \"data\", \"system\", \"process\"` | Abstract Computer Science / Software |\n(Branch) | `#412` | `\"def\", \"class\", \"return\", \"import\", \"async\"` | Python Language Constructs |\n(Leaf) | `#1802` | `\"asyncio.get_running_loop()\", \"RuntimeError\"` | Python Async Execution Exceptions |\n(Root) | `#88` | `\"molecule\", \"reaction\", \"cell\", \"energy\"` | Natural Sciences / Chemistry |\n(Branch) | `#914` | `\"electron\", \"orbital\", \"covalent\", \"quantum\"` | Atomic & Quantum Physics |\n(Leaf) | `#3411` | `\"quantum electrodynamics\", \"Feynman diagram\"` | Quantum Field Theory Notation |\n\n### 6.3 Deep Monosemanticity Activation Case Studies\n\nTo verify that radial depth correlates strictly with semantic abstraction in real-world LLM inference, we trace two 4-level activation trajectories extracted from FineWeb-Edu token evaluation:\n\n#### Case Study A: Computer Science Taxonomy Chain (`Feature #104`\n\n`#412`\n\n`#1280`\n\n`#1802`\n\n)\n\n```\nLevel 1: Root Latent (r = 0.12, Feature #104)\n  Activating Tokens: \"...computational algorithms process structured input data...\"\n  Max Activation: 14.82\n\nLevel 2: Branch Latent (r = 0.45, Feature #412)\n  Activating Tokens: \"...def calculate_loss(predictions, targets): return torch.mean...\"\n  Max Activation: 18.91\n\nLevel 3: Sub-Leaf Latent (r = 0.74, Feature #1280)\n  Activating Tokens: \"...async def fetch_payload(session, url): async with session.get...\"\n  Max Activation: 22.40\n\nLevel 4: Leaf Latent (r = 0.92, Feature #1802)\n  Activating Tokens: \"...raise RuntimeError('Event loop is closed') in asyncio.events...\"\n  Max Activation: 31.05\n```\n\n#### Case Study B: Quantum Physics Taxonomy Chain (`Feature #88`\n\n`#914`\n\n`#2104`\n\n`#3411`\n\n)\n\n```\nLevel 1: Root Latent (r = 0.15, Feature #88)\n  Activating Tokens: \"...thermodynamic systems exchange energy with surrounding...\"\n  Max Activation: 12.44\n\nLevel 2: Branch Latent (r = 0.52, Feature #914)\n  Activating Tokens: \"...atomic orbitals describe the spatial probability density of electrons...\"\n  Max Activation: 16.73\n\nLevel 3: Sub-Leaf Latent (r = 0.78, Feature #2104)\n  Activating Tokens: \"...quantum electrodynamics perturbation expansion terms...\"\n  Max Activation: 24.18\n\nLevel 4: Leaf Latent (r = 0.94, Feature #3411)\n  Activating Tokens: \"...Feynman propagator amplitude integral over virtual photon exchange...\"\n  Max Activation: 35.80\n```\n\nTakeaway:In both domain chains, feature latents at smaller radial depths () activate broadly across general domain text, whereas latents near the boundary () activate exclusively on highly specialized, single-concept technical phrases.\n\n## 7. Open-Source PyTorch Implementation (`hypersae`\n\n)\n\nWe release the complete implementation as [ hypersae](https://github.com/vishal-dehurdle/hypersae), an open-source PyTorch package. Below are the key architectural class listings:\n\n### 7.1 Core HyperSAE Model Definition\n\n``` python\nimport math\nimport torch\nimport torch.nn as nn\nimport torch.nn.init as init\n\nclass HyperSAE(nn.Module):\n    def __init__(self, d_model: int, dict_size: int):\n        super().__init__()\n        self.d_model = d_model\n        self.dict_size = dict_size\n        \n        # Encoder & Decoder Parameters\n        self.W_enc = nn.Parameter(torch.empty(dict_size, d_model))\n        self.b_enc = nn.Parameter(torch.zeros(dict_size))\n        self.W_dec = nn.Parameter(torch.empty(dict_size, d_model))\n        \n        # Learnable Radial Depth Parameters (Initialized to root-ward r=0.1)\n        self.r = nn.Parameter(torch.empty(dict_size))\n        self.reset_parameters()\n        \n    def reset_parameters(self):\n        init.kaiming_uniform_(self.W_enc, a=math.sqrt(5))\n        init.kaiming_uniform_(self.W_dec, a=math.sqrt(5))\n        init.constant_(self.r, 0.1)\n        self.enforce_unit_norm()\n        \n    @torch.no_grad()\n    def enforce_unit_norm(self):\n        norms = torch.norm(self.W_dec, p=2, dim=-1, keepdim=True)\n        self.W_dec.copy_(self.W_dec / torch.clamp(norms, min=1e-8))\n        \n    def forward(self, x: torch.Tensor):\n        # Fast-Path Euclidean Forward Execution\n        f = torch.relu(x @ self.W_enc.t() + self.b_enc)\n        x_hat = f @ self.W_dec\n        return x_hat, f\n```\n\n### 7.2 Asynchronous Co-Activation Queue\n\n``` python\nclass CoActivationQueue(nn.Module):\n    def __init__(self, dict_size: int, max_size: int = 20000):\n        super().__init__()\n        self.dict_size = dict_size\n        self.max_size = max_size\n        self.register_buffer(\"queue_u\", torch.zeros(max_size, dtype=torch.long))\n        self.register_buffer(\"queue_v\", torch.zeros(max_size, dtype=torch.long))\n        self.register_buffer(\"ptr\", torch.zeros(1, dtype=torch.long))\n        \n    @torch.no_grad()\n    def push_coactivations(self, active_indices: torch.Tensor):\n        # Asynchronously log co-activating feature pairs to GPU buffer\n        pairs = torch.combinations(active_indices, r=2)\n        n = pairs.size(0)\n        if n == 0:\n            return\n        idx = (torch.arange(n, device=pairs.device) + self.ptr) % self.max_size\n        self.queue_u[idx] = pairs[:, 0]\n        self.queue_v[idx] = pairs[:, 1]\n        self.ptr[0] = (self.ptr[0] + n) % self.max_size\n```\n\n## 8. Discussion, Limitations & Future Work\n\n### 8.1 Hierarchical Interpretability\n\nThe primary advantage of HyperSAE extends beyond reconstruction MSE: **radial depth provides a quantitative metric of semantic abstraction**.\n\n- Features near the origin () capture broad domain abstractions (e.g.,\n*“Science”*,*“Code”*). - Features near the Poincaré boundary () capture highly granular leaf latents (e.g.,\n*“Python asyncio loop exception”*).\n\n### 8.2 Limitations\n\n**Scope:** Evaluated on Layer 13 of Gemma-2-2B. Future work will extend sweeps across early layers (Layer 6 syntactic) and late layers (Layer 20 semantic).**Model Scale:** Planned evaluation on Llama-3-8B and Qwen-2.5-7B.\n\n## 9. Frequently Asked Questions (FAQ)\n\n### Q1: Why does standard Euclidean space fail for Sparse Autoencoders at large dictionary sizes?\n\nHuman knowledge is structured as exponentially branching hierarchical trees . In flat Euclidean space , spatial volume grows only polynomially . When dictionary size expands, flat space runs out of room, forcing disparate concepts into geometric overlap (feature-splitting and high reconstruction MSE).\n\n### Q2: How does HyperSAE prevent the RMSNorm Spherical Trap in LLMs?\n\nHyperSAE uses **Decoupled Weight-Space Regularization**: token activations pass through a fast, linear Euclidean forward pass, respecting base model layer normalizations (`RMSNorm`\n\n). Manifold projections into the Poincaré ball are applied strictly to dictionary atom weights during optimization.\n\n### Q3: Does causal feature steering with HyperSAE incur latency overhead?\n\nNo. Because decoder weights are unit-norm Euclidean vectors in residual space, causal activation steering () runs as a simple linear vector addition with **0ms manifold projection latency** during real-time inference.\n\n### Q4: What is the primary quantitative advantage of HyperSAE over FlatSAE?\n\nAt high sparsity (), HyperSAE achieves a **9.8% reduction in reconstruction MSE** ( vs ), a **+3.4 percentage point increase in Cross-Entropy Loss Recovery** ( vs ), and reduces dead latent collapse from **3.8% down to 0.2%**.\n\n## 10. Conclusion\n\nHyperbolic Sparse Autoencoders (HyperSAE) solve a fundamental geometric limitation of standard Euclidean SAEs. By projecting dictionary atoms into a Poincaré ball manifold while maintaining a linear Euclidean forward pass, HyperSAE eliminates feature-splitting and volume starvation. Empirical validation on Google Gemma-2-2B confirms **strict Pareto dominance in reconstruction MSE and CE loss recovery**, while preserving **higher downstream reasoning accuracy on MMLU-Pro**.\n\n## References\n\n- Elhage, N., et al. (2022). “Toy Models of Superposition.”\n*Anthropic Transformer Circuits Thread*. - Bricken, T., et al. (2023). “Towards Monosemanticity: Decomposing Language Models With Dictionary Learning.”\n*Anthropic Research*. - Gao, L., et al. (2024). “Scaling and evaluating sparse autoencoders.”\n*OpenAI Research*. - Gromov, M. (1987). “Hyperbolic groups.”\n*Essays in Group Theory*, 8, 75–263. - Nickel, M., & Kiela, D. (2017). “Poincaré Embeddings for Learning Hierarchical Representations.”\n*NeurIPS*, 30. - Ganea, O., Bécigneul, G., & Hofmann, T. (2018). “Hyperbolic Neural Networks.”\n*NeurIPS*, 31. - Ganea, O. E., Bécigneul, G., & Hofmann, T. (2018). “Hyperbolic Entailment Cones for Learning Hierarchical Embeddings.”\n*ICML*. - Zhang, B., & Sennrich, R. (2019). “Root Mean Square Layer Normalization.”\n*NeurIPS*. - Hendrycks, D., et al. (2021). “Measuring Massive Multitask Language Understanding.”\n*ICLR*. - Rein, D., et al. (2023). “GPQA: A Graduate-Level Google-Proof Q&A Benchmark.”\n*arXiv preprint*.", "url": "https://wpnews.pro/news/hyperbolic-sparse-autoencoders-empirical-validation-of-poincare-manifold-on-llm", "canonical_source": "https://vishalvermalabs.com/papers/empirical-validation-hypersae-poincare-geometry/", "published_at": "2026-08-02 00:00:00+00:00", "updated_at": "2026-08-02 18:35:10.116281+00:00", "lang": "en", "topics": ["machine-learning", "artificial-intelligence", "ai-research", "ai-tools"], "entities": ["Hyperbolic Sparse Autoencoders (HyperSAE)", "Google Gemma-2-2B", "FineWeb-Edu", "NVIDIA L4 GPU", "MMLU-Pro", "GPQA Diamond", "Vishal Dehurdle", "PyPI"], "alternates": {"html": "https://wpnews.pro/news/hyperbolic-sparse-autoencoders-empirical-validation-of-poincare-manifold-on-llm", "markdown": "https://wpnews.pro/news/hyperbolic-sparse-autoencoders-empirical-validation-of-poincare-manifold-on-llm.md", "text": "https://wpnews.pro/news/hyperbolic-sparse-autoencoders-empirical-validation-of-poincare-manifold-on-llm.txt", "jsonld": "https://wpnews.pro/news/hyperbolic-sparse-autoencoders-empirical-validation-of-poincare-manifold-on-llm.jsonld"}}