{"slug": "negative-squaring-pre-tilted-3-bit-quantization-beat-naive-4-bit", "title": "Negative squaring – pre-tilted 3-bit quantization beat naive 4-bit", "summary": "Researchers introduced negative squaring, a pre-tilting technique that adjusts weights against expected rounding errors before quantization, achieving 77% error reduction in a toy recurrent network with 3-bit quantization outperforming naive 4-bit. The method clips tilts to half a quantization step to preserve reasoning, but scaling to real LLMs remains unsolved.", "body_md": "Code accompanying the writeup \"Negative Squaring: Pre-Tilting Weights to\nPreserve Reasoning in Quantized Models\" (July 2026). See\n`negative-squaring-paper_1.md`\n\nin this repository for the full plain-language paper.\n\nBefore quantizing a model, tilt each weight *against* the error the rounding\nwill cause across the model's whole multi-step reasoning trajectory — clipped\nto half a quantization step, so the tilt only decides which way borderline\nweights round.\n\n| File | What it does | Key result |\n|---|---|---|\n`1_first_experiment.py` |\nRandom-search pre-tilt vs naive 4-bit quantization on a 12-layer, 30-step recurrent toy network | ~18% trajectory error removed; decision flips 14/20 → 8/20 |\n`2_gradient_attempt.py` |\nStraight-through gradient search, unconstrained | Backfires — test error gets worse (documented negative result) |\n`3_final_with_clipping.py` |\nGradient + random + combo searches, with tilts clipped to half a quantization step | 77% error removed; decision flips 20/50 → 4-5/50 |\n\nRequires only Python 3 and numpy:\n\n```\npip install numpy\npython 3_final_with_clipping.py\n```\n\nEach script is self-contained, seeded, and reproduces the numbers in the\nwriteup. Runtime is seconds to a few minutes on any laptop. To reproduce the 3-bit and 2-bit cliff results, change `BITS = 4`\n\nto `3`\n\nor `2`\n\non line 4 of script 3.\n\n- Toy scale: ~49k weights, tanh recurrence, not a transformer.\n- The toy's dynamics dampen errors; real LLMs often amplify them. Untested there.\n- Full-trajectory backprop is expensive at real scale; the clipping constraint shrinks the search space (only near-boundary weights matter) but efficient scaling is unsolved.\n\nIf you have compute and want to try trajectory-aware rounding on a real\nsub-1B model, or you know prior literature that already does this\n(AdaRound optimizes rounding decisions per-layer; we're looking for\n*whole-trajectory* versions), please reach out in the thread or open an\nissue here.", "url": "https://wpnews.pro/news/negative-squaring-pre-tilted-3-bit-quantization-beat-naive-4-bit", "canonical_source": "https://github.com/elpinyeknom/negative-squaring-/tree/main", "published_at": "2026-07-10 12:09:42+00:00", "updated_at": "2026-07-10 12:35:34.020385+00:00", "lang": "en", "topics": ["machine-learning", "neural-networks", "ai-research"], "entities": [], "alternates": {"html": "https://wpnews.pro/news/negative-squaring-pre-tilted-3-bit-quantization-beat-naive-4-bit", "markdown": "https://wpnews.pro/news/negative-squaring-pre-tilted-3-bit-quantization-beat-naive-4-bit.md", "text": "https://wpnews.pro/news/negative-squaring-pre-tilted-3-bit-quantization-beat-naive-4-bit.txt", "jsonld": "https://wpnews.pro/news/negative-squaring-pre-tilted-3-bit-quantization-beat-naive-4-bit.jsonld"}}