{"slug": "i-built-an-experimental-low-bit-llm-training-method-that-matched-fp32-perplexity", "title": "I built an experimental low bit LLM training method that matched FP32 perplexity at 53M parameters", "summary": "An independent researcher reports that CECP (Co-Evolving Compositional Pre-Training), an experimental compression-aware LLM training method, matched a dense FP32 baseline on TinyStories, reaching a mean validation perplexity of 17.986 ± 0.082 versus 18.146 ± 0.298 for FP32 across three training seeds at roughly 52.8M parameters and 2,500 optimizer steps on an NVIDIA Tesla T4. CECP uses an aggressively compressed representation for its projection weights with a nominal 1-bit-per-weight index payload plus supporting data, though the researcher notes it trains more slowly than dense models and has not been shown to beat BitNet, AQLM or optimized low-bit quantization at larger scales. The implementation remains private while the architecture is under development, with a research overview and benchmark screenshots published on Hugging Face and planned tests at larger model sizes and against stronger low-bit baselines.", "body_md": "Hey everyone!\n\nI’ve been working on an experimental LLM training architecture called **CECP (Co-Evolving Compositional Pre-Training)**, and I wanted to share some early results and get feedback from people working on quantization and model efficiency.\n\nThe idea came from a question I had a while ago:\n\n**What if we trained language models to adapt to extreme compression during training, instead of training them normally and quantizing them afterward?**\n\nThat’s essentially what CECP explores.\n\nThe goal is to minimize the performance degradation associated with low-bit representations, potentially making larger models more accessible on consumer hardware.\n\nI’ve been testing CECP against a conventional FP32 baseline using TinyStories.\n\nMy latest experiment used:\n\nApproximately 52.8M parameters\n\n2,500 optimizer steps\n\nThree independent training seeds\n\nAn NVIDIA Tesla T4\n\nValidation perplexity as the primary metric\n\n**Results (lower is better):**\n\n| Model | Mean Validation Perplexity | \n|---|---|\n| Dense FP32 | 18.146 ± 0.298 | \n| CECP-256 | 17.986 ± 0.082 | \n\nCECP achieved essentially the same validation perplexity as the full-precision baseline in this experiment.\n\nIt uses an aggressively compressed representation for its projection weights, with a nominal 1-bit-per-weight index payload plus additional supporting data.\n\nOne interesting observation is that CECP initially learns more slowly than dense FP32, but the validation curves converge as training progresses.\n\nI want to be clear that these are still early results.\n\nCECP currently requires more time to train than a conventional dense model, and these experiments are relatively small compared with modern LLM pretraining.\n\nI also haven’t demonstrated that CECP outperforms established methods like BitNet, AQLM, or optimized low-bit quantization at larger scales.\n\nThe results suggest that the approach is worth investigating further, but they don’t establish general superiority.\n\nI’m planning to test larger models, evaluate actual exported model sizes, and compare against stronger low-bit baselines.\n\nI’m also interested in whether compression-aware training becomes more effective as model size increases.\n\nFor now, I’ve published a research overview and benchmark screenshots on Hugging Face:\n\nThe complete implementation is private while the architecture is still under development, but I’m sharing the preliminary results and general approach.\n\nI’d genuinely appreciate feedback, particula\n\nrly from anyone who’s worked with quantization-aware training, low-bit representations, or model-compression research.\n\n**Does this look like a promising direction, and what would you consider the most important next benchmark?**", "url": "https://wpnews.pro/news/i-built-an-experimental-low-bit-llm-training-method-that-matched-fp32-perplexity", "canonical_source": "https://discuss.huggingface.co/t/i-built-an-experimental-low-bit-llm-training-method-that-matched-fp32-perplexity-at-53m-parameters/191004#post_1", "published_at": "2026-10-25 18:30:42+00:00", "updated_at": "2026-10-25 18:43:24.307927+00:00", "lang": "en", "topics": ["large-language-models", "machine-learning", "ai-research", "ai-infrastructure"], "entities": ["CECP", "TinyStories", "NVIDIA Tesla T4", "BitNet", "AQLM", "Hugging Face"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/i-built-an-experimental-low-bit-llm-training-method-that-matched-fp32-perplexity", "markdown": "https://wpnews.pro/news/i-built-an-experimental-low-bit-llm-training-method-that-matched-fp32-perplexity.md", "text": "https://wpnews.pro/news/i-built-an-experimental-low-bit-llm-training-method-that-matched-fp32-perplexity.txt", "jsonld": "https://wpnews.pro/news/i-built-an-experimental-low-bit-llm-training-method-that-matched-fp32-perplexity.jsonld"}}