# I built an experimental low bit LLM training method that matched FP32 perplexity at 53M parameters > Source: > Published: 2026-10-25 18:30:42+00:00 Hey everyone! I’ve been working on an experimental LLM training architecture called **CECP (Co-Evolving Compositional Pre-Training)**, and I wanted to share some early results and get feedback from people working on quantization and model efficiency. The idea came from a question I had a while ago: **What if we trained language models to adapt to extreme compression during training, instead of training them normally and quantizing them afterward?** That’s essentially what CECP explores. The goal is to minimize the performance degradation associated with low-bit representations, potentially making larger models more accessible on consumer hardware. I’ve been testing CECP against a conventional FP32 baseline using TinyStories. My latest experiment used: Approximately 52.8M parameters 2,500 optimizer steps Three independent training seeds An NVIDIA Tesla T4 Validation perplexity as the primary metric **Results (lower is better):** | Model | Mean Validation Perplexity | |---|---| | Dense FP32 | 18.146 ± 0.298 | | CECP-256 | 17.986 ± 0.082 | CECP achieved essentially the same validation perplexity as the full-precision baseline in this experiment. It uses an aggressively compressed representation for its projection weights, with a nominal 1-bit-per-weight index payload plus additional supporting data. One interesting observation is that CECP initially learns more slowly than dense FP32, but the validation curves converge as training progresses. I want to be clear that these are still early results. CECP currently requires more time to train than a conventional dense model, and these experiments are relatively small compared with modern LLM pretraining. I also haven’t demonstrated that CECP outperforms established methods like BitNet, AQLM, or optimized low-bit quantization at larger scales. The results suggest that the approach is worth investigating further, but they don’t establish general superiority. I’m planning to test larger models, evaluate actual exported model sizes, and compare against stronger low-bit baselines. I’m also interested in whether compression-aware training becomes more effective as model size increases. For now, I’ve published a research overview and benchmark screenshots on Hugging Face: The complete implementation is private while the architecture is still under development, but I’m sharing the preliminary results and general approach. I’d genuinely appreciate feedback, particula rly from anyone who’s worked with quantization-aware training, low-bit representations, or model-compression research. **Does this look like a promising direction, and what would you consider the most important next benchmark?**