16 Days, 4.7M Params, Zero Black Boxes: Building a White-box Chinese Cognition Engine from Scratch

A developer built a white-box Chinese language engine from scratch in 16 days, designing each of its 4.7 million parameters to have a specific linguistic function and be fully traceable. The project overcame multiple training collapses including mode collapse, repetition collapse, and a gradient chain break that froze the gate for 240 epochs, using a dual-agent debugging system with DeepSeek and Qwen.

Author: Wei Jinqi | June 16, 2026 Every time I use a large language model, the same thought nags at me: I have no idea what's happening inside. 95% accuracy? Great. But which weights fired? What linguistic features were extracted? Did it confuse "bank" river with "bank" financial ? Nobody knows. So I spent 16 days building a Chinese language engine where every weight has a reason and every decision is traceable . Instead of training a transformer on terabytes of text and hoping it learns Chinese, I designed each module to handle a specific linguistic function : | Module | Function | Params | |---|---|---| | P1 | Char → Word encoding | 96K frozen | | P3-L | Multi-dimensional attribute annotation | 0 rule engine | | P7 | Cross-sentence word routing | 226K | | Explore+Meta | Learned gating over decode dims | 101K | | P6 | Sentence → Word sequence decoding | 4.37M | The modules are chained: P1 encodes → P7 routes → Gate modulates → P6 decodes . Every intermediate state can be inspected. Day 1 was smooth. P1 char→word encoder and P3 attribute stack — a rule engine that tags words with person/syntax/semantic/emotion/direction attributes came together quickly. Day 2 introduced P7, the cross-sentence router. And everything broke . I used standard multi-head cross-attention. Every position — regardless of input — routed to the same output word. The dreaded Mode Collapse . What followed was seven failed fixes: The breakthrough came when I noticed Q/K were eye-initialized, meaning each head saw only 1 dimension with zero discrimination power. v8 final : Xavier init for Q/K, eye init for V. Added an Explore network loss → GELU MLP → 64D control signal and a Meta network signal + state → per-word gate . Mode collapse solved. Day 3 built P3-L: 23 groups, 312 independent attention heads, each controlling one attribute dimension. Combined training with P7 via UnifiedExplore→UnifiedMeta gate. Day 4 introduced P6: the sentence→word decoder . It was supposed to take a 256D sentence vector and output 16 distinct word embeddings. It output the same word 16 times. The Repetition Collapse had begun. Six versions over two days: h + pos embed i per head → The simplest fix won. Each head receives the same h but adds a unique learned position embedding. No rep pen. No residuals. No detach. Just position diversity. Epoch after epoch, the gate stayed frozen — all 256 dimensions had std=0.0001 . Three bugs conspired: explore mod.weight zero-initialized → identical signal per dim p3l act zero-initialized → sigmoid 0 =0.5 for all dims bias init scale=0.1 too small → output stuck at 0.5Then I found an even worse bug: gate.item was used in loss computation, converting a tensor to Python float — severing the gradient chain . The gate had been frozen for 240 epochs without anyone noticing. Fix: keep gate as tensor, let gradients flow back through explore and meta. Loss dropped from 0.56 to 0.28 in 3 epochs. I built a dual-agent debugging system: DeepSeek engineer proposes fixes, Qwen reviewer audits them. They debate until convergence. The system diagnosed four major bugs, including the gradient chain break. It would have saved days if I'd built it earlier. Key improvements: ord c 32 filter | Bug | Symptom | Root Cause | Fix | |---|---|---|---| | Mode Collapse | All outputs = same word | Q/K eye-init, zero discrimination | Xavier init + diversity architecture | | Gate Symmetry Lock | All gate dims identical std=0.0001 | Three zero-initializations | Proper random init for explore, act, bias | | Gradient Chain Break | Gate not learning for 240 epochs | .item severed gradient | Keep as tensor | | Repetition Collapse | 16 heads → same word | Parallel heads share identical input | Position embedding V6 | | CUDA OOM | 25.76 GiB allocated | P1 full cross-attention | Batch encoding 50 words | | Space Collapse | Model outputs spaces | HF data formatting | ord c 32 filter | | sent vec Info Loss | Different sentences → similar vectors | Mean pooling | Learnable ±weighted sum | | Metric | Score | |---|---| | Word Accuracy | 92.4% | | Exact Match | 76.3% | | Rouge-L F1 | 93.2 | | Per-word Cosine | 0.96 | | Speed | 14ms/sent 71 sent/s | Epoch 1 from scratch, no pretraining : 43.5% word accuracy on held-out exam set. Target: 95% after 1000 epochs. LLMs are powerful but opaque. When GPT makes a mistake, you can't trace which neurons fired wrong. With V19, you can: This isn't about beating GPT. It's about building something you can understand completely . git clone https://github.com/Xuan-yi-yan/V18-cognitive-architecture cd V18-cognitive-architecture python download public data.py python train v19 full.py --data public --epochs 1000 --display 10 Full model card and architecture docs on Hugging Face https://huggingface.co/ . 16 days. 7 dead bugs. 4.7 million parameters. Zero black boxes. That's just how I like it.