# 16 Days, 4.7M Params, Zero Black Boxes: Building a White-box Chinese Cognition Engine from Scratch

> Source: <https://dev.to/xuanyiyan/16-days-47m-params-zero-black-boxes-building-a-white-box-chinese-cognition-engine-from-scratch-503m>
> Published: 2026-06-16 14:07:46+00:00

**Author: Wei Jinqi | June 16, 2026**

Every time I use a large language model, the same thought nags at me: *I have no idea what's happening inside.*

95% accuracy? Great. But which weights fired? What linguistic features were extracted? Did it confuse "bank" (river) with "bank" (financial)? Nobody knows.

So I spent 16 days building a Chinese language engine where **every weight has a reason and every decision is traceable**.

Instead of training a transformer on terabytes of text and hoping it learns Chinese, I designed each module to handle a **specific linguistic function**:

| Module | Function | Params |
|---|---|---|
| P1 | Char → Word encoding | 96K (frozen) |
| P3-L | Multi-dimensional attribute annotation | 0 (rule engine) |
| P7 | Cross-sentence word routing | 226K |
| Explore+Meta | Learned gating over decode dims | 101K |
| P6 | Sentence → Word sequence decoding | 4.37M |

The modules are chained: **P1 encodes → P7 routes → Gate modulates → P6 decodes**. Every intermediate state can be inspected.

Day 1 was smooth. P1 (char→word encoder) and P3 (attribute stack — a rule engine that tags words with person/syntax/semantic/emotion/direction attributes) came together quickly.

Day 2 introduced P7, the cross-sentence router. And **everything broke**.

I used standard multi-head cross-attention. Every position — regardless of input — routed to the same output word. The dreaded **Mode Collapse**.

What followed was seven failed fixes:

The breakthrough came when I noticed Q/K were eye-initialized, meaning each head saw only 1 dimension with zero discrimination power.

**v8 (final)**: Xavier init for Q/K, eye init for V. Added an Explore network (loss → GELU MLP → 64D control signal) and a Meta network (signal + state → per-word gate). Mode collapse solved.

Day 3 built P3-L: 23 groups, 312 independent attention heads, each controlling one attribute dimension. Combined training with P7 via UnifiedExplore→UnifiedMeta gate.

Day 4 introduced **P6: the sentence→word decoder**. It was supposed to take a 256D sentence vector and output 16 distinct word embeddings.

It output the same word 16 times. The **Repetition Collapse** had begun.

Six versions over two days:

`h + pos_embed[i]`

per head → The simplest fix won. Each head receives the same `h`

but adds a unique learned position embedding. No rep_pen. No residuals. No detach. Just position diversity.

Epoch after epoch, the gate stayed frozen — all 256 dimensions had **std=0.0001**. Three bugs conspired:

`explore_mod.weight`

zero-initialized → identical signal per dim`p3l_act`

zero-initialized → sigmoid(0)=0.5 for all dims`bias init scale=0.1`

too small → output stuck at 0.5Then I found an even worse bug: `gate.item()`

was used in loss computation, converting a tensor to Python float — **severing the gradient chain**. The gate had been frozen for **240 epochs** without anyone noticing.

Fix: keep gate as tensor, let gradients flow back through explore and meta. Loss dropped from 0.56 to 0.28 in 3 epochs.

I built a dual-agent debugging system: **DeepSeek (engineer)** proposes fixes, **Qwen (reviewer)** audits them. They debate until convergence.

The system diagnosed four major bugs, including the gradient chain break. It would have saved days if I'd built it earlier.

Key improvements:

`ord(c) > 32`

filter)| Bug | Symptom | Root Cause | Fix |
|---|---|---|---|
| Mode Collapse | All outputs = same word | Q/K eye-init, zero discrimination | Xavier init + diversity architecture |
| Gate Symmetry Lock | All gate dims identical (std=0.0001) | Three zero-initializations | Proper random init for explore, act, bias |
| Gradient Chain Break | Gate not learning for 240 epochs |
`.item()` severed gradient |
Keep as tensor |
| Repetition Collapse | 16 heads → same word | Parallel heads share identical input | Position embedding V6 |
| CUDA OOM | 25.76 GiB allocated | P1 full cross-attention | Batch encoding (50 words) |
| Space Collapse | Model outputs spaces | HF data formatting |
`ord(c) > 32` filter |
| sent_vec Info Loss | Different sentences → similar vectors | Mean pooling | Learnable ±weighted sum |

| Metric | Score |
|---|---|
| Word Accuracy | 92.4% |
| Exact Match | 76.3% |
| Rouge-L F1 | 93.2 |
| Per-word Cosine | 0.96 |
| Speed | 14ms/sent (71 sent/s) |

Epoch 1 (from scratch, no pretraining): **43.5%** word accuracy on held-out exam set. Target: >95% after 1000 epochs.

LLMs are powerful but opaque. When GPT makes a mistake, you can't trace which neurons fired wrong. With V19, you can:

This isn't about beating GPT. It's about building something **you can understand completely**.

```
git clone https://github.com/Xuan-yi-yan/V18-cognitive-architecture
cd V18-cognitive-architecture
python download_public_data.py
python train_v19_full.py --data public --epochs 1000 --display 10
```

Full model card and architecture docs on [Hugging Face](https://huggingface.co/).

*16 days. 7 dead bugs. 4.7 million parameters. Zero black boxes.*

*That's just how I like it.*