Beyond Scaling Laws: Why Memory Is the LLM's Next Step (with a 3M-Token Recall Experiment)

In my view, scaling laws are not the only key factor behind model capability gains — even the most cutting-edge RL does not count as part of scaling laws.

Every leap in model capability is built on paradigm breakthroughs — on thinking about intelligence at a fundamental level. The Transformer brought the AI era; the LLM capability leap is largely built on CoT. If every model ran in non-thinking mode, could today's models outperform last year's models at the same point in time? Even for the same model with identical post-training, whether it thinks — and how hard it thinks — largely determines performance. And all of these gains rest on our own thinking about intelligence.

Both Transformer and CoT are imitations of higher human functions. The Transformer simulates human semantic understanding rather than being a probabilistic character-guessing machine: it is mathematical semantic understanding, not a mathematical probability game. Traced to its lowest level, the human brain also requires neural signals; the Transformer is nothing more than a silicon-based semantic understanding system. That is remarkable — but it absolutely does not prove that LLMs are greater than humans. The breakthroughs and achievements of LLMs exist essentially because they resemble humans and are striving to approach humans, this higher form of intelligence. CoT is better analogized to a higher human function: explicit logical reasoning. Note the qualifier — explicit logical reasoning only, not logical reasoning as a whole, nor the entirety of the brain's higher thinking. Humans also possess a great deal of implicit thinking and inspiration that emerges from internal parallel mechanisms.

So what I believe truly drives breakthroughs is not the increase of parameters, or not purely the increase of parameters. It is that once we ourselves understand intelligence better and thereby arrive at a better intelligence paradigm, scaling laws begin to take effect — rather than worshipping scaling laws in black-box fashion. If a non-thinking GPT-6 could beat DeepSeek V4.1 High, only then could you believe that AI is an alien mind and that scaling laws are the golden rule. Otherwise, AI is merely a low-level intelligence with powerful compute.

The relationship between every kind of AI and humans should be understood as this: God created Eve from Adam's rib, only because Adam was lonely.

In my view, the LLM's next step should be memory — giving LLMs a human-like memory mechanism instead of only an attention mechanism. Attention alone is incomplete, because the world's information is infinite while your attention window is always finite. This becomes especially acute as LLM output throughput keeps rising: if the LLM's attention can no longer control its own output, the project is bound to spiral out of control.

Humans face the same problem. Why can humans manage code — or exercise engineering control — through architectural awareness and other forms of global awareness? Because humans possess the higher function of state memory, or, put plainly, something in the back of your mind keeps you alert.

That higher function, plainly stated, is memory and recollection. Human memory is fundamentally built on forgetting: details are dropped, but the state at the time is retained. The state is fuzzy, yet the state can serve as meta-information for reasoning — it can become a reasoning head — helping a person advance to the next step. Producing a correct next step does not require the full history. It requires that a massive history be compressed into states, one state after another, each state becoming a reasoning head — like islands in a vast sea, letting your brain use island-hopping tactics, combining the currently traversed information with the key information recalled, to produce the correct next step.

That is how I understand it.

To validate this, I wrote a harness — agent-shell — and the following is my experimental data.

Subject under evaluation: the agent-shell context-engineering system as a whole (tiered compression + envelopes + tombstones + recall fallback), not a single recall component.

Core proposition: recall is the fallback for moments when summarized information is incomplete, not a mandatory step in every task — the evaluation measures two separate tracks: "what the compression main path carries" and "when the fallback fires."

Key readings:

The "depth decay" was an illusion. The 10%/30% bucket losses in r1–r3 correspond one-to-one (100%) with "envelope sections swallowed by the block chunker" (fence bug: mem- envelopes did not count as block boundaries + failed re-selection grew block spans → secondary compression destroyed neighboring envelopes). After the fix, the early buckets scored perfectly, and the 90% bucket never lost a point across three runs — the decay curve was actually a map of where the bug was.

The final 2 needles from r7→r8 (G051/52, historically the semantic-loss needles that existed only in raw-archive): in r8 the compressor preserved the keys this round (evidenced by curated=1/conv=2) — the perfect score is attributable to fidelity of the compression main path, not to recall rescuing them at ASK time (0 tool calls in the ASK round).

Contamination finding (methodology). Reviving r7 and asking directly produced a reply byte-identical to the original run, with 0 tool calls — when the history contains a prior answer to the same question, the model simply replays it. Only after debridement (stripping the two questions and two answers, leaving history at block 64) did a rerun yield a valid comparison. Revival tests must be debrided first.

state_query / ask_recall / mailbox appeared 0 times in reasoning — not "considered and decided against checking," but rather that "check first" did not exist in the decision tree at all.mailbox_read → databus_query → DEEPSEEK_TOKEN_COUNTER overshoots 512K by roughly estimateContextTokens comment explicitly states that "the local count overestimates on highly repetitive corpora, and this is the usage.prompt_tokens on a prompt-cache hit (scanidx2 measured a real 1.2M while usage was stuck at 200K); after discarding the upstream count, local overestimation became a deliberate anti-overflow choice.markBeforeAsk = historyBeforeAsk.length, then after ASK ends, history.slice(markBeforeAsk) retrieves new messages — replaceRange (2 messages → 1) shrinks the array in place within the history region at any time, invalidating the index.historyBeforeAsk was captured at 04:26:26.8; memory.compressed events completed inside the ASK windowS-1790137731964-lqq7ex landing at 04:28:51.964); history was read at 04:28:51.97 → the slice came up empty → log shows 0 assistant msgs after ask. The fallback (scanning the full history for the last assistant string reply) happened to retrieve the correct answer, so markBeforeAsk to sessions-probe50); mailbox has tool_histogram={"mailbox_read":2,"mailbox_markread":1} — mailbox_markread advances the unread queue (matching the prompt guidance word for word);limit, so the error message became an untriggered safety gate (its logic covered by 13 unit tests).mailbox_read_any was already live-fire verified in r7-clean (11 internal calls inside recall, including cross-mailbox paging).benchmark-mrcr-3m/out-r1...r3, r7, r7-clean, r8, probe50 (blocks/ask/reply/score/run.log/trace/violations)sessions-r7, sessions-r7-clean, sessions-r8, Temp\mrcr3m-gold-s42\{r7,r8} (r8 gold uses the same 128-needle set as r7; re-scored and verified equivalent)

If you have better experimental ideas or an interest in the project, I'd love to hear from you. The GitHub repo is here: https://github.com/uos1231234/agent-shell — feel free to read the code, though my experimental data and the project-upgrade PR are not yet uploaded or merged. I am looking for people who share this technical direction to maintain and update the project together. You can reach me at 2424105750@qq.com, or send me a DM.