{"slug": "the-gate-has-to-touch-the-real-system", "title": "The gate has to touch the real system", "summary": "A developer's experiments with an agentic terminal and a fine-tuned model failed because they were measured with self-built instruments that lied, leading to the development of a context engine called ctx that uses verifiers to grant completion. The terminal merged nearly 400 PRs in two weeks but drove zero real workdays, and the fine-tuned Qwen3 model scored 98% on a frozen suite but only 17.5% on a held-out gate, showing the model learned the generator, not the task.", "body_md": "I bought a fully specced MacBook specifically for side experiments: local inference, fine-tuning, agentic builds. I've run three lately. I built `pt`\n\n, an agentic terminal that merged almost 400 PRs in two weeks. I fine-tuned a small local model to draw Mermaid flowcharts. And I kept extending ctx, my personal context engine. The terminal and the fine-tune both failed, for the same reason: I measured them with instruments I built myself, and the instruments lied. Verification is paramount. This post is the evidence, and what I'm building because of it.\n\nThe agent factory\n\n`pt`\n\nis an agentic terminal I wrote in Go: ten days, 89 internal packages, workers on rented remote machines, byte-verified delivery before anything became a pull request. Close to 400 PRs merged in two weeks, with 73 on the biggest day. The full build is [documented here](/blog/pt-attention-first-terminal); this is the tour it recorded of itself:\n\nIt had one rule, written into the repo and enforced in review: the home surface answers *what should I drive to closed today*. After two weeks I had driven zero real workdays through it. I had optimized a factory I never used for its intended job, and the merged PRs made that harder to admit, because every one was a green number I had generated myself.\n\nThe fine-tune\n\nI wanted diagramming to be free and local: hand a small model a plain-English description of a process, get back a correct Mermaid flowchart. I trained LoRA adapters for Qwen3 (0.6B through 4B) in MLX, with a teacher model supplying ground-truth graphs.\n\nOn my own gate, the model looked finished: 98% on a frozen 50-case suite, up from 0% zero-shot. On a gate I had never trained toward (120 cases written by a different model, in six domains my corpus never touched) it collapsed.\n\n| Round (1.7B) | Frozen gate | Held-out gate (fuzzy) | Held-out (strict) |\n|---|---|---|---|\n| r1 — full corpus | 94% | 0.8% | 0% |\n| r2 — mixed registers | 94% | 0.8% | 0% |\n| r3 — open-vocabulary labels | 88% | 12.5% | 3.3% |\n| r3.5 — messiness tiers | 98% | 15.0% | 2.5% |\n| r4 — rejection sampling | 98% | 17.5% | 4.2% |\n| r5 — filtered pool | 92% | 14.2% | 4.2% |\n\nSix rounds of corpus work moved the held-out number from under 1% to about 17%, then it stopped. The model had learned my generator, not the task. It could emit a plausible flowchart with the right shape and the wrong content. Reading arbitrary prose was the real job, and my synthetic corpus had none in it.\n\nVerification is paramount\n\nBoth experiments aced instruments I built, and both numbers lied the same way. The fine-tune's scorer shared the corpus's blind spot. Merged PRs proved code moved through the factory and said nothing about whether the product worked. The thing being tested cannot grade itself.\n\nThe difference is palpable at the keyboard. When your agentic loop can see its own work, you essentially get one-shotted features: it catches the final three issues itself, patches them, verifies the fixes, and hands the work back finished. When the loop can't verify, you burn tokens and hours in frustration. And that frustration is more alienating than I expected, because you can essentially build anything you can imagine now, and you know exactly how much faster it would go if the loop could check itself.\n\nHolding the whole graph of work\n\nThat lesson is what ctx is being built around. Its job is to hold the entire graph of my work: every task, its inputs, the actions taken, the artifacts produced, the evidence collected, and the decisions made, across every agent and machine I use. In that graph, completion is a state a verifier grants, not a claim an agent makes.\n\nHere it is live. A release task's *all PRs merged* gate sits in FAILED because ctx re-read GitHub and found a sixth PR had entered scope. The agent's summary said done. The gate didn't care, and the task stayed open with the follow-up attached.\n\nWhat ctx already does well\n\nTwo ctx capabilities have run untouched since the day I shipped them: shared memory and secrets.\n\nOne memory across every agent\n\nEvery agent I run (Claude Code, Codex, Crush, pi, OpenCode, the one on my phone) starts with the same memories, outcomes, file paths and decisions. No re-explaining who my clients are, what's due this week, or what I decided last month.\n\nCredentials agents can use, but never see\n\nAgents act with my credentials without ever holding them. Values live in Cloudflare Secrets Store, agents carry revocable scoped tokens, and the Worker makes the call on their behalf.\n\nEarlier today one of my own sessions hit the vault with a token that lacked the grant:\n\n```\nerror 403: token lacks 'secrets' grant\n```\n\nMy own request, failing closed. I could not talk the vault into doing otherwise.\n\nNeither capability is coming from a frontier lab. Their memory lives inside their own product; mine has to travel across all of theirs. Their credential handling secures their agent; mine has to hold for whichever agent shows up next month.\n\nThree bets\n\nThis round of refocusing on ctx encodes three bets.\n\nFirst: that centralizing private, context-rich tools behind MCP works across every agent and every harness. One server, reachable from whatever shows up next month, instead of per-tool memory files and per-harness plumbing.\n\nSecond: that a centralized MCP can hold the authoritative graph of work state and use it to steer implementation sessions across agents and harnesses. The graph decides what's open, what's blocked, and what's verified; the agents come and go.\n\nThird: that preserving all of my own tasks in a readable, trainable form, and auto-extracting every artifact along the way (PRs, decisions, generated images), pays off down the road. Research packets, decisions and past projects get injected into new tasks as needed, and the archive itself becomes training data for small models, custom fine-tunes and LoRAs. That's the real corpus the Mermaid experiment was missing, accumulating as a byproduct of work I was doing anyway.\n\nKeep the model swappable\n\nI'm trying to stop betting against model progress. Capabilities I might spend a month building keep landing in the models themselves; `pt`\n\nis the fresh scar. So everything that survives treats the model as a part you swap. In ctx, switching models is literally a dropdown: it switches live, bills through Cloudflare, and involves no provider keys.\n\nThe harness follows the same logic. Building `pt`\n\ntaught me what I actually want from a daily driver, and `pi`\n\nturned out to be the closest existing thing to it, without most of `pt`\n\n's footguns. I don't maintain the glue code or the plumbing. I benefit from open-source extensions and the private ones colleagues and friends share. And I can open source my own extensions and tools instead of welding them into a terminal only I run.", "url": "https://wpnews.pro/news/the-gate-has-to-touch-the-real-system", "canonical_source": "https://zackproser.com/blog/what-survived-the-experiments", "published_at": "2026-08-31 00:00:00+00:00", "updated_at": "2026-08-31 15:53:12.606387+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-agents", "ai-research"], "entities": ["MacBook", "pt", "ctx", "Qwen3", "MLX", "GitHub"], "alternates": {"html": "https://wpnews.pro/news/the-gate-has-to-touch-the-real-system", "markdown": "https://wpnews.pro/news/the-gate-has-to-touch-the-real-system.md", "text": "https://wpnews.pro/news/the-gate-has-to-touch-the-real-system.txt", "jsonld": "https://wpnews.pro/news/the-gate-has-to-touch-the-real-system.jsonld"}}