{"slug": "your-coding-agent-knows-the-internet-s-average-here-s-how-to-make-it-know-yours", "title": "Your Coding Agent Knows the Internet's Average. Here's How to Make It Know Yours.", "summary": "A developer building agentproto argues that coding agents trained on public internet data produce generic answers, and the only way to make an agent specific to a team's codebase is to attach a custom knowledge base. The post outlines three tiers of knowledge integration, from no knowledge base to a served, queryable system, and claims retrieval quality is the dominant factor in agent performance.", "body_md": "Disclosure up front: I build\n\n[agentproto], whose\n\nknowledge and operator primitives are the back half of this piece. The problem\n\nin the first half stands on its own, and the walkthrough uses real, checkable\n\ncommands. Corrections welcome — file an issue.\n\nOpen Claude Code. Open Codex. Open a cheap local model in the same repo. Ask all\n\nthree the same question about your codebase — your retry convention, your deploy\n\nrunbook, why that one service is the way it is.\n\nYou'll get three fluent, confident, near-identical answers. And all three will be\n\nwrong in the same way: they'll describe how a *reasonable* team does it, not how\n\n*yours* does. They're all running the same frontier weights, trained on the same\n\npublic internet, so out of the box they all know exactly the same thing.\n\nThe one idea, if you remember nothing else:\n\nThe weights are everyone's. Your knowledge base is the only part of the stack\n\nthat'syours— and the only thing that makes the agent yours instead of the\n\ninternet's average.\n\nHere's the shift most people missed while arguing about benchmarks. The gap\n\nbetween the best model and the tenth-best is collapsing — [the hub piece](https://dev.to/agentiknet/you-can-parallelize-the-typing-you-cant-parallelize-the-trust-2fkd)\n\nhas the Stanford AI Index receipt (the #1-to-#10 spread fell from 11.9% to 5.4%\n\nin a year). When the weights converge, the weights stop being your edge.\n\nWhat replaces them is system design. One widely-cited 2026 essay on the evolution\n\nof agents put it bluntly: **agent capability has moved from model quality to\nsystem design** — from writing a clever prompt to engineering what the model can\n\nThe receipt that reframes the whole market.In a production-readiness\n\ncomparison of agent frameworks, one finding held across every orchestration\n\nstyle tested:retrieval quality dominates knowledge-agent performanceYou can have the best planner-worker-judge\n\nregardless of orchestration.\n\ntopology in the world; if the agent is retrieving from the internet's average\n\ninstead of your world, it answers like a stranger.\n\nSo the interesting question stopped being \"which model?\" and became \"what does it\n\nknow that no one else's does?\" JetBrains' framing is useful here: some agents are\n\n**action-first** (their value is what they *do*) and some are **data-first**\n\n(their value is what they can *access*). A coding agent for *your* team is\n\ndata-first whether you designed it that way or not.\n\n**The uncomfortable part: if you haven't attached your knowledge to your agent,\nyou're shipping the stranger.**\n\nRun the self-diagnosis before you build anything. There are three tiers, and most\n\nteams are stuck on the first two without noticing.\n\n**Tier 0 — no knowledge base.** The agent knows what the pre-training run knew:\n\npublic GitHub, Stack Overflow, docs up to some cutoff. Ask it about your world and\n\nit pattern-matches to the nearest public example. It's confident and it's generic.\n\n*Ceiling: it will never know a single thing that isn't already on the internet.*\n\n**Tier 1 — pasted into context.** You dump the runbook, the conventions doc, a few\n\npast PRs into the prompt (or a `CLAUDE.md`\n\n) and hope. It works — until the context\n\nfills up. Anthropic's context-engineering post is direct about why this caps out:\n\nyou have to **treat context as a finite attention budget**, because recall decays\n\nas the window grows (\"context rot\").\n\nWhy Tier 1 has a hard ceiling.The same post prescribes the fix —keep, not pre-load everything.\n\nlightweight references and retrieve just-in-time\n\nPasting your whole knowledge base into every prompt is the exact anti-pattern:\n\nit's expensive, it rots, and it doesn't scale past a handful of documents. A\n\nKB you paste is a KB you'll soon truncate.\n\n**Tier 2 — a served, queryable knowledge base.** Your knowledge lives in one place,\n\ngets indexed, and every agent *queries* it on demand — pulling only the three\n\nrelevant entries for the question at hand, with their sources attached. This is\n\njust-in-time retrieval, done properly, shared across every agent you run.\n\n**The jump from Tier 1 to Tier 2 is the whole game — and it's the \"who equips the\nagents\" question from the files-with-contracts piece,\npointed straight at knowledge.** Let's build it.\n\nStart with the packaged shape, because it shows the target. In agentproto, an\n\n**operator** (the AIP-9 `OPERATOR.md`\n\n) is a persona plus policies plus a set of\n\nbound tools — and one of those tools is a knowledge binding.\n\n```\n# operators/house-engineer/OPERATOR.md — persona + policies + a KB binding\n---\nname: House Engineer\nid: house-engineer\nprofile:\n  role: Answer and act using OUR conventions and past incidents, never generic defaults.\n  voice: Concrete, cites the runbook or incident it's drawing from.\ntools:\n  - knowledge-query          # ← the bound knowledge tool\n  - shell\npolicies:\n  - \"policy:engineering/house-rules\"\nmetadata:\n  corpus:\n    knowledgeViews:\n      - corpus: engineering\n        filter: { tags: [conventions, runbooks, incidents] }\n---\n```\n\nRead what that `knowledgeViews`\n\nblock does: it scopes the operator to *your*\n\nengineering corpus, filtered to the tags that matter. The persona decides how it\n\nanswers; the policies decide what it's allowed to do; the knowledge view decides\n\nwhat it *knows*. Three separate concerns, one file.\n\nThis is a real shape, not a mock.The research operators shipped in\n\nagentproto's own corpus —`source-scout`\n\n,`research-analyst`\n\n— carry exactly\n\nthis:`tools: [knowledge-query, ...]`\n\nplus a`metadata.corpus.knowledgeViews`\n\nbinding scoped by domain and quality score. The operator is thepackaging.\n\nUnderneath it is a tool, and the tool is the part that ports to any agent.\n\n**An operator is what \"give the agent your knowledge\" looks like when it's one\ndeclared file.** Now let's build the tool that backs it — the part that reaches\n\nThis is the exact move from [the files-with-contracts piece](https://dev.to/agentiknet/your-claude-skill-is-invisible-to-codex-heres-how-to-fix-it-3l74),\n\nspecialized to knowledge: split the **contract** (what the tool promises) from the\n\n**driver** (the code that answers). Here's the contract — a pure `defineTool`\n\n,\n\nno execute body.\n\n``` js\n// tools/knowledge-search/TOOL.ts — the contract, and nothing else\nimport { defineTool } from \"@agentproto/tool\"\nimport { z } from \"zod\"\n\nexport const knowledgeSearch = defineTool({\n  id: \"knowledge.search\",\n  description:\n    \"Answer from OUR knowledge base: conventions, runbooks, past incidents. \" +\n    \"Returns refined entries, each with the source it came from.\",\n  inputSchema: z.object({\n    query: z.string(),\n    tags: z.array(z.string()).default([]),\n    limit: z.number().default(5),\n  }),\n  outputSchema: z.object({\n    entries: z.array(z.object({\n      kind: z.string(),\n      title: z.string(),\n      body: z.string(),\n      source: z.string(),      // provenance — where this knowledge came from\n    })),\n  }),\n  mutates: [],                 // read-only → no approval gate\n  approval: \"auto\",\n})\n```\n\nNow the driver — the implementation, bound to the contract by `implementTool`\n\n. It\n\nreads your corpus workspace off local disk and returns the matching entries.\n\n``` js\n// drivers/knowledge-corpus.ts — backed by your own corpus workspace\nimport { defineDriver, implementTool } from \"@agentproto/driver\"\nimport { knowledgeSearch } from \"../tools/knowledge-search/TOOL.js\"\nimport { resolveKnowledge } from \"@agentproto/corpus\"\nimport { NodeFsAdapter } from \"@agentproto/corpus/fs\"\n\nexport default defineDriver({\n  id: \"knowledge.search.corpus\",\n  name: \"Knowledge search over your corpus workspace\",\n  kind: \"in-process\",\n  implementations: [\n    implementTool(knowledgeSearch, async ({ input }) => {\n      // the same resolver behind `corpus knowledge <ws> --tags … --max …`\n      const hits = await resolveKnowledge({\n        fs: new NodeFsAdapter({ root: process.env.CORPUS_WS! }),\n        query: { tags: input.tags, maxResults: input.limit },\n      })\n      return {\n        entries: hits.map((h) => ({\n          kind: h.kind,\n          title: h.title,\n          body: h.body,\n          source: h.sources.join(\", \"),\n        })),\n      }\n    }),\n  ],\n  network: { egress: [] },     // reads local files → your knowledge never leaves the box\n})\n```\n\nThat driver is the real corpus resolver — the same `resolveKnowledge`\n\ncall that\n\nsits behind `corpus knowledge <ws> --tags conventions,incidents --max 5`\n\non the\n\ncommand line, now exposed as a tool. The `network.egress: []`\n\nline is doing quiet\n\nbut load-bearing work: **the knowledge is queried from local files, so it never\nleaves your machine.**\n\nProvenance is a feature, not decoration.Every entry carries its`source`\n\n—\n\nthe runbook, the incident, the PR it was distilled from. That's what lets an\n\nexternal check —[the supervision ladder's]whole\n\npoint — verify the answer later: the LLM-as-a-judge literature evaluates RAG on\n\ntwosides, context relevance going in andfaithfulness coming out(did the\n\nanswer actually follow the retrieved source, or wander off it?). No provenance,\n\nno faithfulness check.\n\n**One contract, one driver, and now every agent that mounts the daemon can answer\nfrom your world.** Serve it.\n\nStart the daemon. It reads your `tools/`\n\nand `drivers/`\n\nand serves them over MCP,\n\n`knowledge.search`\n\nincluded.\n\n```\nnpm i -g @agentproto/cli\nagentproto serve                 # serves the /mcp gateway to every agent\n```\n\nNow spawn an agent *with* the tool, and ask it something it could only know from\n\nyour docs — a convention set by a past incident:\n\n```\nagentproto sessions start codex --cwd . \\\n  --prompt \"What's our retry policy for the payments queue? Use knowledge.search.\"\n→ Per incident #2026-04 (double-charge on gateway timeout), payment retries are\n  capped at 3 with jittered backoff, and we NEVER retry a charge without an\n  idempotency key. Source: runbooks/payments-queue.md, incidents/2026-04.md\n```\n\nNow the same agent, same question, **without** the tool:\n\n```\nagentproto sessions start codex --cwd . \\\n  --prompt \"What's our retry policy for the payments queue?\"\n→ A common approach is exponential backoff with 3–5 retries and a dead-letter\n  queue after the final attempt…\n```\n\nThe second answer isn't *wrong* — it's the internet's average, and it's exactly\n\nwhat you'd get from any agent, anywhere. The first answer knows about the\n\ndouble-charge you shipped in April, because it read your incident file. **That\ndelta — a real past incident it could only learn from your knowledge base — is\nthe entire difference between an agent and **\n\nCross-vendor, by construction.That was Codex. Point Claude Code or a cheap\n\nlocal model via Hermes at the same daemon and they call the identical\n\n`knowledge.search`\n\n— because it's served over MCP, not baked into one vendor's\n\nconfig. Author the KB tool once; every agent in your fleet inherits your world.\n\nHere's the honest framing, because overclaiming is how you lose a reader. In\n\n[Guilde](https://guilde.work) — our AI-company product — attaching a knowledge\n\nbase to an agent is *one packaged step*: an operator ships with an attached\n\ncorpus, and you apply it to a scope in a single command. Persona, policies,\n\nknowledge, done.\n\nWhat you just built by hand in agentproto is that same primitive, one floor down:\n\nan operator (Step 1) plus a served knowledge tool (Step 2). **Guilde adds the\none-command packaging; agentproto is the from-primitives path you can run today**,\n\nWhat agentproto doesThere's no singlenotdo yet, stated plainly.\n\n`apply-knowledge-pack`\n\nverb in the open CLI — you wire the tool, the driver,\n\nand the operator yourself, as three files. That's more assembly than the\n\npackaged version. It's also fully inspectable and fully yours, which for a\n\nknowledge base — the most sensitive thing you'll attach to an agent — is often\n\nthe trade you want.\n\nThe primitive is identical either way: **an operator with a queryable knowledge\ntool.** One product packages it; the other hands you the parts.\n\nA knowledge base is not magic, and three caveats keep it honest.\n\n**A wrong KB is worse than no KB.** If your knowledge base contains a stale\n\nconvention, the agent will state it with the same confidence as a correct one —\n\nnow with a *source* attached, which makes it more persuasive and more dangerous.\n\nThis is why the faithfulness check above matters: retrieval quality is the whole\n\nballgame, and a KB is an artifact you maintain, not a bucket you fill once.\n\nDon't let the agent write its own knowledge base.ETH Zurich research\n\n(Gloaguen et al.) found that LLM-generated`AGENTS.md`\n\nfiles providednoA knowledge base curated from real\n\nbenefit and could reduce success rates.\n\nincidents, runbooks, and decisions is yours; one the agent hallucinated about\n\nitself is just the internet's average with your repo's name on it. Curate the\n\nKB like you'd curate docs — because that's what it is.\n\n**And it's the most sensitive thing you'll attach.** Your incidents, your\n\nconventions, your internal decisions — that's exactly the data you don't want\n\nleaving your network. The `network.egress: []`\n\ndriver above isn't a detail; it's\n\nthe reason this whole pattern runs locally. Your knowledge is your edge precisely\n\nbecause it's private, so the tool that serves it should keep it that way.\n\nEvery team in 2026 rents the same frontier weights. That was always going to\n\ncommoditize — the models converge, the benchmarks compress, and the clever prompt\n\nbecomes a thing anyone can copy. What doesn't commoditize is what your agent knows\n\nthat no one else's does: your codebase's conventions, your runbooks, the incident\n\nthat taught you never to retry a charge without an idempotency key.\n\nThat knowledge is the one layer of the stack you actually own. Attaching it isn't\n\na nice-to-have on top of a good model — after the models converged, **it's the\nonly move left that makes your agent yours.** An operator with a queryable\n\nSo run the diagnosis one more time, honestly: does your agent know your world, or\n\njust the internet's average of it? If it's the average, you don't have an agent\n\nproblem. You have a knowledge problem — and it's the good kind, because the fix is\n\nentirely in your hands.\n\nIf your setup already attaches knowledge a cleaner way, or I've mis-described how\n\nyour KB reaches your agents, tell me where — I'll fix the piece.\n\nTen pieces, one argument. Start anywhere; each one cross-links the rest.\n\n| Piece | The one idea | |\n|---|---|---|\n| 1 |\n|\n\n*Written by the maintainer of agentproto (Apache-2.0, source). Same contract as our /compare page — dated facts, named strengths, corrections by issue. Got something wrong? File an issue.*\n\n*Building agentproto in the open — follow @theagentproto and @agentik_ai on X.*", "url": "https://wpnews.pro/news/your-coding-agent-knows-the-internet-s-average-here-s-how-to-make-it-know-yours", "canonical_source": "https://dev.to/agentiknet/your-coding-agent-knows-the-internets-average-heres-how-to-make-it-know-yours-1aeg", "published_at": "2026-07-14 01:36:33+00:00", "updated_at": "2026-07-14 01:57:22.681772+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-agents", "developer-tools"], "entities": ["agentproto", "Claude Code", "Codex", "JetBrains", "Anthropic", "Stanford AI Index"], "alternates": {"html": "https://wpnews.pro/news/your-coding-agent-knows-the-internet-s-average-here-s-how-to-make-it-know-yours", "markdown": "https://wpnews.pro/news/your-coding-agent-knows-the-internet-s-average-here-s-how-to-make-it-know-yours.md", "text": "https://wpnews.pro/news/your-coding-agent-knows-the-internet-s-average-here-s-how-to-make-it-know-yours.txt", "jsonld": "https://wpnews.pro/news/your-coding-agent-knows-the-internet-s-average-here-s-how-to-make-it-know-yours.jsonld"}}