{"slug": "may-the-source-be-with-you-why-your-ai-agent-is-only-as-good-as-its-knowledge", "title": "May the Source Be With You: Why Your AI Agent Is Only as Good as Its Knowledge", "summary": "A developer argues that AI agents are only as useful as the knowledge they are given, and that poor agent performance often stems from missing or poorly structured context rather than model limitations. The post emphasizes the importance of knowledge architecture, distinguishing between rules, skills, and agents, and deciding what should be enforced in code versus stored as knowledge.", "body_md": "Everyone seems to be building AI agents.\n\nGive a model some instructions, connect a few tools, add a system prompt, and suddenly we have an \"agent.\"\n\nExcept there's a problem.\n\nA lot of them aren't particularly useful.\n\nWhen an agent produces a poor answer, the immediate assumption is often that the model isn't capable enough. But sometimes the model isn't the problem at all.\n\nThe agent simply doesn't have the right knowledge to work with.\n\nThat's where things get interesting.\n\nThe Agent Is Only Part of the System\n\nA simplified agent architecture might look something like this:\n\nUser\n\n↓\n\nInstructions\n\n↓\n\nAI Model\n\n↓\n\nTools\n\n↓\n\nAnswer\n\nBut for many real-world agents, something important is missing:\n\nUser\n\n↓\n\nAgent\n\n↓\n\nRelevant project knowledge\n\n↓\n\nReasoning + tools\n\n↓\n\nAnswer\n\nAn agent working on a real project may need to understand architecture decisions, terminology, conventions, constraints, processes, APIs, ownership, and dozens of other things that aren't part of the model's general knowledge.\n\nWithout that context, we're effectively asking the model to fill in the gaps.\n\nAnd models are very good at filling in gaps—even when we'd prefer them not to.\n\nBefore Building an Agent, Ask Whether You Need One\n\nThere's another question that often gets skipped:\n\nDoes this problem actually need an agent?\n\nSometimes a simple rule is enough.\n\nSometimes a reusable skill or deterministic function is a better solution.\n\nAnd sometimes an agent genuinely makes sense because the task requires interpreting context, combining knowledge, using tools, and deciding what to do next.\n\nThinking about that distinction before implementation can save a surprising amount of complexity.\n\nA useful mental model is:\n\nPredictable behaviour?\n\n↓\n\nRule\n\nReusable capability?\n\n↓\n\nSkill\n\nContext + decisions + tools?\n\n↓\n\nAgent\n\nThe boundaries aren't always this clean, but asking the question is useful.\n\nAgents shouldn't be the default answer simply because agents are currently interesting.\n\nWhat Should the Agent Actually Know?\n\nOnce we've decided that an agent makes sense, we reach a harder question:\n\nWhat belongs in its knowledge base?\n\nImagine building an engineering assistant for a platform team.\n\nWe could give it everything:\n\n/docs\n\n/src\n\n/wiki\n\n/slack-export\n\n/architecture\n\n/runbooks\n\n/meeting-notes\n\n/random-document-from-2019\n\nTechnically, that's a lot of context.\n\nPractically, it may be terrible context.\n\nMore information doesn't automatically mean better answers.\n\nA useful knowledge base needs some structure. The agent needs to be able to find the right information at the right moment, and ideally understand which sources are authoritative.\n\nThat means knowledge architecture starts to matter.\n\nKnowledge Architecture Is More Than Uploading Documents\n\nIt's tempting to think of an agent's knowledge base as a folder full of Markdown files.\n\nBut consider two documents:\n\narchitecture.md\n\narchitecture-final-v2-USE-THIS.md\n\nWhich one should the agent trust?\n\nNow imagine 500 documents with overlapping information, outdated decisions, duplicated explanations, and inconsistent terminology.\n\nHumans struggle with that.\n\nAn AI agent will struggle too—although it may sound considerably more confident while doing so.\n\nA good knowledge architecture needs to answer questions such as:\n\nWhat does the agent need to know?\n\nWhich information is authoritative?\n\nHow should knowledge be divided?\n\nWhat terminology should be consistent?\n\nWhich information changes frequently?\n\nWhich relationships between concepts matter?\n\nWhat shouldn't be included at all?\n\nThis is less about storing information and more about designing context.\n\nKnowledge or Code?\n\nThere's another boundary that deserves attention.\n\nNot everything the agent needs should live in its knowledge base.\n\nSuppose we have this business rule:\n\nProduction deployments require two approvals.\n\nShould that be documentation the agent reads?\n\nOr should the deployment system enforce it?\n\nIf something must happen deterministically, relying on an LLM to remember a paragraph in a knowledge base is probably not the strongest architecture.\n\nThis gives us another useful distinction:\n\nKnowledge\n\n\"What does this concept mean?\"\n\nCode\n\n\"What must the system always do?\"\n\nAgent\n\n\"What should I do given this context?\"\n\nAgain, real systems are messier, but separating these responsibilities can make an agent much easier to reason about.\n\nThe Confidently Wrong Problem\n\nOne of the biggest risks with knowledge-based agents isn't that they say:\n\nI don't know.\n\nIt's that they produce a perfectly plausible answer based on incomplete or outdated context.\n\nFor example:\n\nQuestion\n\n↓\n\nRelevant knowledge missing\n\n↓\n\nModel fills the gap\n\n↓\n\nPlausible answer\n\n↓\n\nWrong decision\n\nImproving the prompt may help.\n\nChanging the model may help.\n\nBut neither necessarily fixes the underlying problem.\n\nSometimes the real fix is:\n\nBetter source material\n\n+\n\nBetter structure\n\n+\n\nBetter retrieval\n\n+\n\nClearer boundaries\n\nThat's why I think knowledge architecture deserves much more attention when we're designing agents.\n\nBuilding the Knowledge Before the Agent\n\nThis is the idea I'll be exploring in a hands-on workshop called May the Source Be With You.\n\nRather than starting with:\n\nLet's build an agent.\n\nWe'll start with:\n\nWhat does this agent need to know?\n\nWe'll use Kipi, a knowledge-architecture tool, to plan and generate a structured knowledge base around a real project.\n\nOnly then will we put an agent on top of it.\n\nThe sequence matters:\n\nUnderstand the problem\n\n↓\n\nIdentify required knowledge\n\n↓\n\nStructure the knowledge\n\n↓\n\nDefine agent boundaries\n\n↓\n\nBuild the agent\n\n↓\n\nTest what it actually knows\n\nThe goal isn't simply to end up with another chatbot.\n\nIt's to understand the architecture underneath it.\n\nThe Interesting Part Isn't the Agent\n\nModels will continue to improve. Agent frameworks will change. Today's fashionable abstraction will probably be replaced by another one.\n\nBut the underlying problem remains:\n\nHow do we give an AI system the right context to make useful decisions?\n\nFor teams building agents around proprietary systems, internal processes, products, or engineering platforms, that may become one of the most important parts of the architecture.\n\nThe model gets most of the attention.\n\nThe knowledge might be what determines whether the system is actually useful.\n\nI'll be exploring this practically at CityJS Athens on 22 October 2026 at 09:00, in a hands-on workshop with Mey Beisaron, Senior Platform Engineer at NVIDIA.\n\nParticipants will build both sides of the system: a structured knowledge base using Kipi and a custom agent that operates on top of it.\n\nBring a laptop and, ideally, a project you already understand well.\n\nAnd yes...\n\nMay the Source be with you. ⚔️", "url": "https://wpnews.pro/news/may-the-source-be-with-you-why-your-ai-agent-is-only-as-good-as-its-knowledge", "canonical_source": "https://dev.to/cityjs_conference_5c08941/may-the-source-be-with-you-why-your-ai-agent-is-only-as-good-as-its-knowledge-1hhm", "published_at": "2026-08-22 19:48:18+00:00", "updated_at": "2026-08-22 20:13:57.505220+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-agents", "ai-products", "developer-tools"], "entities": [], "alternates": {"html": "https://wpnews.pro/news/may-the-source-be-with-you-why-your-ai-agent-is-only-as-good-as-its-knowledge", "markdown": "https://wpnews.pro/news/may-the-source-be-with-you-why-your-ai-agent-is-only-as-good-as-its-knowledge.md", "text": "https://wpnews.pro/news/may-the-source-be-with-you-why-your-ai-agent-is-only-as-good-as-its-knowledge.txt", "jsonld": "https://wpnews.pro/news/may-the-source-be-with-you-why-your-ai-agent-is-only-as-good-as-its-knowledge.jsonld"}}