Everyone seems to be building AI agents.
Give a model some instructions, connect a few tools, add a system prompt, and suddenly we have an "agent."
Except there's a problem.
A lot of them aren't particularly useful.
When an agent produces a poor answer, the immediate assumption is often that the model isn't capable enough. But sometimes the model isn't the problem at all.
The agent simply doesn't have the right knowledge to work with.
That's where things get interesting.
The Agent Is Only Part of the System
A simplified agent architecture might look something like this:
User
↓
Instructions
↓
AI Model
↓
Tools
↓
Answer
But for many real-world agents, something important is missing:
User
↓
Agent
↓
Relevant project knowledge
↓
Reasoning + tools
↓
Answer
An agent working on a real project may need to understand architecture decisions, terminology, conventions, constraints, processes, APIs, ownership, and dozens of other things that aren't part of the model's general knowledge.
Without that context, we're effectively asking the model to fill in the gaps.
And models are very good at filling in gaps—even when we'd prefer them not to.
Before Building an Agent, Ask Whether You Need One
There's another question that often gets skipped:
Does this problem actually need an agent?
Sometimes a simple rule is enough.
Sometimes a reusable skill or deterministic function is a better solution.
And sometimes an agent genuinely makes sense because the task requires interpreting context, combining knowledge, using tools, and deciding what to do next.
Thinking about that distinction before implementation can save a surprising amount of complexity.
A useful mental model is:
Predictable behaviour?
↓
Rule
Reusable capability?
↓
Skill
Context + decisions + tools?
↓
Agent
The boundaries aren't always this clean, but asking the question is useful.
Agents shouldn't be the default answer simply because agents are currently interesting.
What Should the Agent Actually Know?
Once we've decided that an agent makes sense, we reach a harder question:
What belongs in its knowledge base?
Imagine building an engineering assistant for a platform team.
We could give it everything:
/docs
/src
/wiki
/slack-export
/architecture
/runbooks
/meeting-notes
/random-document-from-2019 Technically, that's a lot of context.
Practically, it may be terrible context.
More information doesn't automatically mean better answers.
A useful knowledge base needs some structure. The agent needs to be able to find the right information at the right moment, and ideally understand which sources are authoritative.
That means knowledge architecture starts to matter.
Knowledge Architecture Is More Than Up Documents
It's tempting to think of an agent's knowledge base as a folder full of Markdown files.
But consider two documents:
architecture.md
architecture-final-v2-USE-THIS.md Which one should the agent trust?
Now imagine 500 documents with overlapping information, outdated decisions, duplicated explanations, and inconsistent terminology.
Humans struggle with that.
An AI agent will struggle too—although it may sound considerably more confident while doing so.
A good knowledge architecture needs to answer questions such as:
What does the agent need to know?
Which information is authoritative?
How should knowledge be divided?
What terminology should be consistent?
Which information changes frequently?
Which relationships between concepts matter?
What shouldn't be included at all?
This is less about storing information and more about designing context.
Knowledge or Code?
There's another boundary that deserves attention.
Not everything the agent needs should live in its knowledge base.
Suppose we have this business rule:
Production deployments require two approvals.
Should that be documentation the agent reads?
Or should the deployment system enforce it?
If something must happen deterministically, relying on an LLM to remember a paragraph in a knowledge base is probably not the strongest architecture. This gives us another useful distinction:
Knowledge
"What does this concept mean?"
Code
"What must the system always do?"
Agent
"What should I do given this context?"
Again, real systems are messier, but separating these responsibilities can make an agent much easier to reason about.
The Confidently Wrong Problem
One of the biggest risks with knowledge-based agents isn't that they say:
I don't know.
It's that they produce a perfectly plausible answer based on incomplete or outdated context.
For example: Question
↓
Relevant knowledge missing
↓
Model fills the gap
↓
Plausible answer
↓
Wrong decision
Improving the prompt may help.
Changing the model may help.
But neither necessarily fixes the underlying problem.
Sometimes the real fix is:
Better source material
Better structure
Better retrieval
Clearer boundaries
That's why I think knowledge architecture deserves much more attention when we're designing agents.
Building the Knowledge Before the Agent
This is the idea I'll be exploring in a hands-on workshop called May the Source Be With You.
Rather than starting with:
Let's build an agent.
We'll start with:
What does this agent need to know?
We'll use Kipi, a knowledge-architecture tool, to plan and generate a structured knowledge base around a real project.
Only then will we put an agent on top of it.
The sequence matters:
Understand the problem
↓
Identify required knowledge
↓
Structure the knowledge
↓
Define agent boundaries
↓
Build the agent
↓
Test what it actually knows
The goal isn't simply to end up with another chatbot.
It's to understand the architecture underneath it.
The Interesting Part Isn't the Agent
Models will continue to improve. Agent frameworks will change. Today's fashionable abstraction will probably be replaced by another one.
But the underlying problem remains:
How do we give an AI system the right context to make useful decisions?
For teams building agents around proprietary systems, internal processes, products, or engineering platforms, that may become one of the most important parts of the architecture. The model gets most of the attention.
The knowledge might be what determines whether the system is actually useful.
I'll be exploring this practically at CityJS Athens on 22 October 2026 at 09:00, in a hands-on workshop with Mey Beisaron, Senior Platform Engineer at NVIDIA.
Participants will build both sides of the system: a structured knowledge base using Kipi and a custom agent that operates on top of it.
Bring a laptop and, ideally, a project you already understand well.
And yes...
May the Source be with you. ⚔️