# No Dumb Questions: What is AI context architecture? Why not just build your own? > Source: > Published: 2026-08-14 16:12:05.506066+00:00 **Phoebe Sajor: **Hi Doug, Hi Ash, thank you for joining me for this No Dumb Questions. To start—what exactly is context architecture in AI and why do we care? **Doug Whitley: **So, it's kind of the same thing that regular context architecture is when you're talking about the actual architecture in buildings. AI context architecture is about fitting everything around the AI into the system—what you're viewing, working with, and seeing. It's easy to get caught up in minute details when it comes to AI. You may want to have a very focused context that you're keeping in your system. For example, with AI I don't necessarily need to know the thousands of logs that went into the work I’m doing, but I do need to know the information in a very specific set of logs when I'm trying to solve a problem related to them. **Ash Zade: **As a Product Manager, I think of AI context architecture as the guardrails and constraints that we can provide for AI agents. What you're trying to do is remove the ambiguous decision-making and variables from AI agents so the user can have more predictable outcomes. It might look like setting up a system that tells your AI agent, “Only look at this data and given that data only do these three things. If you get stuck, do these two things.” So, not only does the agent have the limited context it needs to work off of, but also the context of what to do when it runs into new situations. **Phoebe Sajor: **I recently interviewed Michael Foree on this show and he taught me [a little bit about context engineering](https://stackoverflow.blog/2026/07/24/no-dumb-questions-ai-bottleneck/). How is context architecture different from context infrastructure? How is context architecture different from context engineering? How do they all work together? **Doug Whitley: **The best way to understand it is to start with context infrastructure, which is when we start getting into ways you're actually serving context. Infrastructure is how you deliver something—like context—or how you organize what you're going to deliver. So for context infrastructure, you'll see things like context libraries and RAG-specific context infrastructure. It’s the mode of storing, surfacing, and giving context to an AI agent so it can solve a problem for you. When we talk about architecture vs. engineering—regardless of the word context in front of it—the lines get a little blurrier. They really go hand-in-hand, but architecture is a little more philosophical. Architecture asks why we might try to do things in certain ways, why we might try to put things together to achieve certain goals. Architecture is the design. The standard design patterns engineers use, for instance, is architecture. Engineering is really where the rubber hits the road—where you're actually building stuff. You might decide to use this particular algorithm because it's very effective or you might use certain languages because you’re building something with RAG. **Phoebe Sajor: **So where does RAG and MCP fit into the AI infrastructure vs. architecture vs. engineering discussion? **Doug Whitley: **When you're talking about MCP, that's architecture. MCP is a defined protocol that has requirements, but how you implement it—how you design it into the system—is completely up to you. You can implement MCP in any language, or you might choose to only implement certain MCP server features or only support certain MCP clients. You’re making decisions about the design of the AI system. RAG is a little bit of all three working together. You're going to have some sort of infrastructure in the back of the RAG—it might be your indexing, it might be a context store that you've done highly performant query setups with, or something else that delivers context and makes it searchable. Maybe it’s a list of your rules and processes for a coding agent saved in Markdown files. The way you store and serve that context to your ML or LLM model is going to be the infrastructure.** **But RAG is also engineering. For RAG, we might build it in .NET, we might build it in Python, and if we're feeling really spicy, we'll build it in Rust. We can build it many different ways, but when we do, we have to follow an architecture. Think of it like this: if we're building a car, the fact that we're building a car in the first place is an architectural decision. We decided we wanted a car, not a boat, not a scooter, not a bicycle. The make and model are the engineering side of it. Where we buy the parts is the infrastructure. **Phoebe Sajor: **Ash, from a PM product perspective, how exactly does context architecture make AI agents work better? **Ash Zade:** I'll borrow Doug's car analogy. We can ask an agent to go out and learn everything it can about different tire options and come back and provide recommendations. If that agent walks into a library and looks for tires, it’ll find information on tires for planes, bicycles, and wheelbarrows. It’ll find matches for tires anywhere because they're all tires. It's valid. But if we’re building a car, we only care about car tires. But no matter how much you try to set up a guardrail for the agent to only look up car tires, if you’re doing it in a prompt you can't guarantee it's only going to come back with car tires. What you want to do is set limits on the information you give the agent access to. When you control what it has access to, it's not going to drift and bring back information about bicycles. Here's the knowledge I want you to care about and use in the task I'm giving you. Another part of building context is agentic memory—making sure the agent remembers the work it’s already done. It’s checked for sports car tires, van tires, and truck tires. You can refine it and instruct the agent to only focus on sports cars. But you can’t build a sports car in a day, so you’ll want the agent to maintain that history and remember it’s working on a sports car. Then you can go away for a day and come back to keep working on the sports car, even when there are multiple things your agent is working on. Part of AI context is saving what the agent is doing, what it's learned, what it's communicated, and what it's built so far. If you multiply this with 10 agents or 100 agents, it’s especially important for you to maintain all that information so you can build upon it. So you have retrieval and the maintaining of context. But you also want to define what the agent should do if it runs into this situation or that situation. That’s guardrails. An example is a human-in-the-loop mechanism, which we have in our product [Stack Internal](https://stackoverflow.co/internal/). So imagine you send your agent out to perform an action, but we don't know if we have all the answers the agent needs to complete it within our knowledge base. We don't want the agent to act upon knowledge that's incomplete or incorrect. In Stack Internal, what we do is we label the knowledge as incomplete or incorrect. Then we tell the agent, “If you run into things that are incomplete or incorrect, here's what you should do about it. Don't just use it. Don't assume it's correct. Don't make a judgment call if it's correct or not.” The way we do this is with [a trust system](https://stackoverflow.co/internal/features/). It’ll score knowledge as high, medium, or low. If the score is medium or low, we tell the agent to use a tool called a subject matter expert validation flow. The agent will automatically route its user to an expert on that topic who can validate or fill in blanks for them. With context architecture, what we're doing is really just removing variables. We remove the variable for an agent that makes it confuse bicycle tires with car tires, and give it boundaries for only sports car tires. We remove the variable of high quality vs. low quality knowledge so it doesn't need to make a judgment call on incomplete or incorrect information. Then we give it a mechanism so if it runs into a wall, it won’t just try to figure it out on its own but instead bring a human-in-the-loop. The whole point is to safely and confidently create tasks for agents with access to knowledge and know that they're going to do the thing you expect them to do. And if they run into a wall, they will also do the thing you expect them to do. **Phoebe Sajor: **When you give agents this much access to information, how do you keep your data safe? Are there privacy and security concerns with giving agents access to knowledge? **Ash Zade: **So, in a lot of cases you'll take a tool like Codex and [you’ll connect](https://stackoverflow.co/internal/integrations/) it to Slack, MS Teams, Google Drive, SharePoint, whatever. And then you'll ask it to do something. The agent is going to pick whichever platform it thinks it should use to find information for your task. But maybe it's going to go into a Slack channel that you don't want it to incorporate data from. It’ll go into an Innovation Slack channel where people are talking about potential things to build and the agent will come back and return facts about something that hasn’t been built yet. So privacy is beyond what the agent should and shouldn’t see. It could simply be that we don't want certain information incorporated into its work at all. That’s why context architecture is so important when giving agents access to data. When it comes to security, it really becomes about permissions. In Stack Internal, we handle this by giving the agent the same permissions as its source permissions. The user’s access permissions are carried over into Stack Internal. And so, if it's working on my behalf as a product manager, it's going to have the same context as I do and only that context. If [I want to restrict it more](https://stackoverflow.blog/2026/06/17/ai-agents-expose-the-security-checks-you-never-actually-wrote/) and say, "Okay, I have access to a lot of things, but I only want you to care about my product area," we have a mechanism called Scopes that allows users to further define what the agent has access to. So there's two layers here. There are the permissions of the source. If it's a private Slack channel and I'm not a member of it, my agent is not going to be a member either. But even if I am a member of that private Slack channel, I still may not want to incorporate that knowledge into my agent's work. When that happens, I can set a Scope around all the knowledge I have access to and only give the agent a subset of that. So I get to curate the things I have access to, but an agent working on my behalf will never have access beyond what I have access to That’s just the retrieval part. But like I said, you want your agents to store memory and create new knowledge context around the things it did. It may be pushing knowledge back into the system. That's where you also want to set some restrictions. You can also do that with Scopes. Okay, so my agent has access to this data, and I want it to capture what it’s doing. But this information is particular to me. I can tell the agent not to put it in the general knowledge pool or share it with my team. Send it to me first. You can control for that, too. **Phoebe Sajor: **One of the most interesting things about AI is that everything is constantly changing all the time. It's both exciting and scary. But how does that work with context architecture? What happens to context when the underlying models or systems change? **Doug Whitley: **The cool thing about context architecture is it doesn't have to be about how it interacts with a specific AI model. We actually do context architecture in our day-to-day life—you may only keep certain things on your desk because you're trying to keep that context very specific to your workflow. Context architecture is something we naturally engage with day-to-day. We find success when we control the context around us, even physically. So, the cool thing about context architecture is it’s making things useful regardless of who's on the other end of it. Whether it's an MCP client or server that's doing something for you as a user, whether it's you directly interacting with the context that's being given to you from a search endpoint, or it's just you trying to do some work, keeping a clean context is really valuable. The same is true for AI context architecture, regardless of what AI model you hook up. We're just being explicit about how it might be more useful to codify it and build a tool around it specifically. **Phoebe Sajor: **What does the actual process of using context architecture look like in AI systems? How do AI agents get fed context through the architecture? **Doug Whitley: **Like anything, it’s going to depend. With Stack Internal, we use a lot of different algorithms to give users the best result. There's indexing, specific tagging that's going to happen, and vector searches you might do that all depend on the question coming in. So part of our process is looking at what's being asked, what's going on in the workflow, and figuring out the best way to search the data. There’s context mapping, which you can imagine as a crazy board with pins and strings connecting things together. There’s the classical algorithm concept of a table where you have stuff on a table and you can easily grab from it or leave it out. And you can think of Scopes like a stack of sticky notes you’ve written about specific contexts. We also use the trust scores that Ash has talked about previously to figure out what information your agent can safely act on. Stack Internal also re-ranks the information and figures out what context is most important for you and your agent. Good AI context architecture takes multiple steps. I like to think of it as casting a really wide net, pulling it in, and saying, "Well, we're looking for fish, not oysters. We're going to throw the oysters out. And actually, we only want this kind of fish, so we’ll throw back the other kinds. And we also only want fish this size, so we’ll get rid of ones that are too small or too big.” We’re taking multiple steps to filter out information you don’t need. **Phoebe Sajor: **I love that net analogy. So why wouldn't a company just build their own net instead of, say, going to the supermarket to buy a fish? What are the benefits to buying context architecture instead of building it on your own? **Ash Zade: **There are two parts that are important to building your own context architecture. There’s the technical side, then there’s how it should actually work. I'll go back to the car analogy. I’ve asked my agent to return information about sports car tires, so it goes to the library. But when it pulls out two books, they have conflicting information about the best sports car tires. What should my agent do when there's a conflict? How do we resolve it? How do we even identify a conflict? Context architecture may look like a simple problem of bringing in all this data, sorting it, indexing it, reranking it, and then serving it up. But now our data is everywhere—it’s in Slack and MS Teams, Google Drive, SharePoint, Confluence, GitHub, Jira. You want access to all of it. It's all valuable. But once you bring it in, it creates a massive problem. And it’s not only the technical problem, which is already huge. You have to figure out what happens when you run into conflicting, incomplete, or incorrect information. How do we pick these things? How do we even rank them? Doug and I work very closely together and have a lot of philosophical conversations about this—honestly, more than technical conversation. Because I think there really is a big cost that may go unnoticed or understated when it comes to the details and the decisions around things like trust or conflicts. You know, trust is in our DNA at Stack Overflow. We've done a lot of research, and we’ve asked people how they know they can trust an AI output. The majority of the time, it comes down to their expectations of what the output should be. They’re doing a customer report that they’ve done 20 times before. Now, they’ve asked AI to help them with it. If it spits something out that doesn't match their expectations as someone who's an expert on the report, they don't trust it. But that only works if you have experience. It's a massive problem for people who don't have experience in the things they’re asking AI to do, like with vibe coding. So how do you build a context architecture that people can actually trust? People with experience are not looking for specific words or a specific image in the output. In order to trust their AI, they have an overall expectation that needs to be met. Once we broke that down, there was actually a lot of effort and philosophical conversations involved in even figuring out what we needed to build. And then we had to build it, which was complicated on its own. It's very difficult to create a system or a product that can [meet the expectations needed for trust](https://stackoverflow.blog/2026/07/29/developers-are-attached-to-tools-because-tools-encode-trust/). There's a lot of pieces involved in order to do that well. I think that’s what gets missed in a lot of the build vs. buy conversations. Yes, it is possible to do the technical coding for your architecture—that’s completely valid. But what about how you treat all the data? How will you sort, filter, and determine what context is best, and do it in a systematic way? **Phoebe Sajor: **From both a product and technical perspective, what constitutes a good context architecture? What should people look out for in their context architecture? **Doug Whitley: **Honestly, good context architecture means you get the perfect answer every time. Getting to that point is the hard part. You need something that is going to be flexible for your use cases. It can't be overly specific or you're going to get a tunnel vision architecture that is only useful for sports car tires. You’ll never be able to look for tires that go on wheelbarrows. Or if you do, it's always going to say you should get a car tire and just put it on the wheelbarrow, which is something I've run into with AIs before. I think what makes a good context architecture is that it works well across users, whether that's a model or an actual individual human user. It provides strong context and helps you catch things you wouldn't have otherwise caught. And maybe it’s giving you extra context you didn't necessarily know. Maybe it realizes that you need a specific PSI for your tires because it knows the make and model of the car and gives you an output based on that extra context. So, good context architecture can also fill in gaps you didn't know were there. **Ash Zade: **I think really good context architecture is based on how predictable and consistent the output is. If I search for tires of a certain PSI, will I get the same answer every time? Will Doug get the same answer? Will you get the same answer? That is the thing we're trying to achieve. So when we say the perfect answer, it's delivered with consistency and predictability. Those are two things that have been missing from AI over the years. It does a lot of great things, but we don't have the confidence to give an agent a task and let it loose because we can't predict what it's going to do. We want predictability so we can have confidence in these tools. If you do context engineering and architecture well, not only will you predictably get the right answer every time, but you also optimize for token costs. If you're asking the agent to go through a library, it will take hours to find all the books. That's a lot of tokens and investment. If you ask it to just go through books on cars, that's less. If you ask it to just go through the pages about tires, that's less. If you ask it to go through the paragraphs on tires and PSIs for sports cars, that's even less. And that's a thing that I don't think is apparent or obvious, especially in the buy vs. build conversation. **Doug Whitley: **And speaking of buy vs. build, I think there are always valid reasons to build. But one of the big things that happens when you buy is you get to benefit from everyone who's come before. All of the problems you’re going to face when you build it on your own, we’ve already solved for other clients. We have experience and generalized knowledge. We can say, "Oh honestly, the day-to-day problems in context architecture that we deal with break down into roughly 20 categories or so." So none of it is new to us. And maybe you have a truly novel use case, but you're still going to benefit from the fact that we've trained on data, looked at data, and considered all of the edge cases that everyone else has. You have to earn those knowledge bits by fighting with them. That's something we already have through the work we’ve done with others. **Phoebe Sajor: **I mean, why do it yourself when you could have Doug and Ash do it for you?