Agent memory and questions to ask

A developer at cognee outlined a structured test for evaluating coding-agent memory systems, using two fictional projects, Atlas and Beacon, to check whether an agent retains a correction, preserves historical commands, distinguishes between projects, recognizes missing information, and traces answers to their source. The exercise instructs testers to isolate memory from repository files and web search, then diagnose failures by capturing what was stored, what retrieval returned, and what reached the model. The developer recommends repeating the current-command question after importing an older README to confirm ingestion order does not silently determine which instruction is treated as current.

Tell a coding agent that your project runs tests with npm test . Start a new session and ask it how to run the tests. It answers correctly. Good first result. Now tell it the project has migrated to pnpm. Start another session. Ask the same question. Does it give you the new command? Does it offer both? Can it explain which instruction is current? That second conversation is where I'd start evaluating an agent's memory. It gives you a small, concrete problem to debug before you feed the system months of conversations. Use an isolated test dataset with two projects, Atlas and Beacon. Keep repository files, web search, and automatic replay of previous conversations out of the test. You want to trace what the memory system contributes. In the first session, give the agent these facts: Maintainer note, September 1: Atlas runs its test suite with npm test. Beacon runs its test suite with npm test. Let the memory write finish. If ingestion happens in the background, wait for it to complete before moving on. Start a fresh session against the same persistent store and ask: How do I run the tests in Atlas? The expected answer is npm test . Check the retrieval trace too: the answer should be supported by the stored note. A plausible guess doesn't establish that memory worked. Now add a correction: Maintainer note, September 10: Atlas has migrated to pnpm. From September 10 onward, run its test suite with pnpm test. This replaces the previous npm test instruction for Atlas. Beacon's test command has not changed. Wait for that write to finish, then run each of the following questions in a separate fresh session. | Question | Expected behavior | |---|---| | How do I run Atlas tests on September 12? | Returns pnpm test . | | How were Atlas tests run on September 1? | Returns npm test , identified as the historical command. | | How do I run Beacon tests on September 12? | Returns npm test . | | Why did Atlas migrate to pnpm? | Says the notes don't provide a reason. | | Which note establishes Atlas's current command? | Identifies the September 10 maintainer note. | These check different things: retaining a correction, preserving history, distinguishing projects, recognizing missing information, and tracing an answer to its source. If your application only needs current state, treat historical recall as an optional requirement. When a question fails, capture three things: what was stored, what retrieval returned, and what actually reached the model. If the September 10 correction never made it into persistent storage, investigate the write path. Changing the retrieval prompt won't recover a missing record. If the correction is stored but retrieval returns only the September 1 note, inspect the query, project filters, and ranking logic. If retrieval returns both versions, check whether the prompt preserves their dates and source information. The model needs enough context to distinguish a current instruction from an older one. If the prompt contains that information and the answer is still wrong, you have a different failure to investigate. Keeping these stages visible makes the debugging much more specific. I'd also repeat the current-command question after importing an old Atlas README dated September 1. Import it after the correction. The answer should remain pnpm test : ingestion order shouldn't silently determine which instruction is current. You can run this exercise against a memory file, a relational table, a vector-backed store, or a knowledge graph. Each implementation has to decide how to represent the correction and retrieve the appropriate version. For example, an application might preserve both notes and attach an explicit supersession relationship. Another might maintain a current-state record alongside an event history. The right choice depends on whether the agent needs to answer questions about the present, the past, or both. At cognee, we work on memory systems that combine graph, vector, and relational storage. Our AI agent memory guide https://www.cognee.ai/agent-memory covers those architecture choices, the memory lifecycle, and evaluation in more detail. For your own agent, pick a fact that actually changes in its workflow: a project command or a customer preference, maybe a deployment decision. Write down what a correct answer should look like before and after the change, then keep those cases as regression checks when you change the memory system.