{"slug": "your-agent-s-memory-has-no-idea-where-it-was", "title": "Your Agent's Memory Has No Idea Where It Was", "summary": "A developer spent two days adding a source-location axis to Vodou's memory store, enabling it to track where each memory came from via source_url, source_host, and source_ref columns, plus a four-tier match endpoint and a browser extension. The project uncovered four defects invisible to unit tests, including a timestamp granularity bug that rendered a 30-minute-old fact as '18h old' and missing source URLs for Wikipedia clips.", "body_md": "Every agent memory system I have read stores *what* was said and *when*. Almost none store *where*. The chunk knows its text, its embedding, a created_at, maybe a session id. It does not know that you learned it on a Wikipedia article, or typed it into a vendor onboarding form, or that it came out of a ChatGPT conversation you had on a specific page.\n\nThat missing axis costs you more than it looks like it should. Without it you cannot answer \"what do I already know about the thing in front of me,\" you cannot forget one source without hunting chunk ids, and you cannot tell a user which of your stored facts came from their bank's website — which, the moment you have a browser extension, is the question that decides whether they trust you at all.\n\nI spent two days adding that axis to Vodou's memory store. Here is what it actually took, including the four defects that every unit test in the repo said did not exist.\n\nThree columns on the memory chunk: `source_url`\n\n, `source_host`\n\n, `source_ref`\n\n. Then a match endpoint that, given a URL, returns what memory holds about it in four tiers.\n\n```\n// MCP-servers/Vodou-Console/src/api/page-match.ts — the tiers\n// T1  exact page   facts whose source_url matches this URL\n// T2  this site    facts from the same host\n// T3  semantic     facts that match the page's title, not its URL\n// T4  documents    Library docs saved from this page\n```\n\nOn top of that: a browser extension that colors its own toolbar icon green when the current page has memory *from* it, a \"Note about this page\" box, insert-anywhere, typing suggestions, form fill from memory with a review card, and a per-site governance model that decides whether Vodou is allowed to look at the tab at all.\n\n**Diagram — What happens when you switch tabs**\n\nA URL enters the gateway, is checked against the per-site mode, then fans out to four memory tiers and back to the panel\n\n``` php\n  [Tab activates] --> [Site mode (fixed)]\n  [Site mode (fixed)] --suggest / collect--> [page_probe]\n  [Site mode (fixed)] --off → nothing, no daemon ask--> [Icon drawn green (fixed)]\n  [page_probe] --> [T1 exact page]\n  [page_probe] --> [T2 this site]\n  [page_probe] --> [T3 semantic (title)]\n  [page_probe] --> [T4 documents]\n  [T1 exact page] --exact + docs only--> [Icon drawn green (fixed)]\n  [T4 documents] --> [Icon drawn green (fixed)]\n\n  notes:\n    Tab activates: url + title only\n    Site mode: off / suggest / collect\n    page_probe: 30s cache per page key\n\n  The mode check happens before the daemon is asked anything. Off means the tab is never read.\n```\n\nThe interesting design constraint is the last edge. The icon is green for T1 and T4 only. That was not the first thing I built.\n\nI finished the unit suites. `page-match.test.ts`\n\n, `page-probe.test.ts`\n\n, `page-site-mode.test.ts`\n\n— all passing, plus four new route tests for the note and link endpoints. Then I loaded build .74 as an unpacked extension with the DevTools Network tab open and used it like a person for twenty minutes: a real ChatGPT capture, a Wikipedia clip, a saved article, the keyboard shortcut.\n\nFour defects. None of them visible to any test I had written.\n\n**A thirty-minute-old fact rendered as \"18h old.\"** The by-page query returned `COALESCE(valid_at, created_at)`\n\n, and for a fact extracted out of a daily log, `valid_at`\n\nis the log *day* at local midnight. So a fact written at 12:30pm carried an instant of 00:00:00 and the age formatter did the arithmetic correctly on a value that was never meant to be read at that precision. The fix is not in the query, it is in the renderer: day-granular instants now render \"today\" / \"yesterday\" / \"3d\", never hours. A timestamp with a granularity is not a timestamp.\n\n**Two Wikipedia clips landed with an empty source_url.** The right-click \"Send selection to Vodou memory\" handler sent the selected text and nothing else. The selection\n\n**The Library lane never stamped anything.** There are two ways a page becomes a document: `add_url`\n\nand `add_text`\n\n. `add_url`\n\nwrote `memory_sources.source_url`\n\nand stamped its chunks. `add_text`\n\ndid not. \"Add this page to Vodou Library\" — the button an actual person clicks — routes through `add_text`\n\n. I had implemented the axis on the lane nobody uses. I mirrored it and hand-stamped doc #225 to match.\n\n**Then the fix broke the feature.** Once that 111-chunk article was correctly stamped, it flooded \"From this page.\" The panel filled with `# Title\\n\\nSource:`\n\nfragments — raw chunk headers, 111 of them — and buried the four facts I had actually clipped. Correct data, useless surface. Fact tiers now exclude `doc:%`\n\nentirely, and documents get their own list that shows each document once instead of each of its chunks.\n\n**Diagram (beforeafter)**\n\nBefore: green unit suites hid four defects. After: live unpacked load surfaced all four.\n\n```\n  BEFORE — Unit suites, all green\n    - page-match tiers correct\n    - page_probe caching correct\n    - site-mode enforcement correct\n    - 4 route tests on note + link\n\n  AFTER — Twenty minutes on real pages\n    - 30-min fact rendered as 18h old\n    - selection clips stored with no URL\n    - the Library lane users click never stamped\n    - fixing that flooded the panel with 111 chunks\n```\n\nEvery one of these lived in the gap between \"the function returns the right value\" and \"a person got what they came for.\" The tests asserted the first. Nothing asserted the second.\n\nThe icon was supposed to be the easy part. Show the user that this page has memory. I built four versions in one evening and threw away three.\n\nA **text badge** with a count: Chrome's badge overlay is fixed-size and ate a 16px icon. Unreadable. A **corner dot**: too small to notice, and it does not say what it means. A **pulse**: motion in a browser toolbar reads as an unending alert — you notice it once and then it is a thing you are ignoring forever. The one that shipped is the icon itself drawn green, redrawn per tab on a transparent OffscreenCanvas from our own artwork.\n\nThat last one had its own trap. Hue-shifting the brand-blue pixels works. A flat `source-atop`\n\nfill does not — it paints over the white eyes in the logo and you get a green blob. `MARK_MODE`\n\nstill keeps `dot`\n\nand `pulse`\n\none line away, because I do not trust that I am done being wrong about this.\n\nThe harder question was not *how* to draw it but *when*. My first cut lit the icon for any tier hit. Two failures, both obvious the moment I used it:\n\nGreen means T1 and T4 only: facts and documents saved *from this page*. That is a claim narrow enough to always be true. An indicator that fires on a weak match is worse than no indicator, because a user calibrates on it once and then never looks again.\n\nThe part I underestimated by the widest margin: this is a browser extension that reads the URL of every tab you open. That is not a feature detail. That is the whole trust question.\n\nThe gateway is the authority, not the extension. Site mode is `off`\n\n/ `suggest`\n\n/ `collect`\n\n. Banks, health portals, tax sites and sign-in hosts default to **off** — Vodou does not read the tab there until the user says otherwise. Resolution order is user rule, then sensitive default, then global default, and a rule covers subdomains.\n\nOff is enforced in four places, not one: `page-match`\n\nreturns empty without asking the daemon, `page_probe`\n\nreturns no icon, and `note`\n\nand `link`\n\nboth 403 unless the mode is `collect`\n\n. The extension-side typing suggestions ask the mode *first* and send nothing until it answers. Four tests on the resolver, four on enforcement.\n\nThen `mem forget --host`\n\n, which is soft (sets `invalid_at`\n\n), covers subdomains, and counts Library documents rather than removing them. First panel click is a dry run that shows you the count; a second click within 8 seconds confirms. Verified live on w3schools.com: hide, 1, undo, 0.\n\nThe QA plan for all of this was two-phase and network-observable, which is the only kind of privacy test I now believe:\n\n```\nPhase A  page memory OFF  → switch tabs → assert ZERO page-match requests\nPhase B  page memory ON   → accept disclosure → assert requests appear,\n                             and the fact walnut-lantern-4417 shows under\n                             \"From this page\" in the ChatGPT tab\n```\n\nPhase A is the important one. It does not test that the feature works. It tests that the feature is *absent*, from the network panel, where a skeptical user would look.\n\n**Your privacy guarantee is a claim about the wire, so test it on the wire.**\n\nNearly every agent system I have seen tests its permission model by asserting the return value of a permission function. That is testing your intent. The user's actual question is \"did you send my data anywhere,\" and the answer to that lives in the network panel, not in your unit suite. Write the test that asserts zero requests. It is a strange test to write — most test frameworks make asserting a *non-event* awkward — and it is the only one that answers the question that was asked.\n\nTwo corollaries, both of which cost me a defect each:\n\n**When you add a field, find every write path, not the one you were looking at.** `add_url`\n\nand `add_text`\n\nboth create documents. I stamped one. The one I missed was the one behind the button. Grep for every writer of the table, not every caller of the function you happened to open.\n\n**A stored timestamp has a granularity, and the granularity does not travel with it.** A `datetime`\n\ncolumn looks second-precise. A value derived from a log *day* is not. If you render it at a precision the data never had, you produce a confident lie — \"18h ago\" for something thirty minutes old. Carry the granularity, or render at the coarsest one you can guarantee.\n\nThe public discussion of agent memory is almost entirely about *tiers*. The [engineering playbook's agent-memory writeup](https://engineering-playbook.vercel.app/agentic/agent-memory) splits long-term memory into episodic, semantic and procedural. [Geodocs' pattern spec](https://geodocs.dev/ai-agents/agent-memory-pattern-spec) adds working memory on top and specifies consolidation, scoring, eviction and write-time PII redaction. Both are good. Both are about *what kind of thing* a memory is.\n\nNeither has an axis for *where the user was standing when it entered the store*. That is not an oversight in the taxonomy so much as an artifact of the assumption underneath it: that memory arrives through your chat interface. Once memory can be captured from any page in a browser, provenance stops being metadata and becomes the primary key for two operations you cannot otherwise perform — \"show me what you know about this\" and \"forget this source.\"\n\nThe [Pockit production-memory guide](https://pockit.tools/blog/ai-agent-memory-architecture-production-guide/) is right that context windows are not memory and that persistence is the wall everyone hits. But durability cuts both ways, which is the point of [\"From Faulty Memories to Corrected Actions\"](https://arxiv.org/html/2608.10502): persistence makes errors durable too, and deleting a bad memory leaves everything already derived from it in place. Their answer is dependency-guided rollback. My cheaper, dumber version of the same insight is that if every chunk carries its origin, \"forget everything that came from this host\" is one soft update instead of an archaeology project. Provenance is not just retrieval — it is the handle you need to undo.\n\nWorth reading alongside these: [agentmemory#648](https://github.com/rohitg00/agentmemory/pull/648), a stability pass where two separate bugs turned out to share one root cause — a daemon wrapper surviving `stop`\n\n— and the [dawnai memory-tooling commit](https://github.com/cacheplane/dawnai/commit/4e3e020c969dd9b1ad9f10c374d2dfd5105e82c0), which adds a gated live-smoke suite for exactly the reason I am writing this post: some behaviors only appear against a real backend, so they gate a separate suite on a real key and keep it out of CI.\n\nThe stamping path has a known conflict. `mem page-link`\n\nwrites the column directly, but re-indexing uses a `COALESCE`\n\nthat prefers a `page:`\n\ntoken found in the chunk's own text. So a chunk whose text carries its own page token will revert to that token on the next sync, overriding the link. I know about it, it is in the plan, it is not fixed.\n\nForm fill is two lanes and only one of them is deterministic. Learn-back — answers you accepted on this page before, or on this site — is exact and ordered, latest wins. Everything past that is one LLM call with a hard \"only what memory supports, never invent\" rule, which is a rule you write down and hope for. The review card exists because I do not trust that lane. Nothing is written to the page until you tick it, and it never submits the form.\n\nSensitive-host defaults are a list. Lists are wrong at the edges. A bank I have never heard of collects by default until somebody notices, and \"somebody notices\" is not a security model.\n\nAnd the whole thing rests on one number I cannot measure from here: how many of those four live-only defects still exist in the paths I did not manually click for twenty minutes. Every unit suite in this repo is green. That was true before I opened the browser, too.\n\nSource: [Your Agent's Memory Has No Idea Where It Was](https://blog.vodou.ai/page-axis-agent-memory/) by Chad Priest, from the \"Building Vodou in Public\" series.", "url": "https://wpnews.pro/news/your-agent-s-memory-has-no-idea-where-it-was", "canonical_source": "https://dev.to/chad_4ba286df4a1dc781a641/your-agents-memory-has-no-idea-where-it-was-3pad", "published_at": "2026-08-26 17:35:14+00:00", "updated_at": "2026-08-26 17:44:38.604781+00:00", "lang": "en", "topics": ["ai-agents", "developer-tools", "ai-infrastructure"], "entities": ["Vodou", "ChatGPT", "Wikipedia"], "alternates": {"html": "https://wpnews.pro/news/your-agent-s-memory-has-no-idea-where-it-was", "markdown": "https://wpnews.pro/news/your-agent-s-memory-has-no-idea-where-it-was.md", "text": "https://wpnews.pro/news/your-agent-s-memory-has-no-idea-where-it-was.txt", "jsonld": "https://wpnews.pro/news/your-agent-s-memory-has-no-idea-where-it-was.jsonld"}}