Every agent memory system I have read stores what was said and when. Almost none store where. The chunk knows its text, its embedding, a created_at, maybe a session id. It does not know that you learned it on a Wikipedia article, or typed it into a vendor onboarding form, or that it came out of a ChatGPT conversation you had on a specific page.
That missing axis costs you more than it looks like it should. Without it you cannot answer "what do I already know about the thing in front of me," you cannot forget one source without hunting chunk ids, and you cannot tell a user which of your stored facts came from their bank's website — which, the moment you have a browser extension, is the question that decides whether they trust you at all.
I spent two days adding that axis to Vodou's memory store. Here is what it actually took, including the four defects that every unit test in the repo said did not exist.
Three columns on the memory chunk: source_url
, source_host
, source_ref
. Then a match endpoint that, given a URL, returns what memory holds about it in four tiers.
// MCP-servers/Vodou-Console/src/api/page-match.ts — the tiers
// T1 exact page facts whose source_url matches this URL
// T2 this site facts from the same host
// T3 semantic facts that match the page's title, not its URL
// T4 documents Library docs saved from this page
On top of that: a browser extension that colors its own toolbar icon green when the current page has memory from it, a "Note about this page" box, insert-anywhere, typing suggestions, form fill from memory with a review card, and a per-site governance model that decides whether Vodou is allowed to look at the tab at all.
Diagram — What happens when you switch tabs
A URL enters the gateway, is checked against the per-site mode, then fans out to four memory tiers and back to the panel
[Tab activates] --> [Site mode (fixed)]
[Site mode (fixed)] --suggest / collect--> [page_probe]
[Site mode (fixed)] --off → nothing, no daemon ask--> [Icon drawn green (fixed)]
[page_probe] --> [T1 exact page]
[page_probe] --> [T2 this site]
[page_probe] --> [T3 semantic (title)]
[page_probe] --> [T4 documents]
[T1 exact page] --exact + docs only--> [Icon drawn green (fixed)]
[T4 documents] --> [Icon drawn green (fixed)]
notes:
Tab activates: url + title only
Site mode: off / suggest / collect
page_probe: 30s cache per page key
The mode check happens before the daemon is asked anything. Off means the tab is never read.
The interesting design constraint is the last edge. The icon is green for T1 and T4 only. That was not the first thing I built.
I finished the unit suites. page-match.test.ts
, page-probe.test.ts
, page-site-mode.test.ts
— all passing, plus four new route tests for the note and link endpoints. Then I loaded build .74 as an unpacked extension with the DevTools Network tab open and used it like a person for twenty minutes: a real ChatGPT capture, a Wikipedia clip, a saved article, the keyboard shortcut.
Four defects. None of them visible to any test I had written.
A thirty-minute-old fact rendered as "18h old." The by-page query returned COALESCE(valid_at, created_at)
, and for a fact extracted out of a daily log, valid_at
is the log day at local midnight. So a fact written at 12:30pm carried an instant of 00:00:00 and the age formatter did the arithmetic correctly on a value that was never meant to be read at that precision. The fix is not in the query, it is in the renderer: day-granular instants now render "today" / "yesterday" / "3d", never hours. A timestamp with a granularity is not a timestamp.
Two Wikipedia clips landed with an empty source_url. The right-click "Send selection to Vodou memory" handler sent the selected text and nothing else. The selection
The Library lane never stamped anything. There are two ways a page becomes a document: add_url
and add_text
. add_url
wrote memory_sources.source_url
and stamped its chunks. add_text
did not. "Add this page to Vodou Library" — the button an actual person clicks — routes through add_text
. I had implemented the axis on the lane nobody uses. I mirrored it and hand-stamped doc #225 to match.
Then the fix broke the feature. Once that 111-chunk article was correctly stamped, it flooded "From this page." The panel filled with # Title\n\nSource:
fragments — raw chunk headers, 111 of them — and buried the four facts I had actually clipped. Correct data, useless surface. Fact tiers now exclude doc:%
entirely, and documents get their own list that shows each document once instead of each of its chunks.
Diagram (beforeafter)
Before: green unit suites hid four defects. After: live unpacked load surfaced all four.
BEFORE — Unit suites, all green
- page-match tiers correct
- page_probe caching correct
- site-mode enforcement correct
- 4 route tests on note + link
AFTER — Twenty minutes on real pages
- 30-min fact rendered as 18h old
- selection clips stored with no URL
- the Library lane users click never stamped
- fixing that flooded the panel with 111 chunks
Every one of these lived in the gap between "the function returns the right value" and "a person got what they came for." The tests asserted the first. Nothing asserted the second.
The icon was supposed to be the easy part. Show the user that this page has memory. I built four versions in one evening and threw away three.
A text badge with a count: Chrome's badge overlay is fixed-size and ate a 16px icon. Unreadable. A corner dot: too small to notice, and it does not say what it means. A pulse: motion in a browser toolbar reads as an unending alert — you notice it once and then it is a thing you are ignoring forever. The one that shipped is the icon itself drawn green, redrawn per tab on a transparent OffscreenCanvas from our own artwork.
That last one had its own trap. Hue-shifting the brand-blue pixels works. A flat source-atop
fill does not — it paints over the white eyes in the logo and you get a green blob. MARK_MODE
still keeps dot
and pulse
one line away, because I do not trust that I am done being wrong about this.
The harder question was not how to draw it but when. My first cut lit the icon for any tier hit. Two failures, both obvious the moment I used it:
Green means T1 and T4 only: facts and documents saved from this page. That is a claim narrow enough to always be true. An indicator that fires on a weak match is worse than no indicator, because a user calibrates on it once and then never looks again.
The part I underestimated by the widest margin: this is a browser extension that reads the URL of every tab you open. That is not a feature detail. That is the whole trust question.
The gateway is the authority, not the extension. Site mode is off
/ suggest
/ collect
. Banks, health portals, tax sites and sign-in hosts default to off — Vodou does not read the tab there until the user says otherwise. Resolution order is user rule, then sensitive default, then global default, and a rule covers subdomains.
Off is enforced in four places, not one: page-match
returns empty without asking the daemon, page_probe
returns no icon, and note
and link
both 403 unless the mode is collect
. The extension-side typing suggestions ask the mode first and send nothing until it answers. Four tests on the resolver, four on enforcement.
Then mem forget --host
, which is soft (sets invalid_at
), covers subdomains, and counts Library documents rather than removing them. First panel click is a dry run that shows you the count; a second click within 8 seconds confirms. Verified live on w3schools.com: hide, 1, undo, 0.
The QA plan for all of this was two-phase and network-observable, which is the only kind of privacy test I now believe:
Phase A page memory OFF → switch tabs → assert ZERO page-match requests
Phase B page memory ON → accept disclosure → assert requests appear,
and the fact walnut-lantern-4417 shows under
"From this page" in the ChatGPT tab
Phase A is the important one. It does not test that the feature works. It tests that the feature is absent, from the network panel, where a skeptical user would look.
Your privacy guarantee is a claim about the wire, so test it on the wire.
Nearly every agent system I have seen tests its permission model by asserting the return value of a permission function. That is testing your intent. The user's actual question is "did you send my data anywhere," and the answer to that lives in the network panel, not in your unit suite. Write the test that asserts zero requests. It is a strange test to write — most test frameworks make asserting a non-event awkward — and it is the only one that answers the question that was asked.
Two corollaries, both of which cost me a defect each:
When you add a field, find every write path, not the one you were looking at. add_url
and add_text
both create documents. I stamped one. The one I missed was the one behind the button. Grep for every writer of the table, not every caller of the function you happened to open.
A stored timestamp has a granularity, and the granularity does not travel with it. A datetime
column looks second-precise. A value derived from a log day is not. If you render it at a precision the data never had, you produce a confident lie — "18h ago" for something thirty minutes old. Carry the granularity, or render at the coarsest one you can guarantee.
The public discussion of agent memory is almost entirely about tiers. The engineering playbook's agent-memory writeup splits long-term memory into episodic, semantic and procedural. Geodocs' pattern spec adds working memory on top and specifies consolidation, scoring, eviction and write-time PII redaction. Both are good. Both are about what kind of thing a memory is.
Neither has an axis for where the user was standing when it entered the store. That is not an oversight in the taxonomy so much as an artifact of the assumption underneath it: that memory arrives through your chat interface. Once memory can be captured from any page in a browser, provenance stops being metadata and becomes the primary key for two operations you cannot otherwise perform — "show me what you know about this" and "forget this source."
The Pockit production-memory guide is right that context windows are not memory and that persistence is the wall everyone hits. But durability cuts both ways, which is the point of "From Faulty Memories to Corrected Actions": persistence makes errors durable too, and deleting a bad memory leaves everything already derived from it in place. Their answer is dependency-guided rollback. My cheaper, dumber version of the same insight is that if every chunk carries its origin, "forget everything that came from this host" is one soft update instead of an archaeology project. Provenance is not just retrieval — it is the handle you need to undo.
Worth reading alongside these: agentmemory#648, a stability pass where two separate bugs turned out to share one root cause — a daemon wrapper surviving stop
— and the dawnai memory-tooling commit, which adds a gated live-smoke suite for exactly the reason I am writing this post: some behaviors only appear against a real backend, so they gate a separate suite on a real key and keep it out of CI.
The stamping path has a known conflict. mem page-link
writes the column directly, but re-indexing uses a COALESCE
that prefers a page:
token found in the chunk's own text. So a chunk whose text carries its own page token will revert to that token on the next sync, overriding the link. I know about it, it is in the plan, it is not fixed.
Form fill is two lanes and only one of them is deterministic. Learn-back — answers you accepted on this page before, or on this site — is exact and ordered, latest wins. Everything past that is one LLM call with a hard "only what memory supports, never invent" rule, which is a rule you write down and hope for. The review card exists because I do not trust that lane. Nothing is written to the page until you tick it, and it never submits the form.
Sensitive-host defaults are a list. Lists are wrong at the edges. A bank I have never heard of collects by default until somebody notices, and "somebody notices" is not a security model.
And the whole thing rests on one number I cannot measure from here: how many of those four live-only defects still exist in the paths I did not manually click for twenty minutes. Every unit suite in this repo is green. That was true before I opened the browser, too.
Source: Your Agent's Memory Has No Idea Where It Was by Chad Priest, from the "Building Vodou in Public" series.