{"slug": "my-agent-s-memory-contradicted-itself-and-nobody-noticed-i-built-a-read-only-cli", "title": "My agent's memory contradicted itself and nobody noticed. I built a read-only audit CLI.", "summary": "A developer built memrot, an MIT-licensed read-only CLI that audits agent memory stored in Relate, Neo4j's embedded graph database for AI agents, rather than trusting it. The tool runs seven deterministic checks inside the store — including contradictions between current facts, superseded facts still served, stale facts, missing provenance and near-duplicates — and returns exit codes (0 clean, 1 error findings, 2 usage/store error) usable directly in CI. On a seeded demo memory with seven planted faults and three clean controls, the audit reported exactly 9 findings, 5 errors and 4 warnings.", "body_md": "Last week I asked a simple question about an agent's memory: at one point in February, what did it believe the database engine was? The answer came back with two values at once. Postgres 15 and Postgres 16. Both marked current. Both believed.\n\nThe agent had not reasoned badly. Its memory was rotten, and nothing in the loop ever checked the store itself. A limit changes and both values stay current. An upgrade supersedes a fact that is never marked superseded. A TTL expires and nobody notices. A fact arrives with no record of where it came from. The agent then acts on stale or contradictory memory with full confidence, and the failure looks like a reasoning bug when it is really a data bug.\n\nI built a small tool that audits the memory instead of trusting it. It is called [memrot](https://github.com/techdevfun/memrot), and it is a read-only CLI over a Relate store (Neo4j's embedded graph database for AI agents). MIT licensed. No server, no LLM, no network, no writes to the store it audits.\n\nI re-verified the rivals live on 13 November 2026 before writing a word of this:\n\nEvery rival stores memory. None of them lets an operator open a store read-only, ask \"what in here is wrong\", and get a machine-checkable answer with an exit code. That is the wedge, and memrot is the audit layer they do not have.\n\nmemrot runs seven deterministic checks, each one a Cypher query or a vector search executed inside Relate, not application-side logic:\n\n| # | Check | Rule | Severity | \n|---|---|---|---|\n| 1 | Contradiction | Two `current` facts, same subject + predicate, different object, overlapping validity windows | error | \n| 2 | Superseded-but-served | A fact superseded by another `current` fact, still marked`current` | error | \n| 3 | Stale | A `current` fact past its`validTo` , or older than`--stale-days` (default 90) with no corroboration and no newer fact on the same subject + predicate | warning | \n| 4 | Orphaned entity | An `Entity` with no relationships at all | warning | \n| 5 | Provenance-less fact | A `Fact` with no`FROM_EPISODE` edge: no record of where it came from | error | \n| 6 | Validity-order violation | `validFrom` after`validTo` , or a superseding fact created before the fact it supersedes | error | \n| 7 | Near-duplicate (lexical) | Two facts whose trigram embeddings score at or above 0.91: the same claim recorded twice under different wording | warning | \n\nThe answer comes back as an exit code a shell or CI step can use directly: **0 = clean, 1 = error-severity findings, 2 = usage error or store error**. Warnings alone do not fail the audit.\n\n```\nmemrot seed demo.rl\nmemrot audit demo.rl\nmemrot audit demo.rl --format json\nmemrot audit demo.rl --as-of 2026-03-01 --stale-days 30\nmemrot believed-at demo.rl --at 2026-02-15\nmemrot believed-at demo.rl --at 2026-02-15 --subject database\nmemrot why demo.rl f-db-15\n```\n\n`believed-at` is time travel: what did the agent believe at this moment. `why` shows a fact's provenance, its supporters, its contradictions, and the fact that supersedes it.\n\nThe GIF above is a real terminal session, not a mockup. `memrot seed` writes a small fictional migration memory: 11 facts, 6 entities, 8 episodes, 7 planted faults and 3 clean controls. The audit finds exactly 9 findings, 5 errors and 4 warnings:\n\n``` bash\n$ memrot audit examples/demo.rl --as-of 2026-11-13\n\nmemrot audit: examples/demo.rl (as of 2026-11-13)\n\n- Contradictory facts: 2 finding(s) [error]\n  ERROR  database.engine has two current values in overlapping windows: \"Postgres 15\" (f-db-15) and \"Postgres 16\" (f-db-16).\n  ERROR  queue.max_payload has two current values in overlapping windows: \"4000 bytes\" (f-queue-4000) and \"8000 bytes\" (f-queue-8000).\n- Superseded facts still served: 1 finding(s) [error]\n  ERROR  database.engine = \"Postgres 15\" (f-db-15) was superseded by f-db-16 but is still marked current, so it can still be served.\n- Stale facts: 2 finding(s) [warning]\n  WARN  cache.ttl = \"300s\" (f-cache-ttl) expired on 2026-06-01 and is still current. It is also old (recorded 2026-01-05T10:00:00Z) with no corroboration and no newer fact on the same subject and predicate.\n  WARN  feature.flag = \"on\" (f-feature-flag) expired on 2026-04-01 and is still current. It is also old (recorded 2026-05-02T09:30:00Z) with no corroboration and no newer fact on the same subject and predicate.\n- Orphaned entities: 1 finding(s) [warning]\n  WARN  Entity \"LegacyCron\" (e-legacycron, kind service) has no relationships: nothing references it and it references nothing.\n- Provenance-less facts: 1 finding(s) [error]\n  ERROR  deploy.region = \"eu-west-1\" (f-deploy-region) has no episode: there is no record of where this fact came from.\n- Validity-order violations: 1 finding(s) [error]\n  ERROR  feature.flag = \"on\" (f-feature-flag) has validFrom 2026-05-01 after validTo 2026-04-01: the validity window is empty.\n- Near-duplicate facts (lexical): 1 finding(s) [warning]\n  WARN  database.engine = \"Postgres 16\" (f-db-16) and database.db_engine = \"PostgreSQL 16\" (f-db-16b) are near-duplicates (lexical cosine 0.93): the same claim recorded twice under different wording.\n\n9 finding(s): 5 error(s), 4 warning(s)\naudit: FAILED (exit 1)\n```\n\nTime travel on the same store shows the contradiction was there from the start:\n\n``` bash\n$ memrot believed-at examples/demo.rl --at 2026-02-15\n\n  cache.ttl = \"300s\" (f-cache-ttl)\n  database.db_engine = \"PostgreSQL 16\" (f-db-16b)\n  database.engine = \"Postgres 15\" (f-db-15)\n  database.engine = \"Postgres 16\" (f-db-16)\n  queue.max_payload = \"8000 bytes\" (f-queue-8000)\n\n5 fact(s) believed at 2026-02-15.\n```\n\nIn February the agent already believed both Postgres 15 and Postgres 16. The March payload change is not believed yet. And `memrot why demo.rl f-db-15` traces the bad fact back to its episode (\"DB still on Postgres 15\", session 2026-02-01), names its superseder (f-db-16) and its contradiction (f-db-16: \"Postgres 16\").\n\nEvery check is asserted by exact fact id in the test suite (15/15 tests pass, against seeded stores with known ground truth), so the audit is proved by CI on every push, not just by the demo.\n\nThe npm package is not published yet. The name `memrot` is free on npm, but publishing is blocked on the owner's npm token, so for now install from source:\n\n```\ngit clone https://github.com/techdevfun/memrot.git\ncd memrot\nnpm install\nnpm run build\n```\n\nRequires Node.js 20 or newer. The only runtime dependency is `@relate/relate` 1.0.1, pinned exact. A clean install from source was measured on 13 November 2026 at about 8 seconds (`npm install` + `npm run build`, no cache). Run it from the repo root, or `npm link` to make `memrot` available everywhere.\n\nOne thing to know before you audit a live agent: memrot opens the store read-only and never writes to it, but Relate has a lock rule I verified live. A `.rl` file held by a running writer cannot be opened read-only, and `relate clone` cannot copy it either. Both fail, and memrot prints the remedy instead of the raw error. A directory store can be opened read-only beside a live writer. So: for a `.rl` file, stop the agent first, then audit; or snapshot it after the writer exits and audit the snapshot.\n\nAlso worth knowing: the `relate` CLI ships with telemetry enabled by default. memrot itself sends nothing (its CI installs only the `@relate/relate` library, never the CLI), but if you install the relate CLI, run `relate telemetry off` first.\n\nHonesty first: memrot is the audit layer, not a replacement memory.\n\nIf you need semantic judgement or retrieval, pick a rival. The README says so, and means it. If you need a standalone answer to \"what in this store is wrong, and can CI fail on it?\", try memrot.\n\n`memrot index` command to build embeddings on a copy of a store.\nA single `.rl` file is the whole point: a memory an agent carries as one file, auditable anywhere, no server to deploy. Directory mode exists for live auditing beside a writer; `:memory:` mode exists for tests. The audit runs fully offline: an audit that needs a model call cannot run in CI on every commit.\n\nIf your agent has persistent memory, run one audit before the next bad decision:\n\nFeedback welcome in the repo issues. The roadmap (embedder-backed semantic near-duplicates, `memrot index`, `--fail-on warning`) is in the README.", "url": "https://wpnews.pro/news/my-agent-s-memory-contradicted-itself-and-nobody-noticed-i-built-a-read-only-cli", "canonical_source": "https://dev.to/techdevfun/my-agents-memory-contradicted-itself-and-nobody-noticed-i-built-a-read-only-audit-cli-n21", "published_at": "2026-11-13 12:12:00+00:00", "updated_at": "2026-11-13 12:22:25.990530+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "developer-tools", "mlops", "artificial-intelligence"], "entities": ["memrot", "Relate", "Neo4j", "techdevfun"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/my-agent-s-memory-contradicted-itself-and-nobody-noticed-i-built-a-read-only-cli", "markdown": "https://wpnews.pro/news/my-agent-s-memory-contradicted-itself-and-nobody-noticed-i-built-a-read-only-cli.md", "text": "https://wpnews.pro/news/my-agent-s-memory-contradicted-itself-and-nobody-noticed-i-built-a-read-only-cli.txt", "jsonld": "https://wpnews.pro/news/my-agent-s-memory-contradicted-itself-and-nobody-noticed-i-built-a-read-only-cli.jsonld"}}