{"slug": "cognoscenti-a-benchmark-for-trustworthy-ai-memory", "title": "Cognoscenti: A Benchmark for Trustworthy AI Memory", "summary": "Achiral AI released Cognoscenti, a Rust-based benchmarking tool for cognitive memory systems, designed to evaluate how AI architectures select and forget memories rather than merely store them. The tool simulates team interactions across five workload types and measures dimensions such as activation precision, selective forgetting, and interference resistance, with support for comparing external memory tools like Mem0, Supermemory, Zep, and Letta. Achiral AI is seeking sponsors to support the benchmark's development and gain visibility among AI/ML developers.", "body_md": "**Cognoscenti** is a benchmarking tool for **cognitive memory systems** or [Emergent Memory Systems](https://achiral.ai/blog/emergent-memory-systems) like [Achiral](https://achiral.ai) that are ACT-R inspired rather than tools of persistence like vector or proximity-based databases, knowledge graphs, or semantic search systems with no concept of forgetting.\n\nThe problem isn't how much is remembered, but what gets chosen as memory.\n\nSupport the development of cognitive memory benchmarks and help shape the future of AI memory systems. [Become a sponsor](/Achiral-AI/cognoscenti/blob/main/SPONSORSHIP.md) and gain visibility among thousands of AI/ML developers building the next generation of memory architectures.\n\n[📢 View Sponsorship Tiers](/Achiral-AI/cognoscenti/blob/main/SPONSORSHIP.md) | [🤝 GitHub Sponsors](https://github.com/Achiral-AI) | [💬 Contact Us](mailto:sponsorships@achiral.ai)\n\nCognoscenti is implemented in Rust and provides a comprehensive framework for benchmarking cognitive memory architectures. The implementation includes:\n\n**Workload Simulation**: Generate synthetic team interactions (founders, engineers, designers, customers, investors) over configurable time periods**Core Evaluation Dimensions**: Measure activation precision, selective forgetting, interference resistance, contextual recall, memory consolidation, adaptation, and efficiency**Metrics Collection**: Automated collection and analysis of benchmark metrics with statistical computations** Visualization**: Generate plots and export results to JSON/CSV for further analysis\n\n```\n# Clone the repository\ngit clone <repository-url>\ncd cognoscenti\n\n# Build the project\ncargo build --release\n```\n\nRun the benchmark with default settings:\n\n```\ncargo run --release\n```\n\nOr customize the benchmark parameters:\n\n```\ncargo run --release -- \\\n  --workload technical \\\n  --duration-months 6 \\\n  --retrieval-count 1000 \\\n  --output-dir ./results\n```\n\nEach workload generates synthetic data patterns to test specific cognitive memory properties:\n\n`strategic`\n\n: High-level decision patterns with infrequent but high-importance memories (tests selective forgetting and activation precision)`technical`\n\n: Frequent repetitive interactions with evolving technical context (tests memory consolidation and adaptation)`creative`\n\n: Varied contextual patterns with cross-domain references (tests contextual recall and interference resistance)`episodic`\n\n: Short-lived interaction patterns requiring rapid forgetting (tests decay rates and junk activation)`analytical`\n\n: Precision-focused patterns with strict accuracy requirements (tests retrieval efficiency and top-k accuracy)\n\nThe benchmark generates:\n\n`benchmark_results.json`\n\n: Complete metrics in JSON format`benchmark_metrics.csv`\n\n: Metrics in CSV format for easy analysis`metrics_overview.png`\n\n: Visual overview of benchmark results`benchmark_report.pdf`\n\n: Printable report with headline metrics\n\nRun the included JSONL benchmark items against the built-in local baseline:\n\n```\ncargo run --release -- \\\n  --items-file examples/memory_tools.jsonl \\\n  --systems local \\\n  --top-k 3 \\\n  --output-dir ./results/memory-tools\n```\n\nCompare multiple configured memory tools with the same items:\n\n```\ncargo run --release -- \\\n  --items-file examples/memory_tools.jsonl \\\n  --systems local,mem0,supermemory,zep,letta \\\n  --top-k 3 \\\n  --output-dir ./results/memory-tools\n```\n\nThe eval runner always supports `local`\n\n. External tools use provider-specific environment variables:\n\n```\nexport COGNOSCENTI_MEM0_INGEST_URL=\"https://...\"\nexport COGNOSCENTI_MEM0_RETRIEVE_URL=\"https://...\"\nexport COGNOSCENTI_MEM0_API_KEY=\"...\"\n```\n\nUse the same pattern for `SUPERMEMORY`\n\n, `ZEP`\n\n, and `LETTA`\n\n. Provider endpoints receive JSON over `POST`\n\n; ingest requests include `system`\n\nand `memories`\n\n, and retrieval requests include `system`\n\n, `query`\n\n, `context`\n\n, and `top_k`\n\n.\n\nThe memory-tool eval generates:\n\n`memory_eval_results.json`\n\n: Per-system item results and aggregate metrics`memory_eval_metrics.csv`\n\n: Comparison-ready metric table`memory_eval_report.pdf`\n\n: Printable comparison report\n\nThe first public organizational-memory run compares three reference architectures on the same JSONL workload:\n\n`rag`\n\n: lexical retrieval over stored memories.`agent-memory`\n\n: lexical retrieval plus project, domain, and conversation scope.`organic-memory`\n\n: scoped retrieval plus lifecycle signals for current, validated, reinforced, stale, superseded, and low-signal memories.\n\nRun date: 2026-08-02\n\nPublic report: [Organizational Memory Benchmark: RAG vs Agent Memory vs Organic Memory](https://achiral.ai/benchmarks/ai-memory-benchmark)\n\nRelease notes: [2026-08-02 AI Memory Benchmark Run](/Achiral-AI/cognoscenti/blob/main/docs/releases/2026-08-02-ai-memory-benchmark.md)\n\n| System | Top-1 Accuracy | Recall@3 | Precision@3 | Distractor Activation |\n|---|---|---|---|---|\n| RAG reference baseline | 50.00% | 83.33% | 27.78% | 25.00% |\n| Agent-memory reference baseline | 75.00% | 100.00% | 33.33% | 38.89% |\n| Organic-memory reference baseline | 91.67% | 100.00% | 33.33% | 16.67% |\n\nReproduce the run:\n\n```\ncargo run --release -- \\\n  --items-file examples/organizational_memory.jsonl \\\n  --systems rag,agent-memory,organic-memory \\\n  --top-k 3 \\\n  --output-dir results/organizational-memory-2026-08-02\n```\n\nArtifacts:\n\n`examples/organizational_memory.jsonl`\n\n`results/organizational-memory-2026-08-02/README.md`\n\n`results/organizational-memory-2026-08-02/memory_eval_results.json`\n\n`results/organizational-memory-2026-08-02/memory_eval_metrics.csv`\n\nThis run does not claim to benchmark Mem0, Zep, Letta, Glean, LangGraph, or any other vendor product. Vendor adapters exist through Cognoscenti's HTTP eval mode, but vendor results should be published only when endpoints, keys, seeds, commands, and raw outputs are recorded.\n\nIf you use Cognoscenti or the public benchmark artifacts, please cite the repository and the benchmark report. GitHub will surface citation metadata from [ CITATION.cff](/Achiral-AI/cognoscenti/blob/main/CITATION.cff).\n\nVersion DOI: [10.5281/zenodo.21995304](https://doi.org/10.5281/zenodo.21995304)\n\nConcept DOI: [10.5281/zenodo.21995303](https://doi.org/10.5281/zenodo.21995303)\n\n```\n@software{achiral_ai_2026_cognoscenti,\n  author = {{Achiral AI}},\n  title = {Cognoscenti: A Benchmark for Trustworthy AI Memory},\n  year = {2026},\n  version = {0.2.1},\n  doi = {10.5281/zenodo.21995304},\n  url = {https://github.com/Achiral-AI/cognoscenti}\n}\n\n@techreport{achiral_ai_2026_ai_memory_benchmark,\n  author = {{Achiral AI}},\n  title = {AI Memory Benchmark: What Makes Memory Trustworthy?},\n  year = {2026},\n  doi = {10.5281/zenodo.21995304},\n  url = {https://achiral.ai/benchmarks/ai-memory-benchmark},\n  note = {Public Cognoscenti benchmark report for the 2026-08-02 organizational-memory run}\n}\n```\n\nZenodo archival metadata is maintained in [ .zenodo.json](/Achiral-AI/cognoscenti/blob/main/.zenodo.json).\n\nWe welcome contributions from the community! Please see [CONTRIBUTING.md](/Achiral-AI/cognoscenti/blob/main/CONTRIBUTING.md) for guidelines on how to contribute to Cognoscenti.\n\nFor detailed developer documentation, API reference, and architecture guides, see [DEVELOPERS.md](/Achiral-AI/cognoscenti/blob/main/DEVELOPERS.md).\n\nBy participating in this project, you agree to abide by our [Code of Conduct](/Achiral-AI/cognoscenti/blob/main/CODE_OF_CONDUCT.md).\n\nTo set up the development environment:\n\n```\n# Clone the repository\ngit clone https://github.com/achiral/cognoscenti.git\ncd cognoscenti\n\n# Build and test\ncargo build\ncargo test\ncargo clippy\ncargo fmt\n```\n\nIf you find a bug or have a feature request, please open an issue on GitHub. For security vulnerabilities, please see [SECURITY.md](/Achiral-AI/cognoscenti/blob/main/SECURITY.md) for our disclosure policy.\n\nExisting benchmarks primarily evaluate retention, long-context retrieval, or task completion. For example, LoCoMo, a traditional memory benchmarks, will primarily answer:\n\nCan the system retrieve a fact from this specific storage?\n\nSuch a test would likely yield a high yes and a few nos—with persistence exhibiting near total recall. There is no benchmarking available for directly evaluating **selective forgetting**, **activation**, **contextual recall**, **memory strengthening**, or **interference resistance**.\n\nCognoscenti instead asks:\n\nShould this memory come to mind right now?\n\nThe benchmark is intended for ACT-R-inspired and other human-like memory architectures that emphasize selective retrieval, strengthening through reuse, contextual recall, and useful forgetting. In several ways, Cognoscenti is a benchmark for cognitive memory systems inspired by human memory architectures such as ACT-R.\n\nA new benchmark specifically for cognitive memory covers:\n\n- Activation precision (did the most relevant memories surface?)\n- Retrieval efficiency (how many memories had to be examined?)\n- Forgetting quality (were obsolete or irrelevant memories ignored?)\n- Interference resistance (can similar memories be distinguished?)\n- Memory strengthening (does repeated information become easier to retrieve?)\n- Temporal adaptation (does new information appropriately replace old assumptions?)\n- Contextual recall (does retrieval change appropriately with project or conversational context?)\n\nThis would highlight what makes an ACT-R-inspired memory layer fundamentally different from a persistent vector store or RAG system.\n\nIn other words, one wouldn't simply abandon LoCoMo–use it for regression test long-term recall, but if your goal is to demonstrate that an ACT-R system behaves more like human cognition than a database (persistence), then you'll likely need a new benchmark centered on selective remembering, useful forgetting, and efficient activation, because no current benchmark directly evaluates those properties.\n\n| Benchmark | Best for | Good fit for ACT-R? |\n|---|---|---|\n| LongMemEval | Long-context memory retrieval | Moderate |\n| Needle-in-a-Haystack | Retrieval under huge contexts | Low |\n| InfiniteBench | Scaling to very long contexts | Low |\n| BABILong | Reasoning over long contexts | Moderate |\n| τ-bench | Stateful agent tasks over time | High |\n| SWE-bench | Long-running engineering tasks | High (if your users are developers) |\n\nUnlike traditional memory benchmarks that focus on **persistence and total recall**, Cognoscenti evaluates **cognitive properties** that mirror human memory systems:\n\n**Selective Forgetting**: Measures how well the system identifies and fades irrelevant information** Activation Dynamics**: Evaluates whether the most relevant memories surface at the right time** Contextual Adaptation**: Tests retrieval accuracy across different conversational and project contexts** Memory Consolidation**: Measures how repeated access strengthens memory retrieval** Interference Resistance**: Evaluates ability to distinguish between similar memories\n\nWhile existing benchmarks answer \"Can you retrieve this fact?\", Cognoscenti asks \"Should this memory come to mind right now?\" — making it uniquely suited for evaluating ACT-R-inspired and other emergent memory systems.\n\nDoes the most relevant memory become active when needed? - Top-1 retrieval accuracy - Irrelevant memory activation - Retrieval latency\n\nDoes unimportant information fade? - Junk activation rate - Retrieval precision - Forgotten-memory ratio\n\nCan similar memories be distinguished? - Top-1 / Top-3 accuracy - Distractor count - Retrieval confidence\n\nDoes context influence retrieval? Examples include switching between engineering, marketing, finance, or design discussions.\n\nDo repeated experiences become easier to retrieve over time?\n\nDoes newer information correctly supersede outdated beliefs while preserving historical context?\n\nDoes retrieval remain selective and fast as total memory grows? Possible metrics: - Memories examined - Activated chunks - Retrieval latency - Token cost\n\nRather than isolated question-answer pairs, simulate long-running teams: - Founders - Engineers - Designers - Customers - Investors\n\nGenerate months of meetings, chats, documents, roadmaps, bugs, and changing project priorities.\n\nLoCoMo remains useful as a regression test for long-term recall.\n\nCognoscenti complements it by measuring whether an AI behaves like an experienced teammate rather than an archive.\n\nPersistent memory asks: \"Can you retrieve this fact?\"\n\nCognoscenti asks: \"Should this memory come to mind right now?\"\n\nThe long-term goal is to establish Cognoscenti as a standard benchmark for evaluating cognitive memory systems inspired by human memory architectures such as ACT-R.\n\nCognoscenti is licensed under the Apache License, Version 2.0. See\n`LICENSE`\n\nand `NOTICE`\n\n.\n\nIf you use Cognoscenti in your research, please cite:\n\n```\n@software{cognoscenti2024,\n  title = {Cognoscenti: A Benchmark for Cognitive Memory Systems},\n  author = {Achiral},\n  year = {2024},\n  url = {https://github.com/Achiral-AI/cognoscenti}\n}\n```\n\nCognoscenti is inspired by cognitive memory architectures such as ACT-R and aims to complement existing memory benchmarks like LoCoMo by evaluating properties unique to human-like memory systems.", "url": "https://wpnews.pro/news/cognoscenti-a-benchmark-for-trustworthy-ai-memory", "canonical_source": "https://github.com/Achiral-AI/cognoscenti", "published_at": "2026-09-02 15:18:04+00:00", "updated_at": "2026-09-02 15:23:17.366262+00:00", "lang": "en", "topics": ["ai-research", "ai-tools", "ai-infrastructure"], "entities": ["Achiral AI", "Cognoscenti", "Mem0", "Supermemory", "Zep", "Letta"], "alternates": {"html": "https://wpnews.pro/news/cognoscenti-a-benchmark-for-trustworthy-ai-memory", "markdown": "https://wpnews.pro/news/cognoscenti-a-benchmark-for-trustworthy-ai-memory.md", "text": "https://wpnews.pro/news/cognoscenti-a-benchmark-for-trustworthy-ai-memory.txt", "jsonld": "https://wpnews.pro/news/cognoscenti-a-benchmark-for-trustworthy-ai-memory.jsonld"}}