# Cognoscenti: A Benchmark for Trustworthy AI Memory > Source: > Published: 2026-09-02 15:18:04+00:00 **Cognoscenti** is a benchmarking tool for **cognitive memory systems** or [Emergent Memory Systems](https://achiral.ai/blog/emergent-memory-systems) like [Achiral](https://achiral.ai) that are ACT-R inspired rather than tools of persistence like vector or proximity-based databases, knowledge graphs, or semantic search systems with no concept of forgetting. The problem isn't how much is remembered, but what gets chosen as memory. Support the development of cognitive memory benchmarks and help shape the future of AI memory systems. [Become a sponsor](/Achiral-AI/cognoscenti/blob/main/SPONSORSHIP.md) and gain visibility among thousands of AI/ML developers building the next generation of memory architectures. [📢 View Sponsorship Tiers](/Achiral-AI/cognoscenti/blob/main/SPONSORSHIP.md) | [🤝 GitHub Sponsors](https://github.com/Achiral-AI) | [💬 Contact Us](mailto:sponsorships@achiral.ai) Cognoscenti is implemented in Rust and provides a comprehensive framework for benchmarking cognitive memory architectures. The implementation includes: **Workload Simulation**: Generate synthetic team interactions (founders, engineers, designers, customers, investors) over configurable time periods**Core Evaluation Dimensions**: Measure activation precision, selective forgetting, interference resistance, contextual recall, memory consolidation, adaptation, and efficiency**Metrics Collection**: Automated collection and analysis of benchmark metrics with statistical computations** Visualization**: Generate plots and export results to JSON/CSV for further analysis ``` # Clone the repository git clone cd cognoscenti # Build the project cargo build --release ``` Run the benchmark with default settings: ``` cargo run --release ``` Or customize the benchmark parameters: ``` cargo run --release -- \ --workload technical \ --duration-months 6 \ --retrieval-count 1000 \ --output-dir ./results ``` Each workload generates synthetic data patterns to test specific cognitive memory properties: `strategic` : High-level decision patterns with infrequent but high-importance memories (tests selective forgetting and activation precision)`technical` : Frequent repetitive interactions with evolving technical context (tests memory consolidation and adaptation)`creative` : Varied contextual patterns with cross-domain references (tests contextual recall and interference resistance)`episodic` : Short-lived interaction patterns requiring rapid forgetting (tests decay rates and junk activation)`analytical` : Precision-focused patterns with strict accuracy requirements (tests retrieval efficiency and top-k accuracy) The benchmark generates: `benchmark_results.json` : Complete metrics in JSON format`benchmark_metrics.csv` : Metrics in CSV format for easy analysis`metrics_overview.png` : Visual overview of benchmark results`benchmark_report.pdf` : Printable report with headline metrics Run the included JSONL benchmark items against the built-in local baseline: ``` cargo run --release -- \ --items-file examples/memory_tools.jsonl \ --systems local \ --top-k 3 \ --output-dir ./results/memory-tools ``` Compare multiple configured memory tools with the same items: ``` cargo run --release -- \ --items-file examples/memory_tools.jsonl \ --systems local,mem0,supermemory,zep,letta \ --top-k 3 \ --output-dir ./results/memory-tools ``` The eval runner always supports `local` . External tools use provider-specific environment variables: ``` export COGNOSCENTI_MEM0_INGEST_URL="https://..." export COGNOSCENTI_MEM0_RETRIEVE_URL="https://..." export COGNOSCENTI_MEM0_API_KEY="..." ``` Use the same pattern for `SUPERMEMORY` , `ZEP` , and `LETTA` . Provider endpoints receive JSON over `POST` ; ingest requests include `system` and `memories` , and retrieval requests include `system` , `query` , `context` , and `top_k` . The memory-tool eval generates: `memory_eval_results.json` : Per-system item results and aggregate metrics`memory_eval_metrics.csv` : Comparison-ready metric table`memory_eval_report.pdf` : Printable comparison report The first public organizational-memory run compares three reference architectures on the same JSONL workload: `rag` : lexical retrieval over stored memories.`agent-memory` : lexical retrieval plus project, domain, and conversation scope.`organic-memory` : scoped retrieval plus lifecycle signals for current, validated, reinforced, stale, superseded, and low-signal memories. Run date: 2026-08-02 Public report: [Organizational Memory Benchmark: RAG vs Agent Memory vs Organic Memory](https://achiral.ai/benchmarks/ai-memory-benchmark) Release notes: [2026-08-02 AI Memory Benchmark Run](/Achiral-AI/cognoscenti/blob/main/docs/releases/2026-08-02-ai-memory-benchmark.md) | System | Top-1 Accuracy | Recall@3 | Precision@3 | Distractor Activation | |---|---|---|---|---| | RAG reference baseline | 50.00% | 83.33% | 27.78% | 25.00% | | Agent-memory reference baseline | 75.00% | 100.00% | 33.33% | 38.89% | | Organic-memory reference baseline | 91.67% | 100.00% | 33.33% | 16.67% | Reproduce the run: ``` cargo run --release -- \ --items-file examples/organizational_memory.jsonl \ --systems rag,agent-memory,organic-memory \ --top-k 3 \ --output-dir results/organizational-memory-2026-08-02 ``` Artifacts: `examples/organizational_memory.jsonl` `results/organizational-memory-2026-08-02/README.md` `results/organizational-memory-2026-08-02/memory_eval_results.json` `results/organizational-memory-2026-08-02/memory_eval_metrics.csv` This run does not claim to benchmark Mem0, Zep, Letta, Glean, LangGraph, or any other vendor product. Vendor adapters exist through Cognoscenti's HTTP eval mode, but vendor results should be published only when endpoints, keys, seeds, commands, and raw outputs are recorded. If you use Cognoscenti or the public benchmark artifacts, please cite the repository and the benchmark report. GitHub will surface citation metadata from [ CITATION.cff](/Achiral-AI/cognoscenti/blob/main/CITATION.cff). Version DOI: [10.5281/zenodo.21995304](https://doi.org/10.5281/zenodo.21995304) Concept DOI: [10.5281/zenodo.21995303](https://doi.org/10.5281/zenodo.21995303) ``` @software{achiral_ai_2026_cognoscenti, author = {{Achiral AI}}, title = {Cognoscenti: A Benchmark for Trustworthy AI Memory}, year = {2026}, version = {0.2.1}, doi = {10.5281/zenodo.21995304}, url = {https://github.com/Achiral-AI/cognoscenti} } @techreport{achiral_ai_2026_ai_memory_benchmark, author = {{Achiral AI}}, title = {AI Memory Benchmark: What Makes Memory Trustworthy?}, year = {2026}, doi = {10.5281/zenodo.21995304}, url = {https://achiral.ai/benchmarks/ai-memory-benchmark}, note = {Public Cognoscenti benchmark report for the 2026-08-02 organizational-memory run} } ``` Zenodo archival metadata is maintained in [ .zenodo.json](/Achiral-AI/cognoscenti/blob/main/.zenodo.json). We welcome contributions from the community! Please see [CONTRIBUTING.md](/Achiral-AI/cognoscenti/blob/main/CONTRIBUTING.md) for guidelines on how to contribute to Cognoscenti. For detailed developer documentation, API reference, and architecture guides, see [DEVELOPERS.md](/Achiral-AI/cognoscenti/blob/main/DEVELOPERS.md). By participating in this project, you agree to abide by our [Code of Conduct](/Achiral-AI/cognoscenti/blob/main/CODE_OF_CONDUCT.md). To set up the development environment: ``` # Clone the repository git clone https://github.com/achiral/cognoscenti.git cd cognoscenti # Build and test cargo build cargo test cargo clippy cargo fmt ``` If you find a bug or have a feature request, please open an issue on GitHub. For security vulnerabilities, please see [SECURITY.md](/Achiral-AI/cognoscenti/blob/main/SECURITY.md) for our disclosure policy. Existing benchmarks primarily evaluate retention, long-context retrieval, or task completion. For example, LoCoMo, a traditional memory benchmarks, will primarily answer: Can the system retrieve a fact from this specific storage? Such a test would likely yield a high yes and a few nos—with persistence exhibiting near total recall. There is no benchmarking available for directly evaluating **selective forgetting**, **activation**, **contextual recall**, **memory strengthening**, or **interference resistance**. Cognoscenti instead asks: Should this memory come to mind right now? The benchmark is intended for ACT-R-inspired and other human-like memory architectures that emphasize selective retrieval, strengthening through reuse, contextual recall, and useful forgetting. In several ways, Cognoscenti is a benchmark for cognitive memory systems inspired by human memory architectures such as ACT-R. A new benchmark specifically for cognitive memory covers: - Activation precision (did the most relevant memories surface?) - Retrieval efficiency (how many memories had to be examined?) - Forgetting quality (were obsolete or irrelevant memories ignored?) - Interference resistance (can similar memories be distinguished?) - Memory strengthening (does repeated information become easier to retrieve?) - Temporal adaptation (does new information appropriately replace old assumptions?) - Contextual recall (does retrieval change appropriately with project or conversational context?) This would highlight what makes an ACT-R-inspired memory layer fundamentally different from a persistent vector store or RAG system. In other words, one wouldn't simply abandon LoCoMo–use it for regression test long-term recall, but if your goal is to demonstrate that an ACT-R system behaves more like human cognition than a database (persistence), then you'll likely need a new benchmark centered on selective remembering, useful forgetting, and efficient activation, because no current benchmark directly evaluates those properties. | Benchmark | Best for | Good fit for ACT-R? | |---|---|---| | LongMemEval | Long-context memory retrieval | Moderate | | Needle-in-a-Haystack | Retrieval under huge contexts | Low | | InfiniteBench | Scaling to very long contexts | Low | | BABILong | Reasoning over long contexts | Moderate | | τ-bench | Stateful agent tasks over time | High | | SWE-bench | Long-running engineering tasks | High (if your users are developers) | Unlike traditional memory benchmarks that focus on **persistence and total recall**, Cognoscenti evaluates **cognitive properties** that mirror human memory systems: **Selective Forgetting**: Measures how well the system identifies and fades irrelevant information** Activation Dynamics**: Evaluates whether the most relevant memories surface at the right time** Contextual Adaptation**: Tests retrieval accuracy across different conversational and project contexts** Memory Consolidation**: Measures how repeated access strengthens memory retrieval** Interference Resistance**: Evaluates ability to distinguish between similar memories While existing benchmarks answer "Can you retrieve this fact?", Cognoscenti asks "Should this memory come to mind right now?" — making it uniquely suited for evaluating ACT-R-inspired and other emergent memory systems. Does the most relevant memory become active when needed? - Top-1 retrieval accuracy - Irrelevant memory activation - Retrieval latency Does unimportant information fade? - Junk activation rate - Retrieval precision - Forgotten-memory ratio Can similar memories be distinguished? - Top-1 / Top-3 accuracy - Distractor count - Retrieval confidence Does context influence retrieval? Examples include switching between engineering, marketing, finance, or design discussions. Do repeated experiences become easier to retrieve over time? Does newer information correctly supersede outdated beliefs while preserving historical context? Does retrieval remain selective and fast as total memory grows? Possible metrics: - Memories examined - Activated chunks - Retrieval latency - Token cost Rather than isolated question-answer pairs, simulate long-running teams: - Founders - Engineers - Designers - Customers - Investors Generate months of meetings, chats, documents, roadmaps, bugs, and changing project priorities. LoCoMo remains useful as a regression test for long-term recall. Cognoscenti complements it by measuring whether an AI behaves like an experienced teammate rather than an archive. Persistent memory asks: "Can you retrieve this fact?" Cognoscenti asks: "Should this memory come to mind right now?" The long-term goal is to establish Cognoscenti as a standard benchmark for evaluating cognitive memory systems inspired by human memory architectures such as ACT-R. Cognoscenti is licensed under the Apache License, Version 2.0. See `LICENSE` and `NOTICE` . If you use Cognoscenti in your research, please cite: ``` @software{cognoscenti2024, title = {Cognoscenti: A Benchmark for Cognitive Memory Systems}, author = {Achiral}, year = {2024}, url = {https://github.com/Achiral-AI/cognoscenti} } ``` Cognoscenti is inspired by cognitive memory architectures such as ACT-R and aims to complement existing memory benchmarks like LoCoMo by evaluating properties unique to human-like memory systems.