Cognoscenti: A Benchmark for Trustworthy AI Memory

Achiral AI released Cognoscenti, a Rust-based benchmarking tool for cognitive memory systems, designed to evaluate how AI architectures select and forget memories rather than merely store them. The tool simulates team interactions across five workload types and measures dimensions such as activation precision, selective forgetting, and interference resistance, with support for comparing external memory tools like Mem0, Supermemory, Zep, and Letta. Achiral AI is seeking sponsors to support the benchmark's development and gain visibility among AI/ML developers.

Cognoscenti is a benchmarking tool for cognitive memory systems or Emergent Memory Systems https://achiral.ai/blog/emergent-memory-systems like Achiral https://achiral.ai that are ACT-R inspired rather than tools of persistence like vector or proximity-based databases, knowledge graphs, or semantic search systems with no concept of forgetting. The problem isn't how much is remembered, but what gets chosen as memory. Support the development of cognitive memory benchmarks and help shape the future of AI memory systems. Become a sponsor /Achiral-AI/cognoscenti/blob/main/SPONSORSHIP.md and gain visibility among thousands of AI/ML developers building the next generation of memory architectures. 📢 View Sponsorship Tiers /Achiral-AI/cognoscenti/blob/main/SPONSORSHIP.md | 🤝 GitHub Sponsors https://github.com/Achiral-AI | 💬 Contact Us mailto:sponsorships@achiral.ai Cognoscenti is implemented in Rust and provides a comprehensive framework for benchmarking cognitive memory architectures. The implementation includes: Workload Simulation : Generate synthetic team interactions founders, engineers, designers, customers, investors over configurable time periods Core Evaluation Dimensions : Measure activation precision, selective forgetting, interference resistance, contextual recall, memory consolidation, adaptation, and efficiency Metrics Collection : Automated collection and analysis of benchmark metrics with statistical computations Visualization : Generate plots and export results to JSON/CSV for further analysis Clone the repository git clone <repository-url cd cognoscenti Build the project cargo build --release Run the benchmark with default settings: cargo run --release Or customize the benchmark parameters: cargo run --release -- \ --workload technical \ --duration-months 6 \ --retrieval-count 1000 \ --output-dir ./results Each workload generates synthetic data patterns to test specific cognitive memory properties: strategic : High-level decision patterns with infrequent but high-importance memories tests selective forgetting and activation precision technical : Frequent repetitive interactions with evolving technical context tests memory consolidation and adaptation creative : Varied contextual patterns with cross-domain references tests contextual recall and interference resistance episodic : Short-lived interaction patterns requiring rapid forgetting tests decay rates and junk activation analytical : Precision-focused patterns with strict accuracy requirements tests retrieval efficiency and top-k accuracy The benchmark generates: benchmark results.json : Complete metrics in JSON format benchmark metrics.csv : Metrics in CSV format for easy analysis metrics overview.png : Visual overview of benchmark results benchmark report.pdf : Printable report with headline metrics Run the included JSONL benchmark items against the built-in local baseline: cargo run --release -- \ --items-file examples/memory tools.jsonl \ --systems local \ --top-k 3 \ --output-dir ./results/memory-tools Compare multiple configured memory tools with the same items: cargo run --release -- \ --items-file examples/memory tools.jsonl \ --systems local,mem0,supermemory,zep,letta \ --top-k 3 \ --output-dir ./results/memory-tools The eval runner always supports local . External tools use provider-specific environment variables: export COGNOSCENTI MEM0 INGEST URL="https://..." export COGNOSCENTI MEM0 RETRIEVE URL="https://..." export COGNOSCENTI MEM0 API KEY="..." Use the same pattern for SUPERMEMORY , ZEP , and LETTA . Provider endpoints receive JSON over POST ; ingest requests include system and memories , and retrieval requests include system , query , context , and top k . The memory-tool eval generates: memory eval results.json : Per-system item results and aggregate metrics memory eval metrics.csv : Comparison-ready metric table memory eval report.pdf : Printable comparison report The first public organizational-memory run compares three reference architectures on the same JSONL workload: rag : lexical retrieval over stored memories. agent-memory : lexical retrieval plus project, domain, and conversation scope. organic-memory : scoped retrieval plus lifecycle signals for current, validated, reinforced, stale, superseded, and low-signal memories. Run date: 2026-08-02 Public report: Organizational Memory Benchmark: RAG vs Agent Memory vs Organic Memory https://achiral.ai/benchmarks/ai-memory-benchmark Release notes: 2026-08-02 AI Memory Benchmark Run /Achiral-AI/cognoscenti/blob/main/docs/releases/2026-08-02-ai-memory-benchmark.md | System | Top-1 Accuracy | Recall@3 | Precision@3 | Distractor Activation | |---|---|---|---|---| | RAG reference baseline | 50.00% | 83.33% | 27.78% | 25.00% | | Agent-memory reference baseline | 75.00% | 100.00% | 33.33% | 38.89% | | Organic-memory reference baseline | 91.67% | 100.00% | 33.33% | 16.67% | Reproduce the run: cargo run --release -- \ --items-file examples/organizational memory.jsonl \ --systems rag,agent-memory,organic-memory \ --top-k 3 \ --output-dir results/organizational-memory-2026-08-02 Artifacts: examples/organizational memory.jsonl results/organizational-memory-2026-08-02/README.md results/organizational-memory-2026-08-02/memory eval results.json results/organizational-memory-2026-08-02/memory eval metrics.csv This run does not claim to benchmark Mem0, Zep, Letta, Glean, LangGraph, or any other vendor product. Vendor adapters exist through Cognoscenti's HTTP eval mode, but vendor results should be published only when endpoints, keys, seeds, commands, and raw outputs are recorded. If you use Cognoscenti or the public benchmark artifacts, please cite the repository and the benchmark report. GitHub will surface citation metadata from CITATION.cff /Achiral-AI/cognoscenti/blob/main/CITATION.cff . Version DOI: 10.5281/zenodo.21995304 https://doi.org/10.5281/zenodo.21995304 Concept DOI: 10.5281/zenodo.21995303 https://doi.org/10.5281/zenodo.21995303 @software{achiral ai 2026 cognoscenti, author = {{Achiral AI}}, title = {Cognoscenti: A Benchmark for Trustworthy AI Memory}, year = {2026}, version = {0.2.1}, doi = {10.5281/zenodo.21995304}, url = {https://github.com/Achiral-AI/cognoscenti} } @techreport{achiral ai 2026 ai memory benchmark, author = {{Achiral AI}}, title = {AI Memory Benchmark: What Makes Memory Trustworthy?}, year = {2026}, doi = {10.5281/zenodo.21995304}, url = {https://achiral.ai/benchmarks/ai-memory-benchmark}, note = {Public Cognoscenti benchmark report for the 2026-08-02 organizational-memory run} } Zenodo archival metadata is maintained in .zenodo.json /Achiral-AI/cognoscenti/blob/main/.zenodo.json . We welcome contributions from the community Please see CONTRIBUTING.md /Achiral-AI/cognoscenti/blob/main/CONTRIBUTING.md for guidelines on how to contribute to Cognoscenti. For detailed developer documentation, API reference, and architecture guides, see DEVELOPERS.md /Achiral-AI/cognoscenti/blob/main/DEVELOPERS.md . By participating in this project, you agree to abide by our Code of Conduct /Achiral-AI/cognoscenti/blob/main/CODE OF CONDUCT.md . To set up the development environment: Clone the repository git clone https://github.com/achiral/cognoscenti.git cd cognoscenti Build and test cargo build cargo test cargo clippy cargo fmt If you find a bug or have a feature request, please open an issue on GitHub. For security vulnerabilities, please see SECURITY.md /Achiral-AI/cognoscenti/blob/main/SECURITY.md for our disclosure policy. Existing benchmarks primarily evaluate retention, long-context retrieval, or task completion. For example, LoCoMo, a traditional memory benchmarks, will primarily answer: Can the system retrieve a fact from this specific storage? Such a test would likely yield a high yes and a few nos—with persistence exhibiting near total recall. There is no benchmarking available for directly evaluating selective forgetting , activation , contextual recall , memory strengthening , or interference resistance . Cognoscenti instead asks: Should this memory come to mind right now? The benchmark is intended for ACT-R-inspired and other human-like memory architectures that emphasize selective retrieval, strengthening through reuse, contextual recall, and useful forgetting. In several ways, Cognoscenti is a benchmark for cognitive memory systems inspired by human memory architectures such as ACT-R. A new benchmark specifically for cognitive memory covers: - Activation precision did the most relevant memories surface? - Retrieval efficiency how many memories had to be examined? - Forgetting quality were obsolete or irrelevant memories ignored? - Interference resistance can similar memories be distinguished? - Memory strengthening does repeated information become easier to retrieve? - Temporal adaptation does new information appropriately replace old assumptions? - Contextual recall does retrieval change appropriately with project or conversational context? This would highlight what makes an ACT-R-inspired memory layer fundamentally different from a persistent vector store or RAG system. In other words, one wouldn't simply abandon LoCoMo–use it for regression test long-term recall, but if your goal is to demonstrate that an ACT-R system behaves more like human cognition than a database persistence , then you'll likely need a new benchmark centered on selective remembering, useful forgetting, and efficient activation, because no current benchmark directly evaluates those properties. | Benchmark | Best for | Good fit for ACT-R? | |---|---|---| | LongMemEval | Long-context memory retrieval | Moderate | | Needle-in-a-Haystack | Retrieval under huge contexts | Low | | InfiniteBench | Scaling to very long contexts | Low | | BABILong | Reasoning over long contexts | Moderate | | τ-bench | Stateful agent tasks over time | High | | SWE-bench | Long-running engineering tasks | High if your users are developers | Unlike traditional memory benchmarks that focus on persistence and total recall , Cognoscenti evaluates cognitive properties that mirror human memory systems: Selective Forgetting : Measures how well the system identifies and fades irrelevant information Activation Dynamics : Evaluates whether the most relevant memories surface at the right time Contextual Adaptation : Tests retrieval accuracy across different conversational and project contexts Memory Consolidation : Measures how repeated access strengthens memory retrieval Interference Resistance : Evaluates ability to distinguish between similar memories While existing benchmarks answer "Can you retrieve this fact?", Cognoscenti asks "Should this memory come to mind right now?" — making it uniquely suited for evaluating ACT-R-inspired and other emergent memory systems. Does the most relevant memory become active when needed? - Top-1 retrieval accuracy - Irrelevant memory activation - Retrieval latency Does unimportant information fade? - Junk activation rate - Retrieval precision - Forgotten-memory ratio Can similar memories be distinguished? - Top-1 / Top-3 accuracy - Distractor count - Retrieval confidence Does context influence retrieval? Examples include switching between engineering, marketing, finance, or design discussions. Do repeated experiences become easier to retrieve over time? Does newer information correctly supersede outdated beliefs while preserving historical context? Does retrieval remain selective and fast as total memory grows? Possible metrics: - Memories examined - Activated chunks - Retrieval latency - Token cost Rather than isolated question-answer pairs, simulate long-running teams: - Founders - Engineers - Designers - Customers - Investors Generate months of meetings, chats, documents, roadmaps, bugs, and changing project priorities. LoCoMo remains useful as a regression test for long-term recall. Cognoscenti complements it by measuring whether an AI behaves like an experienced teammate rather than an archive. Persistent memory asks: "Can you retrieve this fact?" Cognoscenti asks: "Should this memory come to mind right now?" The long-term goal is to establish Cognoscenti as a standard benchmark for evaluating cognitive memory systems inspired by human memory architectures such as ACT-R. Cognoscenti is licensed under the Apache License, Version 2.0. See LICENSE and NOTICE . If you use Cognoscenti in your research, please cite: @software{cognoscenti2024, title = {Cognoscenti: A Benchmark for Cognitive Memory Systems}, author = {Achiral}, year = {2024}, url = {https://github.com/Achiral-AI/cognoscenti} } Cognoscenti is inspired by cognitive memory architectures such as ACT-R and aims to complement existing memory benchmarks like LoCoMo by evaluating properties unique to human-like memory systems.