Cognoscenti is a benchmarking tool for cognitive memory systems or Emergent Memory Systems like Achiral that are ACT-R inspired rather than tools of persistence like vector or proximity-based databases, knowledge graphs, or semantic search systems with no concept of forgetting.
The problem isn't how much is remembered, but what gets chosen as memory.
Support the development of cognitive memory benchmarks and help shape the future of AI memory systems. Become a sponsor and gain visibility among thousands of AI/ML developers building the next generation of memory architectures.
📢 View Sponsorship Tiers | 🤝 GitHub Sponsors | 💬 Contact Us
Cognoscenti is implemented in Rust and provides a comprehensive framework for benchmarking cognitive memory architectures. The implementation includes:
Workload Simulation: Generate synthetic team interactions (founders, engineers, designers, customers, investors) over configurable time periodsCore Evaluation Dimensions: Measure activation precision, selective forgetting, interference resistance, contextual recall, memory consolidation, adaptation, and efficiencyMetrics Collection: Automated collection and analysis of benchmark metrics with statistical computations** Visualization**: Generate plots and export results to JSON/CSV for further analysis
git clone <repository-url>
cd cognoscenti
cargo build --release
Run the benchmark with default settings:
cargo run --release
Or customize the benchmark parameters:
cargo run --release -- \
--workload technical \
--duration-months 6 \
--retrieval-count 1000 \
--output-dir ./results
Each workload generates synthetic data patterns to test specific cognitive memory properties:
strategic
: High-level decision patterns with infrequent but high-importance memories (tests selective forgetting and activation precision)technical
: Frequent repetitive interactions with evolving technical context (tests memory consolidation and adaptation)creative
: Varied contextual patterns with cross-domain references (tests contextual recall and interference resistance)episodic
: Short-lived interaction patterns requiring rapid forgetting (tests decay rates and junk activation)analytical
: Precision-focused patterns with strict accuracy requirements (tests retrieval efficiency and top-k accuracy)
The benchmark generates:
benchmark_results.json
: Complete metrics in JSON formatbenchmark_metrics.csv
: Metrics in CSV format for easy analysismetrics_overview.png
: Visual overview of benchmark resultsbenchmark_report.pdf
: Printable report with headline metrics
Run the included JSONL benchmark items against the built-in local baseline:
cargo run --release -- \
--items-file examples/memory_tools.jsonl \
--systems local \
--top-k 3 \
--output-dir ./results/memory-tools
Compare multiple configured memory tools with the same items:
cargo run --release -- \
--items-file examples/memory_tools.jsonl \
--systems local,mem0,supermemory,zep,letta \
--top-k 3 \
--output-dir ./results/memory-tools
The eval runner always supports local
. External tools use provider-specific environment variables:
export COGNOSCENTI_MEM0_INGEST_URL="https://..."
export COGNOSCENTI_MEM0_RETRIEVE_URL="https://..."
export COGNOSCENTI_MEM0_API_KEY="..."
Use the same pattern for SUPERMEMORY
, ZEP
, and LETTA
. Provider endpoints receive JSON over POST
; ingest requests include system
and memories
, and retrieval requests include system
, query
, context
, and top_k
.
The memory-tool eval generates:
memory_eval_results.json
: Per-system item results and aggregate metricsmemory_eval_metrics.csv
: Comparison-ready metric tablememory_eval_report.pdf
: Printable comparison report
The first public organizational-memory run compares three reference architectures on the same JSONL workload:
rag
: lexical retrieval over stored memories.agent-memory
: lexical retrieval plus project, domain, and conversation scope.organic-memory
: scoped retrieval plus lifecycle signals for current, validated, reinforced, stale, superseded, and low-signal memories.
Run date: 2026-08-02
Public report: Organizational Memory Benchmark: RAG vs Agent Memory vs Organic Memory
Release notes: 2026-08-02 AI Memory Benchmark Run
| System | Top-1 Accuracy | Recall@3 | Precision@3 | Distractor Activation |
|---|---|---|---|---|
| RAG reference baseline | 50.00% | 83.33% | 27.78% | 25.00% |
| Agent-memory reference baseline | 75.00% | 100.00% | 33.33% | 38.89% |
| Organic-memory reference baseline | 91.67% | 100.00% | 33.33% | 16.67% |
Reproduce the run:
cargo run --release -- \
--items-file examples/organizational_memory.jsonl \
--systems rag,agent-memory,organic-memory \
--top-k 3 \
--output-dir results/organizational-memory-2026-08-02
Artifacts:
examples/organizational_memory.jsonl
results/organizational-memory-2026-08-02/README.md
results/organizational-memory-2026-08-02/memory_eval_results.json
results/organizational-memory-2026-08-02/memory_eval_metrics.csv
This run does not claim to benchmark Mem0, Zep, Letta, Glean, LangGraph, or any other vendor product. Vendor adapters exist through Cognoscenti's HTTP eval mode, but vendor results should be published only when endpoints, keys, seeds, commands, and raw outputs are recorded.
If you use Cognoscenti or the public benchmark artifacts, please cite the repository and the benchmark report. GitHub will surface citation metadata from CITATION.cff.
Version DOI: 10.5281/zenodo.21995304
Concept DOI: 10.5281/zenodo.21995303
@software{achiral_ai_2026_cognoscenti,
author = {{Achiral AI}},
title = {Cognoscenti: A Benchmark for Trustworthy AI Memory},
year = {2026},
version = {0.2.1},
doi = {10.5281/zenodo.21995304},
url = {https://github.com/Achiral-AI/cognoscenti}
}
@techreport{achiral_ai_2026_ai_memory_benchmark,
author = {{Achiral AI}},
title = {AI Memory Benchmark: What Makes Memory Trustworthy?},
year = {2026},
doi = {10.5281/zenodo.21995304},
url = {https://achiral.ai/benchmarks/ai-memory-benchmark},
note = {Public Cognoscenti benchmark report for the 2026-08-02 organizational-memory run}
}
Zenodo archival metadata is maintained in .zenodo.json.
We welcome contributions from the community! Please see CONTRIBUTING.md for guidelines on how to contribute to Cognoscenti.
For detailed developer documentation, API reference, and architecture guides, see DEVELOPERS.md.
By participating in this project, you agree to abide by our Code of Conduct.
To set up the development environment:
git clone https://github.com/achiral/cognoscenti.git
cd cognoscenti
cargo build
cargo test
cargo clippy
cargo fmt
If you find a bug or have a feature request, please open an issue on GitHub. For security vulnerabilities, please see SECURITY.md for our disclosure policy.
Existing benchmarks primarily evaluate retention, long-context retrieval, or task completion. For example, LoCoMo, a traditional memory benchmarks, will primarily answer:
Can the system retrieve a fact from this specific storage?
Such a test would likely yield a high yes and a few nos—with persistence exhibiting near total recall. There is no benchmarking available for directly evaluating selective forgetting, activation, contextual recall, memory strengthening, or interference resistance.
Cognoscenti instead asks:
Should this memory come to mind right now?
The benchmark is intended for ACT-R-inspired and other human-like memory architectures that emphasize selective retrieval, strengthening through reuse, contextual recall, and useful forgetting. In several ways, Cognoscenti is a benchmark for cognitive memory systems inspired by human memory architectures such as ACT-R.
A new benchmark specifically for cognitive memory covers:
- Activation precision (did the most relevant memories surface?)
- Retrieval efficiency (how many memories had to be examined?)
- Forgetting quality (were obsolete or irrelevant memories ignored?)
- Interference resistance (can similar memories be distinguished?)
- Memory strengthening (does repeated information become easier to retrieve?)
- Temporal adaptation (does new information appropriately replace old assumptions?)
- Contextual recall (does retrieval change appropriately with project or conversational context?)
This would highlight what makes an ACT-R-inspired memory layer fundamentally different from a persistent vector store or RAG system.
In other words, one wouldn't simply abandon LoCoMo–use it for regression test long-term recall, but if your goal is to demonstrate that an ACT-R system behaves more like human cognition than a database (persistence), then you'll likely need a new benchmark centered on selective remembering, useful forgetting, and efficient activation, because no current benchmark directly evaluates those properties.
| Benchmark | Best for | Good fit for ACT-R? |
|---|---|---|
| LongMemEval | Long-context memory retrieval | Moderate |
| Needle-in-a-Haystack | Retrieval under huge contexts | Low |
| InfiniteBench | Scaling to very long contexts | Low |
| BABILong | Reasoning over long contexts | Moderate |
| τ-bench | Stateful agent tasks over time | High |
| SWE-bench | Long-running engineering tasks | High (if your users are developers) |
Unlike traditional memory benchmarks that focus on persistence and total recall, Cognoscenti evaluates cognitive properties that mirror human memory systems:
Selective Forgetting: Measures how well the system identifies and fades irrelevant information** Activation Dynamics**: Evaluates whether the most relevant memories surface at the right time** Contextual Adaptation**: Tests retrieval accuracy across different conversational and project contexts** Memory Consolidation**: Measures how repeated access strengthens memory retrieval** Interference Resistance**: Evaluates ability to distinguish between similar memories
While existing benchmarks answer "Can you retrieve this fact?", Cognoscenti asks "Should this memory come to mind right now?" — making it uniquely suited for evaluating ACT-R-inspired and other emergent memory systems.
Does the most relevant memory become active when needed? - Top-1 retrieval accuracy - Irrelevant memory activation - Retrieval latency
Does unimportant information fade? - Junk activation rate - Retrieval precision - Forgotten-memory ratio
Can similar memories be distinguished? - Top-1 / Top-3 accuracy - Distractor count - Retrieval confidence
Does context influence retrieval? Examples include switching between engineering, marketing, finance, or design discussions.
Do repeated experiences become easier to retrieve over time?
Does newer information correctly supersede outdated beliefs while preserving historical context?
Does retrieval remain selective and fast as total memory grows? Possible metrics: - Memories examined - Activated chunks - Retrieval latency - Token cost
Rather than isolated question-answer pairs, simulate long-running teams: - Founders - Engineers - Designers - Customers - Investors
Generate months of meetings, chats, documents, roadmaps, bugs, and changing project priorities.
LoCoMo remains useful as a regression test for long-term recall.
Cognoscenti complements it by measuring whether an AI behaves like an experienced teammate rather than an archive.
Persistent memory asks: "Can you retrieve this fact?"
Cognoscenti asks: "Should this memory come to mind right now?"
The long-term goal is to establish Cognoscenti as a standard benchmark for evaluating cognitive memory systems inspired by human memory architectures such as ACT-R.
Cognoscenti is licensed under the Apache License, Version 2.0. See
LICENSE
and NOTICE
.
If you use Cognoscenti in your research, please cite:
@software{cognoscenti2024,
title = {Cognoscenti: A Benchmark for Cognitive Memory Systems},
author = {Achiral},
year = {2024},
url = {https://github.com/Achiral-AI/cognoscenti}
}
Cognoscenti is inspired by cognitive memory architectures such as ACT-R and aims to complement existing memory benchmarks like LoCoMo by evaluating properties unique to human-like memory systems.