A LongMemEval-S number you can reproduce Engrava 0.6.0 scored 81.6% micro on the LongMemEval-S benchmark, a slight drop from the 0.5.0 release's 82.4%, with both runs using identical readers, judges, and scorers. The company emphasizes reproducibility, providing artifacts and noting that the memory pipeline is deterministic with no LLM calls, making it a cost-efficient retrieval-quality result. We held off on posting a benchmark for a long time. Not because we didn't have runs - because most memory benchmarks you read are a number with no way to check it. A blog says "X%", and you have no idea what reader answered the questions, what judge scored them, how much context the retriever was allowed to feed, or whether an LLM quietly did the hard part inside the "memory" layer. So the number tells you almost nothing about the memory system. Here is one we're comfortable standing behind, because you can run it yourself. On LongMemEval-S , the full 500-question set, Engrava 0.6.0 scored 81.6% micro in August 2026 - 81.76% averaged across the six question categories. The run uses the canonical LongMemEval scorer pinned to a known upstream commit , the standard gpt-4o-2024-08-06 reader and judge over the OpenAI API, and a top k of 20 retrieved turns. Nothing about the reader, the prompt, or the scorer is ours; the only thing we swapped in is the memory. It is compared against the previous release: 0.5.0, run in July 2026, scored 82.4% micro / 82.58% macro on the same 500 questions, same reader, same judge, same scorer, same top k . Both rows are on the leaderboard, both verified , and both ship their reproduction artifacts. We are leading with 0.6.0 because that is the version this post is about; the older row stays because removing it when the number goes down is exactly the move that makes benchmark pages worthless. | 0.5.0 2026-07-10 | 0.6.0 2026-08-11 | | |---|---|---| | micro | 82.4% | 81.6% | | macro | 82.58% | 81.76% | | n | 500 | 500 | Both figures are dated on purpose. This post is a record of two specific runs, not a running scoreboard; the current table, whatever version is newest when you read this, lives on the Engrava benchmarks page https://engrava.ai/benchmarks/ . The run also has no LLM in the memory pipeline. Ingestion and retrieval are deterministic - hybrid search over a typed graph, no model doing extraction, summarization, or re-ranking behind the curtain. In the benchmark's own terms this is a Group A run: memory pipeline llms: . So whatever the score reflects, it is not a second language model inside the memory layer doing part of the work - and not one you'd have to pay for on every write. That last part is a cost property, not just an architectural one. A memory layer that calls a generative model on every write - to decide what to store, to summarize it, to re-rank it on read - pays for that model on every operation, so the bill tracks how much the agent reads and writes, not how much it has stored. Engrava's ingest and retrieval are deterministic, so writing and reading memory doesn't spend generative-LLM tokens. It isn't free of model calls entirely - vector search needs an embedding at write time - but that's a cheap, pluggable embedder you can run fully local, not a generative model doing the expensive work on every operation. So it's a retrieval-quality result in the sense that retrieval is the only part we swapped: the measurement runs end to end through a fixed reader and a fixed judge, and holding those constant is what makes two runs comparable. It does not make the score ours alone. The gap between the two rows is four questions out of five hundred. The next section is about what we can and cannot say about those four. The obvious question about two runs four questions apart is whether the newer version got worse. We went and looked at both runs' artifacts rather than guessing. The artifacts support neither "it regressed" nor "it's just noise": Between the two runs, engrava handed the reader identical context on 457 of the 500 questions, and on the 43 where the retrieved context differed at all, not one answer changed. Every one of the 32 questions whose outcome moved - 18 down, 14 up, netting the four-question difference - received byte-identical retrieved context in both runs, same passages in the same order. Those flips therefore cannot be attributed to the memory layer: with the input to the reader unchanged, what varied was the reader and judge themselves, which are not deterministic even at temperature zero. We are not claiming the two versions are equivalent, and we have not run a replicate of this configuration, so we have no measured variance for the score itself and will not invent a confidence interval. What we can say precisely is narrower and stronger: this difference is not something engrava's retrieval did. Two things that paragraph deliberately does not say, and we won't say them either. It does not call the difference noise - that would be a claim about measurement variance, and measuring that needs replicate runs we have not paid for. And the churn underneath the four-question net - 32 individual outcomes moving, 6.4% of the set - is a measurement from these two runs. It is not an estimate of how much a score wobbles between runs, and we won't present it as one. Engrava isn't a vector index with a graph bolted on. The pieces that move a score like this are the same ones in the free package: a typed knowledge graph thoughts as nodes, seven edge types between them , hybrid search that fuses vector similarity, BM25 over the text, and recency in one query, and turn-level granularity so the retriever can land on the exact user turn a question depends on rather than a blurry session average. Those live alongside the rest of what Engrava ships, in one embedded SQLite store. The benchmark exercises the retrieval slice and nothing else. What it shows is that the retrieval half of a local, no-server memory layer holds up on a public long-horizon test. The number is not a result about consolidation. Engrava's background consolidation "dreaming" is deterministic memory hygiene, and it is switched off for this run - so whatever it does or doesn't do for long-horizon recall, this figure does not measure it and we are not reaching for it here. The run lives in a public repo - sovantica/engrava-benchmark https://github.com/sovantica/engrava-benchmark , MIT. It isn't a package you install; it's a repo you clone and run against the public engrava on PyPI. The machine-readable leaderboard.json in that repo is the number of record; this post just describes two of its rows. git clone https://github.com/sovantica/engrava-benchmark.git cd engrava-benchmark git checkout a45dde9 the runner commit this result pins python -m venv .venv source .venv/bin/activate make install pip install "engrava==0.6.0" the exact version the result pins export OPENAI API KEY=... reader + judge, OpenAI-direct export ENGRAVA BENCH LONGMEMEVAL S=