{"slug": "a-longmemeval-s-number-you-can-reproduce", "title": "A LongMemEval-S number you can reproduce", "summary": "Engrava 0.6.0 scored 81.6% micro on the LongMemEval-S benchmark, a slight drop from the 0.5.0 release's 82.4%, with both runs using identical readers, judges, and scorers. The company emphasizes reproducibility, providing artifacts and noting that the memory pipeline is deterministic with no LLM calls, making it a cost-efficient retrieval-quality result.", "body_md": "We held off on posting a benchmark for a long time. Not because we didn't have runs - because most memory benchmarks you read are a number with no way to check it. A blog says \"X%\", and you have no idea what reader answered the questions, what judge scored them, how much context the retriever was allowed to feed, or whether an LLM quietly did the hard part inside the \"memory\" layer. So the number tells you almost nothing about the memory system.\n\nHere is one we're comfortable standing behind, because you can run it yourself.\n\nOn **LongMemEval-S**, the full 500-question set, **Engrava 0.6.0 scored 81.6% micro in August 2026** - 81.76% averaged across the six question categories. The run uses the **canonical LongMemEval scorer** (pinned to a known upstream commit), the standard `gpt-4o-2024-08-06`\n\nreader and judge over the OpenAI API, and a `top_k`\n\nof 20 retrieved turns. Nothing about the reader, the prompt, or the scorer is ours; the only thing we swapped in is the memory.\n\nIt is compared against the previous release: **0.5.0, run in July 2026, scored 82.4% micro / 82.58% macro** on the same 500 questions, same reader, same judge, same scorer, same `top_k`\n\n. Both rows are on the leaderboard, both `verified`\n\n, and both ship their reproduction artifacts. We are leading with 0.6.0 because that is the version this post is about; the older row stays because removing it when the number goes down is exactly the move that makes benchmark pages worthless.\n\n| 0.5.0 (2026-07-10) | 0.6.0 (2026-08-11) | |\n|---|---|---|\n| micro | 82.4% | 81.6% |\n| macro | 82.58% | 81.76% |\n| n | 500 | 500 |\n\nBoth figures are dated on purpose. This post is a record of two specific runs, not a running scoreboard; the current table, whatever version is newest when you read this, lives on the [Engrava benchmarks page](https://engrava.ai/benchmarks/).\n\nThe run also has **no LLM in the memory pipeline.** Ingestion and retrieval are deterministic - hybrid search over a typed graph, no model doing extraction, summarization, or re-ranking behind the curtain. In the benchmark's own terms this is a **Group A** run: `memory_pipeline_llms: []`\n\n. So whatever the score reflects, it is not a second language model inside the memory layer doing part of the work - and not one you'd have to pay for on every write.\n\nThat last part is a cost property, not just an architectural one. A memory layer that calls a generative model on every write - to decide what to store, to summarize it, to re-rank it on read - pays for that model on every operation, so the bill tracks how much the agent reads and writes, not how much it has stored. Engrava's ingest and retrieval are deterministic, so writing and reading memory doesn't spend generative-LLM tokens. It isn't free of model calls entirely - vector search needs an embedding at write time - but that's a cheap, pluggable embedder you can run fully local, not a generative model doing the expensive work on every operation.\n\nSo it's a **retrieval-quality** result in the sense that retrieval is the only part we swapped: the measurement runs end to end through a fixed reader and a fixed judge, and holding those constant is what makes two runs comparable. It does not make the score ours alone.\n\nThe gap between the two rows is four questions out of five hundred. The next section is about what we can and cannot say about those four.\n\nThe obvious question about two runs four questions apart is whether the newer version got worse. We went and looked at both runs' artifacts rather than guessing. The artifacts support neither \"it regressed\" nor \"it's just noise\":\n\nBetween the two runs, engrava handed the reader identical context on 457 of the 500 questions, and on the 43 where the retrieved context differed at all, not one answer changed. Every one of the 32 questions whose outcome moved - 18 down, 14 up, netting the four-question difference - received byte-identical retrieved context in both runs, same passages in the same order. Those flips therefore cannot be attributed to the memory layer: with the input to the reader unchanged, what varied was the reader and judge themselves, which are not deterministic even at temperature zero. We are not claiming the two versions are equivalent, and we have not run a replicate of this configuration, so we have no measured variance for the score itself and will not invent a confidence interval. What we can say precisely is narrower and stronger: this difference is not something engrava's retrieval did.\n\nTwo things that paragraph deliberately does not say, and we won't say them either. It does not call the difference noise - that would be a claim about measurement variance, and measuring that needs replicate runs we have not paid for. And the churn underneath the four-question net - 32 individual outcomes moving, 6.4% of the set - is a measurement from these two runs. It is not an estimate of how much a score wobbles between runs, and we won't present it as one.\n\nEngrava isn't a vector index with a graph bolted on. The pieces that move a score like this are the same ones in the free package: a **typed knowledge graph** (thoughts as nodes, seven edge types between them), **hybrid search** that fuses vector similarity, BM25 over the text, and recency in one query, and turn-level granularity so the retriever can land on the exact user turn a question depends on rather than a blurry session average.\n\nThose live alongside the rest of what Engrava ships, in one embedded SQLite store. The benchmark exercises the retrieval slice and nothing else. What it shows is that the retrieval half of a local, no-server memory layer holds up on a public long-horizon test.\n\nThe number is **not** a result about consolidation. Engrava's background consolidation (\"dreaming\") is deterministic memory hygiene, and it is switched off for this run - so whatever it does or doesn't do for long-horizon recall, this figure does not measure it and we are not reaching for it here.\n\nThe run lives in a public repo - [ sovantica/engrava-benchmark](https://github.com/sovantica/engrava-benchmark), MIT. It isn't a package you install; it's a repo you clone and run against the public\n\n`engrava`\n\non PyPI. The machine-readable `leaderboard.json`\n\nin that repo is the number of record; this post just describes two of its rows.\n\n```\ngit clone https://github.com/sovantica/engrava-benchmark.git\ncd engrava-benchmark\ngit checkout a45dde9                 # the runner commit this result pins\npython -m venv .venv\nsource .venv/bin/activate\nmake install\npip install \"engrava==0.6.0\"        # the exact version the result pins\n\nexport OPENAI_API_KEY=...            # reader + judge, OpenAI-direct\nexport ENGRAVA_BENCH_LONGMEMEVAL_S=<path>/longmemeval_s_cleaned.json  # the cleaned split (see note below)\n\npython runners/longmemeval/run.py    # no flags - the bare command is the canonical run\nmake validate\nmake leaderboard\n```\n\nThe bare command with no flags **is** the canonical configuration - the same reader, judge, scorer, and `top_k`\n\nevery published number uses. Any flag that overrides a model or endpoint moves the row out of the comparable segment, so it's no longer the headline. If you want to check the wiring before spending anything on API calls, `python runners/longmemeval/run.py --smoke`\n\nruns real Engrava retrieval against a local embedder and a mock reader/judge for free.\n\nGet the dataset right: it is the authors' **cleaned** LongMemEval-S release, from Hugging Face [ xiaowu0162/longmemeval-cleaned](https://huggingface.co/datasets/xiaowu0162/longmemeval-cleaned) (\n\n`longmemeval_s_cleaned.json`\n\n) - not the raw `longmemeval_s.json`\n\n. The result row pins that dataset by sha256, so before you run, confirm your file's hash matches the row's `dataset_revision`\n\n; a different revision produces a different score.Every result row pins the axes that move a score - engrava version and distribution hash, runner commit, reader and judge snapshots and endpoints, scorer version, retriever, granularity, `top_k`\n\n- plus a reproduction artifact and its checksum.\n\nThe two rows sit in separate comparability segments, and exactly one axis separates them: the harness commit. Dataset revision by hash, reader and judge snapshots, scorer commit, retriever and `top_k`\n\nare identical. That is why they are worth putting side by side, and why the board still keeps them apart - a number is comparable only within a segment matching on all of those axes, harness included, and different segments are never merged into one ranked table. Both sit in the canonical `gpt-4o-2024-08-06`\n\nreader and judge; a score produced with a different reader isn't rank-comparable to either, and we don't present it that way. This is one retrieval benchmark on one dataset - a real signal about long-horizon recall, not a universal statement about every workload. And it's the retrieval slice specifically: the structural guarantees Engrava also ships (the hash-linked journal, typed edges, MindQL) are their own thing, verifiable in their own right, and not what this figure measures.\n\nIf you're evaluating agent memory, don't take the percentage on faith - run the command above, then swap in whatever else you're weighing by writing one adapter, and read both numbers off the same reader and scorer. That's the comparison that actually tells you something.\n\n```\npip install engrava\n```\n\n", "url": "https://wpnews.pro/news/a-longmemeval-s-number-you-can-reproduce", "canonical_source": "https://dev.to/sovantica/a-longmemeval-s-number-you-can-reproduce-2l0n", "published_at": "2026-08-27 21:07:28+00:00", "updated_at": "2026-08-27 21:20:48.746317+00:00", "lang": "en", "topics": ["machine-learning", "ai-research", "ai-tools", "developer-tools"], "entities": ["Engrava", "LongMemEval-S", "OpenAI", "gpt-4o"], "alternates": {"html": "https://wpnews.pro/news/a-longmemeval-s-number-you-can-reproduce", "markdown": "https://wpnews.pro/news/a-longmemeval-s-number-you-can-reproduce.md", "text": "https://wpnews.pro/news/a-longmemeval-s-number-you-can-reproduce.txt", "jsonld": "https://wpnews.pro/news/a-longmemeval-s-number-you-can-reproduce.jsonld"}}