{"slug": "comparing-three-llms-across-hindsight-s-memory-operations", "title": "Comparing Three LLMs Across Hindsight's Memory Operations", "summary": "A blind benchmark of three LLMs on Hindsight 0.10.2's memory operations found that gpt-6.1-sol extracted significantly better fact lists on retain and produced banks that scored higher on a 50-question gold set, while gpt-5.6-luna remained the better all-around choice and gpt-6-luna was faster on every operation but trended worse on quality and missed more at retain. The test covered retain, page refresh and reflect across 26 banks and about 1.85 million rows, with consolidation measured only in a follow-up where it stayed on gpt-5.6-luna. gpt-6.1-sol was slower than gpt-5.6-luna on every measured operation and showed no quality gain on reflect or page refresh, and the scripts and aggregate results are published as hindsight-llm-bench.", "body_md": "Article\n\n# Comparing Three LLMs Across Hindsight's Memory Operations\n\nA blind benchmark of three LLMs on Hindsight's retain, page refresh and reflect, plus a downstream Q&A test. gpt-6.1-sol extracted better facts; gpt-5.6-luna stayed for the rest.\n\nWe run a local [Hindsight](https://github.com/vectorize-io/hindsight)\ninstance as the memory for our coding agents. It runs in Docker, and the\nHindsight coding-agent plugin connects it to agent sessions. Each repository\nhas its own bank. Session transcripts and commit history are retained into\nit, and agents read it back through knowledge-page search and reflect.\nHindsight’s LLM calls go through the `openai-codex` provider on a ChatGPT\nsubscription. All of them currently use gpt-5.6-luna.\n\nWe measured two newer models, gpt-6-luna and gpt-6.1-sol, against gpt-5.6-luna on each Hindsight operation that calls an LLM, and then measured how the choice of retain model affects consolidation cost and answer accuracy downstream. gpt-6.1-sol was slower than gpt-5.6-luna on every operation we measured and showed no quality gain on reflect or page refresh. On retain it produced significantly better fact lists, and banks built from its facts scored higher on a gold question set. gpt-6-luna was faster on every operation we measured but trended worse on quality and missed more at retain.\n\nHindsight 0.10.2, reasoning effort low for every model. Retain verdicts from Claude (5 ties, 2 losses); wrong answers are those both graders marked wrong on 50 gold questions. Luna- and sol-retained banks were built with gpt-5.6-luna or gpt-6.1-sol as the retain model.\n\n## The operations under test\n\nFour Hindsight operations call an LLM, and each can be given its own model:\n\n- **Retain** extracts facts from a new document, such as a session\ntranscript, guided by the bank’s retain mission. Ours asks for technical\ndecisions, conventions, tooling setup and recurring gotchas, and to\ndeprioritize transient debugging chatter. Retain runs in the background.\nIts model is set with`HINDSIGHT_API_RETAIN_LLM_MODEL` .\n- **Consolidation** merges facts into observations in the background and\nmarks the knowledge pages that depend on them as stale.\n- **Page refresh** rebuilds a stale knowledge page by running the page’s\nstanding question through the reflect pipeline. Without its own setting,\nit uses the reflect model.\n- **Reflect** answers a question with an agent loop that searches pages and\nmemories, then writes the answer. It runs while the caller waits.\n\nRecall, the search that reflect uses, makes no LLM call. The article on\n[recall, reflect and the knowledge page](https://feniix-hq.net/blog/recall-reflect-and-the-knowledge-page/)\ndescribes how these layers relate.\n\nThe main comparison covered retain, page refresh and reflect. Consolidation was measured only in the follow-up, where it stayed on gpt-5.6-luna and the variable was the retain model feeding it.\n\n## Setup\n\nWe ran the measurements on a separate, throwaway Hindsight 0.10.2 container, restored from a backup of the production instance: 26 banks, about 1.85 million rows. It had its own Codex login and the same environment as production, and its background worker was off, so every run read identical memory. We read production only once, for the backup.\n\nThe run used six blocks per phase. Each block ran all three models in a\ndifferent order, so each model ran first, second and third twice. The\ncontainer was recreated before every run, and only the model setting for\nthe operation under test changed between runs. All three models ran at\nreasoning effort `low`. The scripts, the methodology and the aggregate\nresults are published as\n[hindsight-llm-bench](https://github.com/feniix/hindsight-llm-bench). They\ntarget banks written by the coding-agent plugin, and the repository lists\nwhat to adapt for other kinds of bank.\n\n- **Reflect:** 28 existing benchmark queries. Each answer was returned with\nthe sources it used, so judges could check its claims against them.\n- **Page refresh:** 12 knowledge pages, rebuilt through Hindsight’s\ndry-run refresh endpoint, which runs the production refresh pipeline\nwithout saving the result.\n- **Retain:** 24 real documents from 8 banks, about 3,000 to 38,500\ncharacters each, run through the dry-run extract endpoint with each\ndocument’s own bank mission and retain strategy.\n\nA logging proxy between Hindsight and Codex recorded exact token counts per call. Codex returns a usage block, including reasoning tokens, in its final streaming event, but Hindsight’s Codex provider does not read it. Hindsight’s own traces for this provider estimate input and output tokens from text length (characters ÷ 4) and carry no reasoning tokens. The exact numbers below come from the proxy.\n\nWe ran the quality judging blind against gpt-5.6-luna, one pair at a time; the judge was not told which model produced which side. Claude Opus judged reflect answers and pages. Claude and, separately, Gemini 3.1 Pro judged the retain fact lists, and TypeSafe’s Jev model scored every extracted fact on its own, as described in the retain section.\n\n## Latency\n\nEach ratio below is the geometric mean of per-item time ratios against gpt-5.6-luna on the same queries, pages or documents, with a 95% bootstrap interval.\n\ngpt-6-luna was faster than gpt-5.6-luna on all three operations. gpt-6.1-sol was slower on all three, most of all on reflect.\n\nThe traces show where gpt-6.1-sol’s reflect time went. Its median reflect\nmade 5 LLM calls against gpt-5.6-luna’s 2 and read 111k input tokens\nagainst 29k, because it ran more search rounds before answering and each\nround added results to the next prompt. The extra time did not go into\nreasoning: at effort `low` it produced 0.9k reasoning tokens across the\nwhole run, against 9.3k for gpt-5.6-luna and 1.4k for gpt-6-luna.\n\n| Whole run, from the proxy | gpt-5.6-luna | gpt-6-luna | gpt-6.1-sol | \n|---|---|---|---|\n| Input tokens | 2.06M | 1.76M | 6.70M | \n| Output tokens | 117k | 65k | 135k | \n| Reasoning tokens | 9.3k | 1.4k | 0.9k | \n| Reflect LLM calls per query (median) | 2 | 2 | 5 | \n\nTotals across reflect, page refresh and retain. 8 of 707 proxied calls lost their usage block when a stream was cut at a deadline, 6 of them gpt-6.1-sol’s, so its totals are a slight undercount.\n\nThe production time limits also applied. Reflect allows 180 seconds per LLM call and 300 seconds per request, and page refresh allows 120 seconds per call. Three of gpt-6.1-sol’s 28 reflects exceeded the 300-second limit and returned HTTP 504. One of its page refreshes had a call cut at 120 seconds; Hindsight recovered and produced the page after 251 seconds. Neither luna model hit a limit.\n\n## Reflect and page refresh quality\n\nBlind pairs against gpt-5.6-luna, judged by Claude. p comes from a two-sided sign test over the decisive pairs, which are the ones not tied. It is the chance of a split at least this lopsided if both models were equally good. Below 0.05 is the usual bar for calling a difference significant. gpt-6.1-sol’s three reflect timeouts are counted as losses.\n\nNeither candidate was better than gpt-5.6-luna on reflect or page refresh. Both leaned worse, but no split was statistically significant at this sample size. gpt-6-luna wrote less output than gpt-5.6-luna: 0.48× on reflect and 0.58× on page refresh. With no quality gain to offset them, gpt-6.1-sol’s latency, its timeouts and its 3.8× median input tokens on reflect rule it out for these operations. gpt-6-luna’s speed did not outweigh its trend toward worse answers.\n\n## Retain quality\n\nRetain makes one extraction call per chunk and has no search loop, so gpt-6.1-sol’s extra search rounds do not apply. Its retain latency was 1.64× gpt-5.6-luna’s, and retain runs in the background.\n\nBlind pairs against gpt-5.6-luna on the same scale as the reflect figure. Each pair of fact lists was judged against the source document and its bank’s retain mission; p is a two-sided sign test.\n\nBoth judges found gpt-6.1-sol significantly better at retain. Their stated reasons were consistent. gpt-6.1-sol kept the exact identifiers the missions ask for, such as commit hashes, issue and pull request numbers, endpoints, line ranges and version pins, which gpt-5.6-luna sometimes dropped or altered; in one document gpt-5.6-luna changed one character of a 12-character hash. gpt-6.1-sol also tended to record final outcomes rather than intermediate states. Its main weakness, flagged by both Claude and Jev, was occasionally keeping a superseded “still pending” status as if it were current.\n\ngpt-6.1-sol also extracted more facts: 164 against 134 for gpt-5.6-luna on the same 24 documents. More facts are useful when they capture items that would otherwise be missing, and costly when they are redundant or off-mission, since each fact adds consolidation work and competes for recall candidates. To tell these apart, we measured coverage and per-fact quality separately.\n\nFor coverage, Gemini listed the key items each document’s mission asked to keep, without seeing any model’s output: 194 reference items across the 24 documents. Jev then checked which items each model’s facts covered, and scored every fact for support, relevance, chatter, redundancy and whether it stands on its own.\n\n| Per fact, Jev | gpt-5.6-luna | gpt-6-luna | gpt-6.1-sol | \n|---|---|---|---|\n| Facts per document | 5.6 | 4.4 | 6.8 | \n| Supported by the document | 0.97 | 0.97 | 0.96 | \n| Contradicted by the document | 0 of 134 | 1 of 105 | 3 of 164 | \n| Relevant to the mission | 0.79 | 0.79 | 0.79 | \n| Chatter | 0.18 | 0.18 | 0.17 | \n| Redundant | 0.18 | 0.18 | 0.20 | \n| Stands alone (0–2) | 1.72 | 1.71 | 1.77 | \n| Coverage of reference items | 0.84 | 0.74 | 0.86 | \n\nMean probability per fact unless noted. Stands alone is scored 0 to 2: 0 is unclear without the document, 2 names its subject, the concrete identifiers or values, and the reason where one exists. Change in coverage per document against gpt-5.6-luna: gpt-6-luna −0.10 [−0.21, −0.01], gpt-6.1-sol +0.02 [−0.02, +0.06]. gpt-6-luna returned no facts for two documents.\n\ngpt-6.1-sol’s additional facts scored the same as gpt-5.6-luna’s on support, relevance, chatter and redundancy, and slightly higher on standing alone. Its coverage gain was small and not significant, so the judges’ preference seems to come from precision (exact identifiers and final outcomes) more than from extra items captured. gpt-6-luna’s coverage was significantly lower than gpt-5.6-luna’s.\n\n## Downstream follow-up\n\nA fact list is an intermediate product. Agents read answers built from observations and pages, which consolidation and page refresh derive from those facts. The follow-up measured whether the retain result carries through that chain, and what it costs.\n\nWe took two complete corpora from real banks: this website’s repository (32 documents, 553k characters) and the Hindsight install’s own repository (33 documents, 721k characters). We retained each corpus twice into fresh banks, once per retain model, in date order and with the source bank’s configuration. We call the results the luna-retained and sol-retained banks, after the model that ran retain. Everything after retain used gpt-5.6-luna: consolidation, the source bank’s knowledge pages (9 and 7), and reflect.\n\nGemini wrote 25 gold questions per corpus from the raw documents, without seeing any extracted facts. About a third ask for the current state of something that changed, a third ask why, and a third ask for a specific fact. Jev and Gemini each graded every reflect answer on its own against its gold answer.\n\n| Gold Q&A score | luna-retained bank | sol-retained bank | Change | \n|---|---|---|---|\n| Hindsight install corpus, Jev | 0.74 | 0.92 | +0.18 | \n| Hindsight install corpus, Gemini | 0.80 | 0.94 | +0.14 | \n| Website corpus, Jev | 0.62 | 0.78 | +0.16 | \n| Website corpus, Gemini | 0.70 | 0.80 | +0.10 | \n| All 50 questions, mean of both graders |  |  | +0.145 | \n\nCorrect = 1, partial = 0.5. The pooled change, +0.145, is the mean difference per question on that 0-to-1 score; its 95% bootstrap interval is [+0.035, +0.260]. The two graders gave the same grade to 90 of 100 answers. Per question, the sol-retained bank was better on 15, the same on 29 and worse on 6 (sign test p = 0.078; Jev alone 0.052, Gemini alone 0.118).\n\nBy question type, the sol-retained banks were better on 6 current-state questions and worse on 1, better on 5 specific-fact questions and worse on none, and better on 4 “why” questions and worse on 5. The luna-retained banks produced 3 answers both graders marked wrong; the sol-retained banks produced none. The sol-retained banks also answered “not in memory” less often, 4 times against 7.\n\nClaude also judged the 16 rebuilt knowledge pages blind, and they came out even: 4 better, 8 tied, 4 worse for the sol-retained banks. The answer improvement therefore did not come with a detectable page improvement. One untested explanation is that page summaries do not carry the identifiers and current states where the two fact sets differ, while specific questions can reach the facts directly.\n\n### Consolidation cost\n\n| luna-retained → sol-retained | Hindsight install | Website | \n|---|---|---|\n| Facts stored at retain | 279 → 321 (+15%) | 233 → 286 (+23%) | \n| Observations | 90 → 108 (+20%) | 90 → 127 (+41%) | \n| Consolidation LLM calls | +21% | +16% | \n| Consolidation input tokens | +17% | +26% | \n| Consolidation LLM time | +33% | +30% | \n| Consolidation failures | 0 / 0 | 0 / 0 | \n| Retain wall time | 19 → 27 min | 14 → 21 min | \n\nCalls and tokens from Hindsight’s trace table, joined to the logging proxy by start time (1,061 of 1,076 calls matched).\n\nThe additional facts produced proportionally as many or more additional observations, which suggests they were mostly distinct rather than duplicates that consolidation would have merged. Consolidation on gpt-5.6-luna did 16–33% more work for the sol-retained banks, by calls, input tokens and LLM time, with no failures. On a ChatGPT subscription this is not billed per token; it appears as background processing time and as usage against the same quota that interactive reflect uses.\n\nAs of writing, production has not been switched; this is the next step.\n\n## Limitations\n\n- **Single build per bank.** Each follow-up bank was built once, and\nreflect answers vary between runs. The score interval excludes zero, but\nthe per-question sign tests are borderline (p 0.05–0.12).\n- **Model judges.** All quality judgments came from Claude, Gemini and Jev.\nClaude and Gemini agreed on the retain result, and Claude and Jev both\nflagged gpt-6.1-sol’s stale-status weakness. No person reviewed every\npair.\n- **Concurrent consolidation.** The copied bank configurations had\nauto-consolidation enabled, which overrides the server-wide setting, so\nthe four follow-up banks consolidated at the same time. Conditions were\nthe same for both retain models and cost was attributed per bank from\nthe traces, but per-bank wall-clock time is not comparable.\n- **Self-reference.** The Hindsight install corpus includes transcripts of\nearlier parts of the session that ran this benchmark.\n- **One reasoning setting.** All models ran at effort`low` . Results at\nother effort levels may differ, particularly for gpt-6.1-sol, which\nbarely reasoned at this setting.\n\n## Summary\n\n| Operation | gpt-6-luna vs gpt-5.6-luna | gpt-6.1-sol vs gpt-5.6-luna | Planned model | \n|---|---|---|---|\n| Retain | 0.72× time; lower coverage | 1.64× time; better facts (Claude 17–5–2, Gemini 15–6–3) | gpt-6.1-sol | \n| Consolidation | not tested | not tested as the consolidation model; its facts added 16–33% consolidation work | gpt-5.6-luna | \n| Page refresh | 0.67× time; 3–2–7 | 1.86× time; 4–2–6 | gpt-5.6-luna | \n| Reflect | 0.58× time; 7–9–12 | 2.55× time; 10–3–15, including 3 timeouts counted as worse | gpt-5.6-luna | \n\nTriplets are blind verdicts against gpt-5.6-luna, counted as better–tie–worse for the candidate: 17–5–2 means its output was judged better on 17 items, tied on 5 and worse on 2. Reflect and page refresh were judged by Claude. Time is relative to gpt-5.6-luna.\n\nBecause each Hindsight operation can use its own model, the result depends on which operation is measured. A comparison of reflect alone would have missed gpt-6.1-sol’s retain result, and a comparison of retain alone would have missed that it exceeds the production time limits on reflect.\n\nThe scripts and methodology behind this comparison are published as\n[hindsight-llm-bench](https://feniix-hq.net/projects/hindsight-llm-bench/), for anyone who wants\nto repeat it on their own install or with other models.", "url": "https://wpnews.pro/news/comparing-three-llms-across-hindsight-s-memory-operations", "canonical_source": "https://feniix-hq.net/blog/hindsight-llm-comparison/", "published_at": "2026-10-01 00:00:00+00:00", "updated_at": "2026-10-19 21:14:59.204054+00:00", "lang": "en", "topics": ["large-language-models", "ai-agents", "ai-tools", "ai-research"], "entities": ["Hindsight", "gpt-6.1-sol", "gpt-5.6-luna", "gpt-6-luna", "OpenAI", "ChatGPT", "vectorize-io", "hindsight-llm-bench"], "also_reported_by": [], "takeaway": "gpt-6.1-sol as the retain model produced better fact lists and higher downstream gold-question accuracy than gpt-5.6-luna, but was slower on every operation and gave no quality gain on reflect or page refresh.", "recommendation": null, "alternates": {"html": "https://wpnews.pro/news/comparing-three-llms-across-hindsight-s-memory-operations", "markdown": "https://wpnews.pro/news/comparing-three-llms-across-hindsight-s-memory-operations.md", "text": "https://wpnews.pro/news/comparing-three-llms-across-hindsight-s-memory-operations.txt", "jsonld": "https://wpnews.pro/news/comparing-three-llms-across-hindsight-s-memory-operations.jsonld"}}