Comparing Three LLMs Across Hindsight's Memory Operations A blind benchmark of three LLMs on Hindsight 0.10.2's memory operations found that gpt-6.1-sol extracted significantly better fact lists on retain and produced banks that scored higher on a 50-question gold set, while gpt-5.6-luna remained the better all-around choice and gpt-6-luna was faster on every operation but trended worse on quality and missed more at retain. The test covered retain, page refresh and reflect across 26 banks and about 1.85 million rows, with consolidation measured only in a follow-up where it stayed on gpt-5.6-luna. gpt-6.1-sol was slower than gpt-5.6-luna on every measured operation and showed no quality gain on reflect or page refresh, and the scripts and aggregate results are published as hindsight-llm-bench. Article Comparing Three LLMs Across Hindsight's Memory Operations A blind benchmark of three LLMs on Hindsight's retain, page refresh and reflect, plus a downstream Q&A test. gpt-6.1-sol extracted better facts; gpt-5.6-luna stayed for the rest. We run a local Hindsight https://github.com/vectorize-io/hindsight instance as the memory for our coding agents. It runs in Docker, and the Hindsight coding-agent plugin connects it to agent sessions. Each repository has its own bank. Session transcripts and commit history are retained into it, and agents read it back through knowledge-page search and reflect. Hindsight’s LLM calls go through the openai-codex provider on a ChatGPT subscription. All of them currently use gpt-5.6-luna. We measured two newer models, gpt-6-luna and gpt-6.1-sol, against gpt-5.6-luna on each Hindsight operation that calls an LLM, and then measured how the choice of retain model affects consolidation cost and answer accuracy downstream. gpt-6.1-sol was slower than gpt-5.6-luna on every operation we measured and showed no quality gain on reflect or page refresh. On retain it produced significantly better fact lists, and banks built from its facts scored higher on a gold question set. gpt-6-luna was faster on every operation we measured but trended worse on quality and missed more at retain. Hindsight 0.10.2, reasoning effort low for every model. Retain verdicts from Claude 5 ties, 2 losses ; wrong answers are those both graders marked wrong on 50 gold questions. Luna- and sol-retained banks were built with gpt-5.6-luna or gpt-6.1-sol as the retain model. The operations under test Four Hindsight operations call an LLM, and each can be given its own model: - Retain extracts facts from a new document, such as a session transcript, guided by the bank’s retain mission. Ours asks for technical decisions, conventions, tooling setup and recurring gotchas, and to deprioritize transient debugging chatter. Retain runs in the background. Its model is set with HINDSIGHT API RETAIN LLM MODEL . - Consolidation merges facts into observations in the background and marks the knowledge pages that depend on them as stale. - Page refresh rebuilds a stale knowledge page by running the page’s standing question through the reflect pipeline. Without its own setting, it uses the reflect model. - Reflect answers a question with an agent loop that searches pages and memories, then writes the answer. It runs while the caller waits. Recall, the search that reflect uses, makes no LLM call. The article on recall, reflect and the knowledge page https://feniix-hq.net/blog/recall-reflect-and-the-knowledge-page/ describes how these layers relate. The main comparison covered retain, page refresh and reflect. Consolidation was measured only in the follow-up, where it stayed on gpt-5.6-luna and the variable was the retain model feeding it. Setup We ran the measurements on a separate, throwaway Hindsight 0.10.2 container, restored from a backup of the production instance: 26 banks, about 1.85 million rows. It had its own Codex login and the same environment as production, and its background worker was off, so every run read identical memory. We read production only once, for the backup. The run used six blocks per phase. Each block ran all three models in a different order, so each model ran first, second and third twice. The container was recreated before every run, and only the model setting for the operation under test changed between runs. All three models ran at reasoning effort low . The scripts, the methodology and the aggregate results are published as hindsight-llm-bench https://github.com/feniix/hindsight-llm-bench . They target banks written by the coding-agent plugin, and the repository lists what to adapt for other kinds of bank. - Reflect: 28 existing benchmark queries. Each answer was returned with the sources it used, so judges could check its claims against them. - Page refresh: 12 knowledge pages, rebuilt through Hindsight’s dry-run refresh endpoint, which runs the production refresh pipeline without saving the result. - Retain: 24 real documents from 8 banks, about 3,000 to 38,500 characters each, run through the dry-run extract endpoint with each document’s own bank mission and retain strategy. A logging proxy between Hindsight and Codex recorded exact token counts per call. Codex returns a usage block, including reasoning tokens, in its final streaming event, but Hindsight’s Codex provider does not read it. Hindsight’s own traces for this provider estimate input and output tokens from text length characters ÷ 4 and carry no reasoning tokens. The exact numbers below come from the proxy. We ran the quality judging blind against gpt-5.6-luna, one pair at a time; the judge was not told which model produced which side. Claude Opus judged reflect answers and pages. Claude and, separately, Gemini 3.1 Pro judged the retain fact lists, and TypeSafe’s Jev model scored every extracted fact on its own, as described in the retain section. Latency Each ratio below is the geometric mean of per-item time ratios against gpt-5.6-luna on the same queries, pages or documents, with a 95% bootstrap interval. gpt-6-luna was faster than gpt-5.6-luna on all three operations. gpt-6.1-sol was slower on all three, most of all on reflect. The traces show where gpt-6.1-sol’s reflect time went. Its median reflect made 5 LLM calls against gpt-5.6-luna’s 2 and read 111k input tokens against 29k, because it ran more search rounds before answering and each round added results to the next prompt. The extra time did not go into reasoning: at effort low it produced 0.9k reasoning tokens across the whole run, against 9.3k for gpt-5.6-luna and 1.4k for gpt-6-luna. | Whole run, from the proxy | gpt-5.6-luna | gpt-6-luna | gpt-6.1-sol | |---|---|---|---| | Input tokens | 2.06M | 1.76M | 6.70M | | Output tokens | 117k | 65k | 135k | | Reasoning tokens | 9.3k | 1.4k | 0.9k | | Reflect LLM calls per query median | 2 | 2 | 5 | Totals across reflect, page refresh and retain. 8 of 707 proxied calls lost their usage block when a stream was cut at a deadline, 6 of them gpt-6.1-sol’s, so its totals are a slight undercount. The production time limits also applied. Reflect allows 180 seconds per LLM call and 300 seconds per request, and page refresh allows 120 seconds per call. Three of gpt-6.1-sol’s 28 reflects exceeded the 300-second limit and returned HTTP 504. One of its page refreshes had a call cut at 120 seconds; Hindsight recovered and produced the page after 251 seconds. Neither luna model hit a limit. Reflect and page refresh quality Blind pairs against gpt-5.6-luna, judged by Claude. p comes from a two-sided sign test over the decisive pairs, which are the ones not tied. It is the chance of a split at least this lopsided if both models were equally good. Below 0.05 is the usual bar for calling a difference significant. gpt-6.1-sol’s three reflect timeouts are counted as losses. Neither candidate was better than gpt-5.6-luna on reflect or page refresh. Both leaned worse, but no split was statistically significant at this sample size. gpt-6-luna wrote less output than gpt-5.6-luna: 0.48× on reflect and 0.58× on page refresh. With no quality gain to offset them, gpt-6.1-sol’s latency, its timeouts and its 3.8× median input tokens on reflect rule it out for these operations. gpt-6-luna’s speed did not outweigh its trend toward worse answers. Retain quality Retain makes one extraction call per chunk and has no search loop, so gpt-6.1-sol’s extra search rounds do not apply. Its retain latency was 1.64× gpt-5.6-luna’s, and retain runs in the background. Blind pairs against gpt-5.6-luna on the same scale as the reflect figure. Each pair of fact lists was judged against the source document and its bank’s retain mission; p is a two-sided sign test. Both judges found gpt-6.1-sol significantly better at retain. Their stated reasons were consistent. gpt-6.1-sol kept the exact identifiers the missions ask for, such as commit hashes, issue and pull request numbers, endpoints, line ranges and version pins, which gpt-5.6-luna sometimes dropped or altered; in one document gpt-5.6-luna changed one character of a 12-character hash. gpt-6.1-sol also tended to record final outcomes rather than intermediate states. Its main weakness, flagged by both Claude and Jev, was occasionally keeping a superseded “still pending” status as if it were current. gpt-6.1-sol also extracted more facts: 164 against 134 for gpt-5.6-luna on the same 24 documents. More facts are useful when they capture items that would otherwise be missing, and costly when they are redundant or off-mission, since each fact adds consolidation work and competes for recall candidates. To tell these apart, we measured coverage and per-fact quality separately. For coverage, Gemini listed the key items each document’s mission asked to keep, without seeing any model’s output: 194 reference items across the 24 documents. Jev then checked which items each model’s facts covered, and scored every fact for support, relevance, chatter, redundancy and whether it stands on its own. | Per fact, Jev | gpt-5.6-luna | gpt-6-luna | gpt-6.1-sol | |---|---|---|---| | Facts per document | 5.6 | 4.4 | 6.8 | | Supported by the document | 0.97 | 0.97 | 0.96 | | Contradicted by the document | 0 of 134 | 1 of 105 | 3 of 164 | | Relevant to the mission | 0.79 | 0.79 | 0.79 | | Chatter | 0.18 | 0.18 | 0.17 | | Redundant | 0.18 | 0.18 | 0.20 | | Stands alone 0–2 | 1.72 | 1.71 | 1.77 | | Coverage of reference items | 0.84 | 0.74 | 0.86 | Mean probability per fact unless noted. Stands alone is scored 0 to 2: 0 is unclear without the document, 2 names its subject, the concrete identifiers or values, and the reason where one exists. Change in coverage per document against gpt-5.6-luna: gpt-6-luna −0.10 −0.21, −0.01 , gpt-6.1-sol +0.02 −0.02, +0.06 . gpt-6-luna returned no facts for two documents. gpt-6.1-sol’s additional facts scored the same as gpt-5.6-luna’s on support, relevance, chatter and redundancy, and slightly higher on standing alone. Its coverage gain was small and not significant, so the judges’ preference seems to come from precision exact identifiers and final outcomes more than from extra items captured. gpt-6-luna’s coverage was significantly lower than gpt-5.6-luna’s. Downstream follow-up A fact list is an intermediate product. Agents read answers built from observations and pages, which consolidation and page refresh derive from those facts. The follow-up measured whether the retain result carries through that chain, and what it costs. We took two complete corpora from real banks: this website’s repository 32 documents, 553k characters and the Hindsight install’s own repository 33 documents, 721k characters . We retained each corpus twice into fresh banks, once per retain model, in date order and with the source bank’s configuration. We call the results the luna-retained and sol-retained banks, after the model that ran retain. Everything after retain used gpt-5.6-luna: consolidation, the source bank’s knowledge pages 9 and 7 , and reflect. Gemini wrote 25 gold questions per corpus from the raw documents, without seeing any extracted facts. About a third ask for the current state of something that changed, a third ask why, and a third ask for a specific fact. Jev and Gemini each graded every reflect answer on its own against its gold answer. | Gold Q&A score | luna-retained bank | sol-retained bank | Change | |---|---|---|---| | Hindsight install corpus, Jev | 0.74 | 0.92 | +0.18 | | Hindsight install corpus, Gemini | 0.80 | 0.94 | +0.14 | | Website corpus, Jev | 0.62 | 0.78 | +0.16 | | Website corpus, Gemini | 0.70 | 0.80 | +0.10 | | All 50 questions, mean of both graders | | | +0.145 | Correct = 1, partial = 0.5. The pooled change, +0.145, is the mean difference per question on that 0-to-1 score; its 95% bootstrap interval is +0.035, +0.260 . The two graders gave the same grade to 90 of 100 answers. Per question, the sol-retained bank was better on 15, the same on 29 and worse on 6 sign test p = 0.078; Jev alone 0.052, Gemini alone 0.118 . By question type, the sol-retained banks were better on 6 current-state questions and worse on 1, better on 5 specific-fact questions and worse on none, and better on 4 “why” questions and worse on 5. The luna-retained banks produced 3 answers both graders marked wrong; the sol-retained banks produced none. The sol-retained banks also answered “not in memory” less often, 4 times against 7. Claude also judged the 16 rebuilt knowledge pages blind, and they came out even: 4 better, 8 tied, 4 worse for the sol-retained banks. The answer improvement therefore did not come with a detectable page improvement. One untested explanation is that page summaries do not carry the identifiers and current states where the two fact sets differ, while specific questions can reach the facts directly. Consolidation cost | luna-retained → sol-retained | Hindsight install | Website | |---|---|---| | Facts stored at retain | 279 → 321 +15% | 233 → 286 +23% | | Observations | 90 → 108 +20% | 90 → 127 +41% | | Consolidation LLM calls | +21% | +16% | | Consolidation input tokens | +17% | +26% | | Consolidation LLM time | +33% | +30% | | Consolidation failures | 0 / 0 | 0 / 0 | | Retain wall time | 19 → 27 min | 14 → 21 min | Calls and tokens from Hindsight’s trace table, joined to the logging proxy by start time 1,061 of 1,076 calls matched . The additional facts produced proportionally as many or more additional observations, which suggests they were mostly distinct rather than duplicates that consolidation would have merged. Consolidation on gpt-5.6-luna did 16–33% more work for the sol-retained banks, by calls, input tokens and LLM time, with no failures. On a ChatGPT subscription this is not billed per token; it appears as background processing time and as usage against the same quota that interactive reflect uses. As of writing, production has not been switched; this is the next step. Limitations - Single build per bank. Each follow-up bank was built once, and reflect answers vary between runs. The score interval excludes zero, but the per-question sign tests are borderline p 0.05–0.12 . - Model judges. All quality judgments came from Claude, Gemini and Jev. Claude and Gemini agreed on the retain result, and Claude and Jev both flagged gpt-6.1-sol’s stale-status weakness. No person reviewed every pair. - Concurrent consolidation. The copied bank configurations had auto-consolidation enabled, which overrides the server-wide setting, so the four follow-up banks consolidated at the same time. Conditions were the same for both retain models and cost was attributed per bank from the traces, but per-bank wall-clock time is not comparable. - Self-reference. The Hindsight install corpus includes transcripts of earlier parts of the session that ran this benchmark. - One reasoning setting. All models ran at effort low . Results at other effort levels may differ, particularly for gpt-6.1-sol, which barely reasoned at this setting. Summary | Operation | gpt-6-luna vs gpt-5.6-luna | gpt-6.1-sol vs gpt-5.6-luna | Planned model | |---|---|---|---| | Retain | 0.72× time; lower coverage | 1.64× time; better facts Claude 17–5–2, Gemini 15–6–3 | gpt-6.1-sol | | Consolidation | not tested | not tested as the consolidation model; its facts added 16–33% consolidation work | gpt-5.6-luna | | Page refresh | 0.67× time; 3–2–7 | 1.86× time; 4–2–6 | gpt-5.6-luna | | Reflect | 0.58× time; 7–9–12 | 2.55× time; 10–3–15, including 3 timeouts counted as worse | gpt-5.6-luna | Triplets are blind verdicts against gpt-5.6-luna, counted as better–tie–worse for the candidate: 17–5–2 means its output was judged better on 17 items, tied on 5 and worse on 2. Reflect and page refresh were judged by Claude. Time is relative to gpt-5.6-luna. Because each Hindsight operation can use its own model, the result depends on which operation is measured. A comparison of reflect alone would have missed gpt-6.1-sol’s retain result, and a comparison of retain alone would have missed that it exceeds the production time limits on reflect. The scripts and methodology behind this comparison are published as hindsight-llm-bench https://feniix-hq.net/projects/hindsight-llm-bench/ , for anyone who wants to repeat it on their own install or with other models.