{"slug": "fine-tuned-decision-model-for-agent-memory", "title": "fine tuned decision model for agent memory", "summary": "Hindsight released three models built for agent memory, led by the 0.8B-parameter decision model hs-decide-1-flash, which the company reports makes six of seven LLMs give better answers across Hindsight while five get cheaper. On gemini-3.8-flash, the operations involving the decision model cost 2.5x less, spend 1.9x less time in model calls and score 2.2 points higher, with the LLM making 3.1 calls per case instead of 4.5. All three models are hosted behind the new Hindsight Inference API from $0.005 per million input tokens, with weights on Hugging Face under Apache 2.0 for the decision model and MIT for hs-rerank-1 and hs-embed-1.", "body_md": "# Introducing Hindsight Models: memory 2.5x cheaper and 2x faster, with better answers\n\nThe LLM writes, the decision model judges. hs-decide-1-flash takes over the small judgments inside Hindsight: better answers from six of seven LLMs, cheaper for five. With a reranker and an embedding model built for agent memory, open weights and hosted.\n\n**The LLM writes, the decision model judges.** Most of what an AI application asks an LLM isn't writing. It's small judgments: is this new, which of these matters, is this enough. Today we're releasing three models built for agent memory, starting with `hs-decide-1-flash`, a decision model that answers those judgments directly, so the LLM is left with the writing.\n\n## TL;DR\n\n- **Better and cheaper.** With`hs-decide-1-flash` making the small judgments, six of seven LLMs give better answers across Hindsight, and five get cheaper. On`gemini-3.8-flash` , the operations the decision model takes part in cost 2.5x less, spend 1.9x less time in model calls and score 2.2 points higher.\n- **New: the Hindsight Inference API, and open weights.** All three models are hosted behind one API, from $0.005 per million input tokens, that works with Cohere and OpenAI clients; new accounts get $5 of free credit.**[Sign up for Hindsight Inference →](https://inference.hindsight.vectorize.io)** ([docs](https://docs.hindsight.vectorize.io/cloud/inference/overview/) ) The weights are on[Hugging Face](https://huggingface.co/hindsight-ai) , the decision model under Apache 2.0 and the other two under MIT.\n- **A cheap LLM near a frontier one.**`glm-5.3-flash` with the decision model scores within 2 points of`gemini-3.8-flash` alone, at 19x less cost.\n- **167 ms per decision-model request** at the median; an LLM call takes 1 to 7 seconds.\n- **Leaner recall.** Recall can drop irrelevant and repeated memories before they reach your agent, so your agent's own LLM reads fewer tokens.\n- **A reranker and an embedding model built for agent memory.**`hs-rerank-1` and`hs-embed-1` score highest overall against the small open models people run today, in eight languages, and run on a CPU.\n\n## Judgments are most of the work\n\nA memory system is full of judgments. Between storing a memory and recalling it, Hindsight asks: is this fact new, or already known? Does it change what we believe? Which of these memories matter for this question? Is this evidence enough to answer? Could this new memory change that knowledge page?\n\nHindsight used to ask an LLM every time. Each answer was a full generation call: seconds of latency, a bill for every token of context, and free text to parse back into a yes or no. On our evals, with a decision model answering the judgments, the LLM makes about a third fewer calls: 3.1 per case instead of 4.5.\n\nA decision model is built for exactly this shape of question. You give it the state and the question, and it returns a typed answer: a choice with a probability for each option, a point on a scale, or how likely something is true. It doesn't generate text, so it answers in a fraction of the time and price.\n\n## Where it started\n\nWe first tried this with [Jev](https://typesafe.ai), TypeSafe's decision model, and saw how much of Hindsight's work had never needed an LLM. So we trained our own on the judgments Hindsight makes: 0.8B parameters, open weights, and the same System One requests, so an app written for one runs on the other by changing the model name.\n\n## How Hindsight uses it\n\nHindsight asks the decision model at four points. Without one, each runs exactly as before.\n\n**Recall: a smaller bill for your agent.** Every memory recall returns ends up in your agent's prompt, and for most teams the agent's own LLM is the biggest cost they have. Recall used to fill its token budget whether the memories helped or not. With a decision model, it can hand your agent only what answers the question:\n\n- **Prune** drops the memories that are no help to the query. It reuses the ranking answers, so it costs no extra request, and the best memory always stays.\n- **Dedup** drops a memory that repeats a better-ranked one, so a fact retained from several documents reaches the agent once. It costs one more decision-model request, asked only about memories whose embeddings are alike; memories dated differently, or of two kinds, are never called repeats.\n\nFewer, more relevant memories means fewer tokens on every agent turn, and less noise for the agent to reason past. Both are off by default; turn them on per request (`\"rerank\": {\"prune\": true, \"dedup\": true}`) or per bank. Reflect and page refreshes search through recall, so they get it too. ([Recall docs](https://hindsight.vectorize.io/developer/api/recall#rerank))\n\n**Reflect: is this evidence enough?** Reflect answers a question by searching memory in rounds. Before, an LLM agent ran each round: it picked a search tool, wrote a query, read every result and decided whether to search again. With a decision model:\n\n1. One LLM call turns the request into a search query, or finds there's nothing to search and returns at once.\n2. The searches run in parallel with that query.\n3. The decision model **trims the evidence** : it ranks what came back against the request and cuts what doesn't bear on it, so the LLM reads less.\n4. It **judges whether the evidence already answers the request** . If yes, the next LLM call writes the answer. If not, the LLM writes the next query and another round runs.\n5. When the bank has knowledge pages, it first asks **whether the pages alone answer it** , and skips the raw search when they do.\n\nIf the decision model fails, reflect falls back to agent turns.\n\n**Consolidation: is this fact new?** Consolidation turns new facts into observations, the bank's current beliefs. Before, every new fact went to the LLM with the observations it might touch. With a decision model, it settles the easy facts first, one question per fact: one that **restates** an observation is attached as evidence and **chatter** is skipped, neither with an LLM call. Only the facts that change what an observation says, add something new, or that it isn't sure about, go to the LLM, which writes every observation's text.\n\n**Knowledge pages: could this change the page?** Rebuilding a page runs reflect, so it gets everything above. Updating a page with what's new is where the saving is. Before, every update ran a reflect over the new memories and rewrote the page. With a decision model, it asks of each new memory *could this change the page?* and edits the page from those alone; when none can, the update makes no LLM call at all.\n\n## Results\n\nEvery number here is on [benchmarks.hindsight.vectorize.io](https://benchmarks.hindsight.vectorize.io): Hindsight's own system evals, run on Hindsight 1.2.0 against a fresh server through the public API. Each LLM runs alone and with `hs-decide-1-flash` on the same cases, all graded by the same judge, `gemini-3.1-flash-lite` (also one of the seven LLMs tested, so its own rows are self-graded).\n\n- **Quality** is the share of cases passed, each graded by the judge; for all of Hindsight, the mean over operations.\n- **Cost** is the tokens really used at list prices, decision-model tokens included at $0.02 per million, per thousand cases. \"All of Hindsight\" is an equal mix of the eval's operations (retain, consolidation, reflect, knowledge pages); a real workload weighs them differently.\n- **Time** is the time spent in model calls (LLM and decision model) per operation, from the same runs' traces, not end-to-end latency.\n\nEach chart plots quality against cost, one point per LLM; an arrow joins an LLM to the same LLM with the decision model. Up and to the left is better.\n\n### All of Hindsight\n\nQuality goes up for six of the seven LLMs, most for the ones that need it most (`glm-5.3-flash` 73.8 → 87.1, `gemma-4-26b-a4b` 76.6 → 81.1, `deepseek-v4.1-flash` 81.9 → 85.1), and holds level for `gemini-3.1-flash-lite` (87.1 → 86.6). The LLM makes about a third fewer calls and reads half the input tokens or less, so five of them get cheaper: `gemini-3.8-flash` $147 → $61 (2.4x less), `gemini-3.1-flash-lite` $42 → $22, `claude-haiku-5.5` $31 → $18, `gpt-6-luna` $16 → $9.88, `gemma-4-26b-a4b` $16 → $12. Time in model calls drops for all seven, from 1.2x (`gemini-3.1-flash-lite`) to 1.9x (` gemini-3.8-flash`).\n\nRetain doesn't use the decision model, so it costs the same either way and waters down the totals above. Leave it out and count only consolidation, reflect and knowledge pages, the operations the decision model takes part in: on `gemini-3.8-flash` they cost 2.5x less, spend 1.9x less time in model calls, and score 2.2 points higher.\n\n**A cheap LLM gets close to a frontier setup.** `glm-5.3-flash` with the decision model scores 87.1 at $7.72 per thousand cases, and `gpt-6-luna` with it scores 87.0 at $9.88. `gemini-3.8-flash` alone scores 89.0 at $147: about 2 points more, at 15–19x the cost. Switching to a cheaper LLM and adding the decision model is the biggest saving in these results.\n\n**Faster calls.** A decision-model request takes 167 ms at the median (350 ms at p90) and answers every question about a fact or a memory at once; the LLM call it replaces takes 1 to 7 seconds at the median, depending on the model.\n\n**The decision model is a small part of the bill for most LLMs.** Its own tokens cost $4 to $7 per thousand cases: $3.93 of the $61 on `gemini-3.8-flash`. On the cheapest LLMs that's most of the bill, which is why they cost more with it (below).\n\n### The trade-offs\n\nThe decision model makes Hindsight cheaper and faster almost everywhere, and better in most places, but not all:\n\n- **Reflect** gives the biggest saving and is where quality dips. On`gemini-3.8-flash` , reflect costs 4.6x less, and quality goes from 94.1 to 91.2. Four of the seven LLMs dip by about 3 points, and`gemini-3.1-flash-lite` drops from 94.1 to 85.3.\n- **The two cheapest LLMs get much better answers, for a little more money.**`glm-5.3-flash` goes from 73.8 to 87.1 across Hindsight ($4.58 → $7.72 per thousand cases), and from 36 to 84 on consolidation.`deepseek-v4.1-flash` goes from 81.9 to 85.1 ($10.57 → $12.21), and from 68 to 86 on consolidation. They cost more because the LLM is so cheap that the decision model's own tokens outweigh what it saves.\n\n### Reflect\n\nFive of the seven arrows point left. The LLM takes about half the turns (4.4 → 2.5 per question) and reads a fraction of the evidence, because the decision model trims it first: `gemini-3.8-flash` reads 7.5K input tokens per question instead of 45K. Cost drops 4.6x on `gemini-3.8-flash` ($36 → $7.91 per thousand questions), 2.9x on `gemini-3.1-flash-lite`, 2.2x on `claude-haiku-5.5` and 2x on `gpt-6-luna`; time in model calls drops 1.1x to 2.1x. Quality rises for `glm-5.3-flash` (91.2 → 94.1), holds for `gpt-6-luna` (94.1), dips about 3 points for four LLMs, and drops from 94.1 to 85.3 for `gemini-3.1-flash-lite`.\n\n### Knowledge pages\n\nHere the arrows point up and left at once. `gpt-6-luna` goes from 82.5 to 90.0 at less than half the cost ($7.33 → $3.16), `gemini-3.1-flash-lite` from 85.0 to 90.0 at a third ($17 → $5.78), `gemini-3.8-flash` from 82.5 to 90.0 at 3.6x less ($65 → $18). Five of the seven LLMs keep pages more accurately, and the LLM makes half the calls (about 8.5 → 4 per case), since a memory that can't change the page never reaches it.\n\n### Consolidation\n\nEvery arrow points up: all seven LLMs keep observations more accurately with the decision model. The biggest jump is `glm-5.3-flash`, from 36 to 84. Alone, its calls all succeed, but it rarely updates the observations: most of its failed checks are a new fact that no observation cites. With the decision model settling the restatements and chatter first, the LLM only gets the few facts that need rewriting, and it handles those. `deepseek-v4.1-flash` goes from 68 to 86 and `gemma-4-26b-a4b` from 66 to 82. Here the gain is quality and time rather than money: the LLM makes about 20% fewer calls, consolidation spends up to 2.3x less time in model calls, and cost stays about level.\n\n\"Is this new, or already known?\" is a question your own app may ask too. Outside Hindsight, it's one call to the [Decisions API](#hindsight-inference).\n\n## Use them with Hindsight\n\n**Every result above, in your own Hindsight: one login and one environment variable, no GPU.** In open-source Hindsight 1.2 the decision model is off until you set one up. With the `hindsight` provider it runs consolidation, reflect and knowledge pages, and recall keeps its reranker unless you turn it on for recall too. New accounts start with $5 of free credit, enough for the decision model's part of about 4,000 reflects on our evals.\n\n```\nhindsight-admin cloud login     # signs in to Hindsight Cloud and saves a keyexport HINDSIGHT_API_DECISION_MODEL_PROVIDER=hindsightexport HINDSIGHT_API_RECALL_DECISION_MODEL_PROVIDER=hindsight   # optional: rank recall with it tooexport HINDSIGHT_API_RECALL_DECISION_MODEL_PRUNE=true          # optional: drop results that don't helpexport HINDSIGHT_API_RECALL_DECISION_MODEL_DEDUP=true          # optional: drop repeatsexport HINDSIGHT_API_RERANKER_PROVIDER=hindsightexport HINDSIGHT_API_EMBEDDINGS_PROVIDER=hindsight\n```\n\nThe first two lines are all the results above need; the rest are optional. If you'd rather run everything yourself, the reranker and the embedding model run on your own machine, inside the Hindsight API process:\n\n```\nexport HINDSIGHT_API_RERANKER_PROVIDER=localexport HINDSIGHT_API_RERANKER_LOCAL_MODEL=hindsight-ai/hs-rerank-1export HINDSIGHT_API_EMBEDDINGS_PROVIDER=localexport HINDSIGHT_API_EMBEDDINGS_LOCAL_MODEL=hindsight-ai/hs-embed-1\n```\n\nA bank's stored vectors must be re-embedded to switch embedding models. To serve the decision model yourself on vLLM, see [Run It Yourself](https://hindsight.vectorize.io/developer/decision-models#run-it-yourself); for every setting, see [Decision Models](https://hindsight.vectorize.io/developer/decision-models) and [Models](https://hindsight.vectorize.io/developer/models).\n\n## Open weights on Hugging Face\n\nAll three models are in the [`hindsight-ai`](https://huggingface.co/hindsight-ai) organization on Hugging Face, and all three work in English, Spanish, French, German, Portuguese, Japanese, Korean and Chinese.\n\n| Model | What it does | Size | Built on | License | Context | \n|---|---|---|---|---|---|\n| [`hs-decide-1-flash`](https://huggingface.co/hindsight-ai/hs-decide-1-flash) | Typed judgments: choice, score, how likely | 0.8B | `Qwen/Qwen3.5-0.8B` | Apache 2.0 | 65,536 tokens | \n| [`hs-rerank-1`](https://huggingface.co/hindsight-ai/hs-rerank-1) | Orders documents by relevance to a query | 114M | `jhu-clsp/mmBERT-small` | MIT | 8,192 tokens | \n| [`hs-embed-1`](https://huggingface.co/hindsight-ai/hs-embed-1) | Turns text into 384-dimension vectors | 112M | `hotchpotch/bekko-embedding-v1-a25m` | MIT | 8,192 tokens | \n\nThe decision model answers every question about a request in one pass and serves on vLLM with the plugin in its repository's `serving/` folder. The reranker and the embedding model run 15M- and 13M-parameter transformers per pass (most of their weights are the vocabulary table), so they're fast on CPU; both ship an int8 ONNX export and serve on Text Embeddings Inference as is.\n\n### How they compare\n\nFull tables: [Hindsight Models benchmarks](https://benchmarks.hindsight.vectorize.io/models).\n\n**Decision model.** We compared `hs-decide-1-flash` with the other open decision models of its size: `StartLux-Decision 0.8B Instant`, `d3-lite` (vLLM Semantic Router) and `decider-0.8b` (Mapika). It has the best mean over eight benchmarks (75.6, against 74.5, 73.4 and 66.6), and its lead is in the memory judgments Hindsight makes: telling whether a new memory changes a page (79.5, against 56.0–72.0) and skipping refreshes that wouldn't change anything (89.4, against 45.0–78.3). On general decision benchmarks it trails: the public Decision Index (36.1, against 53.8 for StartLux) and dialogue inference (75.8, against 83.0 for StartLux and 88.0 for `d3-lite`).\n\n**Reranker.** For a Hindsight user, it means better recall in eight languages out of the box, and the hosted version takes reranking off your API server's CPU. We compared it against open rerankers up to 278M parameters, on seven benchmarks: English (NanoBEIR, BRIGHT), multilingual (MIRACL, and a bank of 26 tasks in 8 languages) and memory (KnowMeBench, MemBench, ConvoMem). Scores are nDCG@10; speed is the median time to rerank 50 memories on two CPU threads.\n\n**Embeddings.** For a Hindsight user, it means memories found by meaning in eight languages, and over long conversations and documents where small English models fall off. We compared it against open embedding models up to 118M parameters, on twelve benchmarks: English and multilingual retrieval, long conversations (BEAM), long documents (LongEmbed) and memory retrieval (LMEB). Speed is the median time per query on two CPU threads. The image shows one benchmark per group; the mean covers all twelve, and the full table is one click away. The fair comparison is with the multilingual models; the English-only ones are listed because they’re common defaults.\n\n## Hindsight Inference\n\nHindsight Inference serves the same three models to any application. It's part of Hindsight Cloud: sign in at [inference.hindsight.vectorize.io](https://inference.hindsight.vectorize.io) with your Cloud account (same organization, teams and credit balance), and new accounts start with $5 of free credit.\n\n**Try every model in the Playground before writing any code.** Pick reranking, embeddings or a decision, and it comes prefilled with an example you can edit. Each run shows the answer, how long it took, the tokens and what it cost. Here the reranker, asked which company Mia joined as principal ML engineer, puts the right memory first, and with **Drop irrelevant documents** on it hands back two of the four instead of all of them, the same pruning recall uses:\n\nWhen it does what you want, the Playground shows the same request as code (curl, Python, the Cohere or OpenAI SDK, or a Hindsight config) to copy into your app; create a key under **API Keys** and you're done. Every API is a drop-in for one you may already use:\n\n| API | Endpoint | Models | Drop-in for | Price per million input tokens | \n|---|---|---|---|---|\n| Decisions | `POST /v1/systemone` | `hs-decide-1-flash` | System One | $0.02 | \n| Reranker | `POST /v1/rerank` | `hs-rerank-1` , or`hs-decide-1-flash` to also drop irrelevant documents | Cohere Rerank | $0.015 / $0.02 | \n| Embeddings | `POST /v1/embeddings` | `hs-embed-1` | OpenAI Embeddings | $0.005 | \n\n**Already on Cohere Rerank?** Point your client here and set the model to `hs-decide-1-flash`: same code, and it hands back only what's relevant.\n\nNone of the three generates text, so you pay for input tokens only. Point your Cohere client at `https://inference.hindsight.vectorize.io` or your OpenAI client at `https://inference.hindsight.vectorize.io/v1`, change the model name, and keep the rest of your code. Full reference: [Inference docs](https://docs.hindsight.vectorize.io/cloud/inference/overview/).\n\n## Try it\n\n- **Self-hosted Hindsight:** run`hindsight-admin cloud login` and[turn on the decision model](https://hindsight.vectorize.io/developer/decision-models) . $5 of free credit covers the decision model for about 4,000 reflects.\n- **Your own app:**[open the Playground](https://inference.hindsight.vectorize.io/playground) , try the three models, then create an API key ([Inference docs](https://docs.hindsight.vectorize.io/cloud/inference/overview/) ).\n- **Open weights:** get them on[Hugging Face](https://huggingface.co/hindsight-ai) , and see every result on[benchmarks.hindsight.vectorize.io](https://benchmarks.hindsight.vectorize.io) .\n- Star us on [GitHub](https://github.com/vectorize-io/hindsight) .", "url": "https://wpnews.pro/news/fine-tuned-decision-model-for-agent-memory", "canonical_source": "https://hindsight.vectorize.io/blog/2026/11/03/hindsight-models", "published_at": "2026-11-03 16:47:17+00:00", "updated_at": "2026-11-03 16:52:47.229439+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "large-language-models", "ai-infrastructure"], "entities": ["Hindsight", "hs-decide-1-flash", "hs-rerank-1", "hs-embed-1", "Hindsight Inference API", "Hugging Face", "TypeSafe", "Jev"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/fine-tuned-decision-model-for-agent-memory", "markdown": "https://wpnews.pro/news/fine-tuned-decision-model-for-agent-memory.md", "text": "https://wpnews.pro/news/fine-tuned-decision-model-for-agent-memory.txt", "jsonld": "https://wpnews.pro/news/fine-tuned-decision-model-for-agent-memory.jsonld"}}