My custom AI bot keeps giving bad advice because it cannot retain feedback from past calls

A developer has described a pattern for giving custom retrieval-augmented support bots a feedback loop that survives past a single call, by logging each call's question, answer and the ids of the stored notes it was built on, then attaching later verdicts (thumbs ratings, user replies, human handovers, repeat tickets) as ratings on those notes so recall can rank failed advice below advice that worked. The approach is offered as a fix for the common failure where a bot keeps repeating advice that did not work, such as telling customers to clear an app cache for a sync error, because each call ends before the outcome is known and similarity search treats a repeatedly wrong note as equal to a correct one.

Your support bot tells a customer that a sync error goes away if they clear the app cache. It does not. The customer writes back, a person on the team finds the real fix, re-authorising the integration, and the ticket closes. Next Tuesday another customer reports the same error, and the bot recommends clearing the cache again, with the same confidence. The short answer: each call ends before anyone knows whether its advice worked, so the feedback arrives with nothing to attach to. Keep a record of every call with the ids of the stored notes the answer was built on. When the verdict comes in, from the user, the ticket or the person who took over, send it back as a rating on those notes, and have the bot recall from a memory that reads ratings when it ranks. Next time, the advice that failed and the advice that worked are no longer equals. Nothing is erased: the failed advice stays on record, with its outcome attached. A typical custom bot is a model call wrapped in retrieval: a system prompt, a search over help articles or stored notes, and the question. The verdict on last week's answer is in none of those. It is in the customer's reply and the helpdesk. Teams usually try three fixes, in this order. Edit the system prompt. Add "do not suggest clearing the cache for sync errors". A prompt is the right home for policy, but it scales like any document edited by hand: every bad call adds a line, nobody prunes them, and important rules compete with stale ones for the model's attention. Store answers and corrections in a memory. The correction now survives the call, but so does the cache advice, and similarity search ranks by closeness to the question. Both notes match, both come back, and the model picks one in context. A wrong note is as similar to the question after ten failures as after none. Attach the outcome to the note. When a call's result is known, record it on the notes that shaped the answer. The failed note now carries its failure and the working one its success, and a recall that weighs outcomes can tell them apart. Only this fix has a path from what happened back to what gets recalled next. The answer is written during the call. The verdict arrives after it, sometimes days later, in another process. So the call must leave something for the verdict to find: Call record: what a bot keeps from each call so that a verdict arriving later can find what to judge: the question, the answer, and the ids of the stored notes the answer was built on, under a key the verdict will carry. The ids are per note, not per conversation. One piece of advice failed, not the exchange. If the model can say which recalled notes it relied on, keep those; otherwise keep every note that was in the prompt and accept that they move together. The key is one the verdict will carry: a ticket id, a thread id, the session id your feedback button already sends. When one key covers several answers, the code below keeps the latest. Keep call records in your own database, apart from the notes the bot recalls. Written into the same memory, every logged answer becomes one more candidate in the next recall, the bad ones included. Explicit ratings are rare. Other outcomes are often already recorded somewhere, and wiring those places in works better than waiting for a thumbs-down. | signal | when it arrives | what it says about the advice | what to send | |---|---|---|---| | thumbs up or down on the answer | seconds later, if at all | how the answer looked, often before it was tried | positive or negative | | the user says it did not work | the next message, or the next day | it was tried and it failed | negative | | a person takes over and solves it another way | when the handover closes | it failed, and here is what worked | negative, plus the fix as a new note | | the same user returns with the same problem | hours or days later | the first answer did not hold | negative, smaller | | the ticket closes and stays closed | after a quiet period you choose | it probably worked | positive, smaller | | nothing happens | never | nothing | nothing | Two rows need care. A handover for a reason the bot was never meant to handle, a refund or an angry customer, says nothing about the advice. And silence, the commonest row, is not a verdict. The third row does the most work. A rating can lower the advice that failed, but only a note can supply the advice that worked, so the person's fix goes into the memory the bot recalls from. A rating cannot change the answer already given. It changes the order of what the next recall brings back. Keep the failed note rather than deleting it. Clearing the cache is right for a different error, and a deleted note cannot answer "why did the bot tell customers to clear the cache in October?". Ratings tune ranking, not truth. If a help article is wrong, fix the article. The loop needs no particular product: a list of failed advice the bot checks approximates it, a store you run can carry an outcome field, a managed memory layer keeps the bookkeeping. What matters is the path from outcome back to recall. This uses mnemoverse 0.4.0, the Python SDK of Mnemoverse the author's company , set up as on the Python SDK page https://mnemoverse.com/docs/api/python-sdk . The model call is a placeholder, and call records live in SQLite. python import json import sqlite3 from mnemoverse import MnemoClient memory = MnemoClient reads MNEMOVERSE API KEY DOMAIN = "bot:support" db = sqlite3.connect "calls.db" db.execute "CREATE TABLE IF NOT EXISTS calls " " call id TEXT PRIMARY KEY, question TEXT, answer TEXT, used TEXT, concepts TEXT " def answer call id: str, question: str - str: recalled = memory.read question, domain=DOMAIN, top k=5 reply, relied on = bot.reply question, recalled.items your model call used = str item.atom id for item in relied on db.execute "INSERT OR REPLACE INTO calls VALUES ?, ?, ?, ?, ? ", latest answer per key call id, question, reply, json.dumps used , json.dumps recalled.query concepts db.commit return reply def on verdict call id: str, outcome: float, fix: str | None = None - None: row = db.execute "SELECT question, used, concepts FROM calls WHERE call id = ?", call id, .fetchone if row is None: return no call record: the verdict has nowhere to land question, used, concepts = row 0 , json.loads row 1 , json.loads row 2 if used: result = memory.feedback atom ids=used, outcome=outcome, -1.0 to +1.0 query concepts=concepts, domain=DOMAIN print "updated:", result.updated count if fix: what actually worked, from the person who solved it note = memory.write f"Problem: {question}\nWhat worked: {fix}", domain=DOMAIN if not note.stored: print "Not stored:", note.reason on verdict "ticket-1042", -1.0, fix="Re-authorise the integration under Settings, Connections." on verdict "ticket-1043", 1.0 the customer confirmed it worked feedback gets the same domain as the read; the SDK page notes that without it the SDK sends "general" . A write that is too similar to the nearest memory in the same domain is not stored, so the code checks stored . The API docs state the effect in one line: "Ratings re-rank what comes back next." The overview says what stays: "feedback changes the order of the next read, not the stored text". A bot on an MCP client rates through the memory feedback tool instead. answer , and check that the two call records share no id. Print the valence of the notes each call relied on, "Outcome polarity -1, +1 " in the API reference, and stop the process. updated count , "Number of atoms updated" in the reference, should equal the number of ids recorded for that call. If it is zero, check the ids and the domain first. The library page https://mnemoverse.com/docs/library/agent-repeats-mistakes follows the same loop for coding agents, where the verdict is a build or a test run, with the limits of outcome memory in more detail. Where does the verdict on your bot's answers end up today?