The Boring Backend Work That Made PayEcho's Memory Actually Reliable

An engineer on the PayEcho team described building the backend that makes the product's Hindsight memory layer reliable, using a synced_to_memory flag on a recovery_attempts table and a per-row sync job so finalized invoice-recovery events become narrative memories exactly once. The pipeline only writes a memory after an outcome is final, and the flag is flipped per row after each successful write so a mid-batch crash cannot produce duplicate memories. The work addresses duplicate webhook retries that previously caused the recommendation agent to see the same payment outcome multiple times.

Everyone wants to talk about the AI part. Nobody wants to talk about the part where an invoice update arrives twice because a webhook retried, and now the agent has “remembered” the same payment outcome three times. That's the part I owned, and it turns out it matters more than the prompt. PayEcho recommends how to recover an overdue invoice — which channel to use, when to follow up, whether a customer's history should factor into a new credit decision — by recalling what actually happened with that customer before. That recall only works if the data feeding it is correct, deduplicated, and current. My piece was the backend: the schema, the sync pipeline that turns transactional events into memories, and the API layer the dashboard and the recommendation agent both depend on. It's easy to think of Hindsight as “where the AI part lives” and the backend as plumbing. In practice, the plumbing decides what the AI part is even allowed to know. 01 Two stores, one truth PayEcho keeps two separate stores, on purpose. A relational database holds the transactional truth — customers, invoices, payment records, recovery actions, timestamps. Hindsight holds the interpreted version of that truth: short narrative memories tied to a customer, built for recall, not for joins. CREATE TABLE recovery attempts id UUID PRIMARY KEY DEFAULT gen random uuid , customer id UUID NOT NULL REFERENCES customers id , invoice id UUID NOT NULL REFERENCES invoices id , channel TEXT NOT NULL, sent at TIMESTAMPTZ NOT NULL, responded at TIMESTAMPTZ, outcome TEXT, synced to memory BOOLEAN NOT NULL DEFAULT FALSE ; That synced to memory flag is the one column that made the rest of the pipeline sane. Without it, I had no reliable way to answer “has this event already become a memory,” and the agent started seeing duplicate versions of the same outcome — which, if you're not paying attention, looks exactly like the agent being more confident about a pattern than it actually is. 02 Deciding when a row becomes a memory The interesting backend problem was never storing the data; it was deciding when a transactional row becomes worth writing to memory, and how to phrase it. A row only becomes a memory once its outcome is final, since a reminder that hasn't been responded to yet isn't a story worth telling the agent: def sync pending attempts : attempts = db.query "SELECT FROM recovery attempts " "WHERE outcome IS NOT NULL AND synced to memory = FALSE" for attempt in attempts: days = attempt.responded at - attempt.sent at .days if attempt.responded at else None memory.retain agent id="payecho-recovery", user id=str attempt.customer id , content=build narrative attempt, days , metadata={ "invoice id": str attempt.invoice id , "channel": attempt.channel, "outcome": attempt.outcome, }, db.execute "UPDATE recovery attempts SET synced to memory = TRUE WHERE id = %s", attempt.id, , build narrative is a small function, but it's the one that decides what the agent actually gets to reason over — it turns a row into a sentence like “sent a reminder via WhatsApp, customer responded after 3 days, outcome: paid.” The sync job runs on a schedule, picks up anything finalized and unsynced, writes it, and only then flips the flag. If the job crashes mid-batch, nothing gets double-written, because the flag update happens per row, right after that row's memory write succeeds, rather than as one commit at the end of the batch. ■ TERMINAL — the workspace the sync job and memory panel run in day to day. 03 One API, not two The dashboard and the recommendation agent both need customer data, but they need different shapes of it — the dashboard wants aggregates like outstanding, overdue, and recovered totals, while the agent wants a specific customer's recalled memories plus their live invoice status. Rather than let two parts of the system fetch that separately, I built one internal API that both read from, so they never drift out of sync with each other: @app.get "/customers/{customer id}/context" def get customer context customer id: str : invoices = db.get open invoices customer id memories = memory.recall agent id="payecho-recovery", user id=customer id, query="past recovery attempts, channel responses, and payment outcomes", return { "open invoices": invoices, "recalled history": memories, } This one endpoint feeds both the customer detail page and the recommendation prompt. Early on I had two separate code paths doing roughly the same fetch, and they'd occasionally disagree — the dashboard would show an invoice as paid while the agent's recall still referenced it as outstanding, because one path was reading a cache the other wasn't. Collapsing them into a single endpoint got rid of that class of bug entirely. ■ EXPECTED OUTPUT — the same customer context, read from the one endpoint, behind both the dashboard and the recommendation. 04 The case that proved it The clearest proof this backend work paid off showed up in a case that had nothing to do with prompt wording at all. A payment came in through a bank reconciliation import, not through the WhatsApp or email flow the agent was watching. Because the reconciliation import wrote to the same invoices table and went through the same sync job, the agent picked it up on the next recall exactly like any other outcome, and stopped recommending follow-ups for an invoice that was already settled. I hadn't written any agent-side logic for “ignore invoices paid through other channels” — it fell out of the fact that there was only one path from “something happened” to “the agent knows about it,” regardless of which part of the system noticed it first. That's the part that convinced me the backend design mattered as much as the memory design: an agent is only as reliable as the pipeline that decides what counts as an event worth remembering. 05 How it fits together None of this lives in the recommendation logic. It sits underneath it — the schema, the sync job, and the one shared API that both the dashboard and the agent read from, so nothing downstream ever has to guess which version of a customer's history is current. . 06 What I'd rebuild from day one ● Design idempotency in from the start. I added the synced to memory flag only after seeing a duplicate memory in testing for the second time, and rebuilding around it after the fact cost more time than starting with it would have. ● Don't sync in-flight events. My first attempt wrote a memory the moment a reminder was sent and then tried to update it once a response came in, but Hindsight memories aren't really designed to be mutated like a database row — writing once, after the outcome is known, produced far cleaner recall. ● One API beats two parallel fetches. Anywhere the dashboard and the agent both need the same underlying data, parallel fetch paths eventually drift — and when they drift, the agent's recommendation and the human's dashboard can quietly disagree with each other, which is worse than either being wrong on its own. ● Commit per row, not per batch. Batch sync jobs needed per-row commits rather than per-batch ones, so a crash mid-batch wouldn't lose progress or risk double-writing. Most of the bugs that actually affected the agent's behavior weren't in the recommendation logic at all — they were in the pipeline deciding what became a memory and when, which is a boundary that deserved more attention early on than I initially gave it. If you're building an agent whose intelligence depends on remembering real outcomes, the backend that decides what “real” and “final” mean is not a supporting detail. It's the thing that determines whether the memory layer is trustworthy at all. RESOURCES Hindsight on GitHub → Hindsight Documentation → Agent Memory — Vectorize →