{"slug": "the-boring-backend-work-that-made-payecho-s-memory-actually-reliable", "title": "The Boring Backend Work That Made PayEcho's Memory Actually Reliable", "summary": "An engineer on the PayEcho team described building the backend that makes the product's Hindsight memory layer reliable, using a synced_to_memory flag on a recovery_attempts table and a per-row sync job so finalized invoice-recovery events become narrative memories exactly once. The pipeline only writes a memory after an outcome is final, and the flag is flipped per row after each successful write so a mid-batch crash cannot produce duplicate memories. The work addresses duplicate webhook retries that previously caused the recommendation agent to see the same payment outcome multiple times.", "body_md": "Everyone wants to talk about the AI part. Nobody wants to talk about the part where an invoice update arrives twice because a webhook retried, and now the agent has “remembered” the same payment outcome three times. That's the part I owned, and it turns out it matters more than the prompt.\n\nPayEcho recommends how to recover an overdue invoice — which channel to use, when to follow up, whether a customer's history should factor into a new credit decision — by recalling what actually happened with that customer before. That recall only works if the data feeding it is correct, deduplicated, and current. My piece was the backend: the schema, the sync pipeline that turns transactional events into memories, and the API layer the dashboard and the recommendation agent both depend on. It's easy to think of Hindsight as “where the AI part lives” and the backend as plumbing. In practice, the plumbing decides what the AI part is even allowed to know.\n\n01\n\nTwo stores, one truth\n\nPayEcho keeps two separate stores, on purpose. A relational database holds the transactional truth — customers, invoices, payment records, recovery actions, timestamps. Hindsight holds the interpreted version of that truth: short narrative memories tied to a customer, built for recall, not for joins.\n\nCREATE TABLE recovery_attempts (\n\n    id            UUID PRIMARY KEY DEFAULT gen_random_uuid(),\n\n    customer_id   UUID NOT NULL REFERENCES customers(id),\n\n    invoice_id    UUID NOT NULL REFERENCES invoices(id),\n\n    channel       TEXT NOT NULL,\n\n    sent_at       TIMESTAMPTZ NOT NULL,\n\n    responded_at  TIMESTAMPTZ,\n\n    outcome       TEXT,\n\n    synced_to_memory BOOLEAN NOT NULL DEFAULT FALSE\n\n);\n\nThat synced_to_memory flag is the one column that made the rest of the pipeline sane. Without it, I had no reliable way to answer “has this event already become a memory,” and the agent started seeing duplicate versions of the same outcome — which, if you're not paying attention, looks exactly like the agent being more confident about a pattern than it actually is.\n\n02\n\nDeciding when a row becomes a memory\n\nThe interesting backend problem was never storing the data; it was deciding when a transactional row becomes worth writing to memory, and how to phrase it. A row only becomes a memory once its outcome is final, since a reminder that hasn't been responded to yet isn't a story worth telling the agent:\n\ndef sync_pending_attempts():\n\n    attempts = db.query(\n\n        \"SELECT * FROM recovery_attempts \"\n\n        \"WHERE outcome IS NOT NULL AND synced_to_memory = FALSE\"\n\n    )\n\n```\nfor attempt in attempts:\n    days = (attempt.responded_at - attempt.sent_at).days if attempt.responded_at else None\n\n    memory.retain(\n        agent_id=\"payecho-recovery\",\n        user_id=str(attempt.customer_id),\n        content=build_narrative(attempt, days),\n        metadata={\n            \"invoice_id\": str(attempt.invoice_id),\n            \"channel\": attempt.channel,\n            \"outcome\": attempt.outcome,\n        },\n    )\n\n    db.execute(\n        \"UPDATE recovery_attempts SET synced_to_memory = TRUE WHERE id = %s\",\n        (attempt.id,),\n    )\n```\n\nbuild_narrative() is a small function, but it's the one that decides what the agent actually gets to reason over — it turns a row into a sentence like “sent a reminder via WhatsApp, customer responded after 3 days, outcome: paid.” The sync job runs on a schedule, picks up anything finalized and unsynced, writes it, and only then flips the flag. If the job crashes mid-batch, nothing gets double-written, because the flag update happens per row, right after that row's memory write succeeds, rather than as one commit at the end of the batch.\n\n■ TERMINAL — the workspace the sync job and memory panel run in day to day.\n\n03\n\nOne API, not two\n\nThe dashboard and the recommendation agent both need customer data, but they need different shapes of it — the dashboard wants aggregates like outstanding, overdue, and recovered totals, while the agent wants a specific customer's recalled memories plus their live invoice status. Rather than let two parts of the system fetch that separately, I built one internal API that both read from, so they never drift out of sync with each other:\n\n@app.get(\"/customers/{customer_id}/context\")\n\ndef get_customer_context(customer_id: str):\n\n    invoices = db.get_open_invoices(customer_id)\n\n    memories = memory.recall(\n\n        agent_id=\"payecho-recovery\",\n\n        user_id=customer_id,\n\n        query=\"past recovery attempts, channel responses, and payment outcomes\",\n\n    )\n\n    return {\n\n        \"open_invoices\": invoices,\n\n        \"recalled_history\": memories,\n\n    }\n\nThis one endpoint feeds both the customer detail page and the recommendation prompt. Early on I had two separate code paths doing roughly the same fetch, and they'd occasionally disagree — the dashboard would show an invoice as paid while the agent's recall still referenced it as outstanding, because one path was reading a cache the other wasn't. Collapsing them into a single endpoint got rid of that class of bug entirely.\n\n■ EXPECTED OUTPUT  — the same customer context, read from the one endpoint, behind both the dashboard and the recommendation.\n\n04\n\nThe case that proved it\n\nThe clearest proof this backend work paid off showed up in a case that had nothing to do with prompt wording at all. A payment came in through a bank reconciliation import, not through the WhatsApp or email flow the agent was watching. Because the reconciliation import wrote to the same invoices table and went through the same sync job, the agent picked it up on the next recall exactly like any other outcome, and stopped recommending follow-ups for an invoice that was already settled.\n\nI hadn't written any agent-side logic for “ignore invoices paid through other channels” — it fell out of the fact that there was only one path from “something happened” to “the agent knows about it,” regardless of which part of the system noticed it first.\n\nThat's the part that convinced me the backend design mattered as much as the memory design: an agent is only as reliable as the pipeline that decides what counts as an event worth remembering.\n\n05\n\nHow it fits together\n\nNone of this lives in the recommendation logic. It sits underneath it — the schema, the sync job, and the one shared API that both the dashboard and the agent read from, so nothing downstream ever has to guess which version of a customer's history is current.\n\n.\n\n06\n\nWhat I'd rebuild from day one\n\n● Design idempotency in from the start. I added the synced_to_memory flag only after seeing a duplicate memory in testing for the second time, and rebuilding around it after the fact cost more time than starting with it would have.\n\n● Don't sync in-flight events. My first attempt wrote a memory the moment a reminder was sent and then tried to update it once a response came in, but Hindsight memories aren't really designed to be mutated like a database row — writing once, after the outcome is known, produced far cleaner recall.\n\n● One API beats two parallel fetches. Anywhere the dashboard and the agent both need the same underlying data, parallel fetch paths eventually drift — and when they drift, the agent's recommendation and the human's dashboard can quietly disagree with each other, which is worse than either being wrong on its own.\n\n● Commit per row, not per batch. Batch sync jobs needed per-row commits rather than per-batch ones, so a crash mid-batch wouldn't lose progress or risk double-writing.\n\nMost of the bugs that actually affected the agent's behavior weren't in the recommendation logic at all — they were in the pipeline deciding what became a memory and when, which is a boundary that deserved more attention early on than I initially gave it. If you're building an agent whose intelligence depends on remembering real outcomes, the backend that decides what “real” and “final” mean is not a supporting detail. It's the thing that determines whether the memory layer is trustworthy at all.\n\nRESOURCES\n\nHindsight on GitHub →\n\nHindsight Documentation →\n\nAgent Memory — Vectorize →", "url": "https://wpnews.pro/news/the-boring-backend-work-that-made-payecho-s-memory-actually-reliable", "canonical_source": "https://dev.to/aparna8074/the-boring-backend-work-that-made-payechos-memory-actually-reliable-n7j", "published_at": "2026-09-27 17:06:20+00:00", "updated_at": "2026-09-27 17:31:13.668355+00:00", "lang": "en", "topics": ["ai-agents", "ai-infrastructure", "mlops", "ai-products"], "entities": ["PayEcho", "Hindsight"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/the-boring-backend-work-that-made-payecho-s-memory-actually-reliable", "markdown": "https://wpnews.pro/news/the-boring-backend-work-that-made-payecho-s-memory-actually-reliable.md", "text": "https://wpnews.pro/news/the-boring-backend-work-that-made-payecho-s-memory-actually-reliable.txt", "jsonld": "https://wpnews.pro/news/the-boring-backend-work-that-made-payecho-s-memory-actually-reliable.jsonld"}}