{"slug": "a-question-on-discord-found-two-bugs-in-my-rag-system-fixing-them-found-a-third", "title": "A question on Discord found two bugs in my RAG system. Fixing them found a third.", "summary": "A developer traced two reported bugs in a retrieval-augmented generation system back to flawed assumptions in chunk ranking and agent-memory recency scoring, then uncovered a third bug in a newly built Kafka Connect ingestion sink. The fixes group retrieved chunks by document id so a single source is cited once, and measure memory age from when a fact was last stated rather than last read, since refreshing on recall would let contradicting memories tie on recency. The developer notes the original test suite passed throughout because its recency test backdated the same created_at column the buggy query read.", "body_md": "*Every test passed. The tests had the same blind spots as the code.*\n\nA developer saw Ossian's MCP server in a community showcase and asked two questions:\n\nHow do you handle cases where retrieved docs and stored memories conflict? Or when multiple\n\npieces of context ultimately come from the same underlying source?\n\nI had a confident answer to both. Documents and agent memory live in separate tables, are reached\n\nthrough separate tools, and never meet in a prompt. Duplicates are caught at ingest by content hash.\n\nDone.\n\nBefore replying I read the code to make sure the answer was true. It was half true, and the half\n\nthat wasn't is the interesting part.\n\nRetrieval returns chunks, not documents. Ossian took the top six and numbered them for the model,\n\nso a prompt could look like this:\n\n```\n[1] engineering-handbook.txt  …\n[2] engineering-handbook.txt  …\n[3] engineering-handbook.txt  …\n[4] platform-architecture.md  …\n```\n\nTo the model — and to the person reading the citations — that is three sources agreeing and one\n\ndissenting. It is one source said three times. The system prompt even tells the model *\"if the context conflicts with itself, say which sources disagree\"*, and there was no way for it to tell one\n\nIngest-time deduplication does nothing here. These are distinct chunks of one legitimate document.\n\nThe fix groups chunks by document id before the prompt is built: one number per document, every\n\npassage kept under it, the document ranked by its best chunk. Two files that merely share a filename\n\nstay separate, because the grouping is on id. On a live question, six retrieved chunks now become\n\nfive citations, with one runbook contributing two passages under a single number.\n\nAgent memory is ranked `similarity × importance × 0.5^(age / 30 days)`. Recency matters for memory in\n\na way it never does for documents: a runbook from three years ago is as true as one from today, a\n\nstated preference from three years ago is not.\n\nThe question is what \"age\" means. The query measured it from `created_at`.\n\nAn agent that restates a fact — \"the user still prefers British English\" — hits a deduplicating\n\nupsert, which updates `updated_at` and nothing the ranking reads. A preference confirmed every day\n\nfor three months decayed exactly as if it had been said once, three months ago.\n\nThe obvious fix is wrong. Recall already records `last_used_at`, and a comment on it claimed that\n\nrecording use *\"keeps a live memory from decaying away.\"* It didn't — and it shouldn't. Recall\n\nreturns everything that matches, so an old preference and the newer one contradicting it are\n\nrecalled *together*. Refresh both on read and they tie on recency, which is the one signal that\n\nlets the newer one win.\n\nSo age now runs from the last time something was **said**, not the last time it was **read**.\n\nMeasured against the running system, after three recalls of both:\n\n| Memory | Score | \n|---|---|\n| \"switched the editor to the light theme\" (fresh) | 0.765 | \n| \"prefers the dark theme\" (90 days old) | 0.102 | \n| …then the dark-theme preference is restated | 0.817 | \n\nThe test suite had a recency test, and it passed the whole time. It backdated `created_at` — the\n\nsame column the bug read. The test and the code shared an assumption, so the test could only ever\n\nconfirm it. The new test fails against the old query; I checked by putting the old line back.\n\nWith those fixed, I built what I'd meant to build anyway: a Kafka Connect sink, so a table that\n\nDebezium streams into Kafka becomes a corpus that follows the database.\n\nIts first end-to-end run showed a batch of changes failing with HTTP 500, twice, then succeeding on\n\nthe third attempt. The sink's backoff did its job. The backend's log said:\n\n```\ninsert or update on table \"ingest_events\" violates foreign key constraint\nKey (document_id)=(…) is not present in table \"documents\".\n```\n\nA `DELETE` event removed the document, then recorded the event row pointing at the id it had just\n\ndeleted. The foreign key rejected it. Because the batch loop didn't catch it, every event in the\n\nbatch failed with it.\n\nAnd it only happened once per document. The retry found nothing to delete, recorded a null id, and\n\nsucceeded. Any pipeline that retries — which is every pipeline worth running — hid it completely.\n\nThere were no tests for the event API at all, so nothing else was going to.\n\nA few decisions that are easy to get wrong:\n\n**The event id comes from Kafka, not from Debezium.** Ossian's event API is idempotent on a\n\ncaller-supplied id, so the sink needs one that is stable under redelivery. Debezium's source position\n\nlooks ideal and isn't: every row of an initial snapshot shares one LSN, so two different rows would\n\ncollapse into one id and the second would be silently discarded as a duplicate. The sink uses the\n\nconnector name, topic, partition and offset, plus the record timestamp to survive a topic being\n\nrecreated with offsets starting from zero.\n\n**A blanked row is removed, not skipped.** If an update empties every text column, leaving the old\n\ndocument in place means the corpus keeps answering from text the source no longer has. If *none* of\n\nthe configured columns exist on the record, though, it's rejected as a misconfiguration — otherwise a\n\ntypo in `ossian.text.fields` deletes the whole table from the corpus.\n\n**Placeholders are refused.** With Postgres' default replica identity, an update that doesn't touch\n\na large column sends Debezium's `__debezium_unavailable_value` instead of the text. Indexing that\n\nwould replace a real article with a sentinel string.\n\n**Offsets never move ahead of delivery.** `put()` is synchronous. Rate limits and 5xx back off using\n\n`Retry-After` and retry; a record Ossian rejects goes to the dead-letter queue while the rest keep\n\nflowing; a 401 stops the task, so a bad key can't drain an entire topic into the DLQ.\n\nEnd to end against a Postgres table: a snapshot of three rows became three documents answerable with\n\ncitations; an update from 180 to 90 days changed the answer and left no old chunk still saying 180;\n\na delete removed the document and its chunks; a blanked row disappeared; and resetting the sink's\n\noffsets replayed the whole topic — 18 events before, 18 after, no new documents.\n\nThe original question isn't fully answered, and I said so in the reply.\n\nMemories have no link back to the document they were learned from, so a memory that goes stale when\n\nits source document changes is never flagged. And deduplication is exact: \"prefers dark mode\" and\n\n\"likes dark mode\" both persist, and a contradicting memory doesn't supersede the older one — both come\n\nback, and recency decides. Keeping documents and memory apart is staying. The rest is an\n\n[open design discussion](https://github.com/dockndevai/ossian/issues/3).\n\nNone of these three were exotic. Each survived because something *around* it agreed with it: a test\n\nthat backdated the same column the query read, a citation format that looked right in every\n\nscreenshot, a retry loop that turned a deterministic failure into a transient one.\n\nThe fastest way I know to find that kind of bug is to explain the system to someone who asks a\n\nprecise question — and check the code before you hit send.\n\n*Ossian: [dockndevai.github.io/ossian-site](https://dockndevai.github.io/ossian-site/) ·\n[github.com/dockndevai/ossian](https://github.com/dockndevai/ossian) ·\nKafka Connect sink: [github.com/dockndevai/ossian-kafka-connect](https://github.com/dockndevai/ossian-kafka-connect) ·\nIssues: [#1](https://github.com/dockndevai/ossian/issues/1),\n[#2](https://github.com/dockndevai/ossian/issues/2),\n[#5](https://github.com/dockndevai/ossian/issues/5)*\n\n*Originally published on [Medium](https://dockndev.medium.com/a-question-on-discord-found-two-bugs-in-my-rag-system-fixing-them-found-a-third-03a5cd6237d9).*\n\n*Written with Claude (Anthropic), working in the codebase it describes; every number and code sample\nwas verified against the running system. Reviewed and published by\n[@dockndevai](https://github.com/dockndevai).*", "url": "https://wpnews.pro/news/a-question-on-discord-found-two-bugs-in-my-rag-system-fixing-them-found-a-third", "canonical_source": "https://dev.to/dockndevai/a-question-on-discord-found-two-bugs-in-my-rag-system-fixing-them-found-a-third-58m6", "published_at": "2026-09-13 14:19:05+00:00", "updated_at": "2026-09-13 14:40:34.005091+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-agents", "ai-infrastructure", "mlops"], "entities": ["Ossian", "Kafka Connect", "Debezium", "Kafka", "Discord"], "alternates": {"html": "https://wpnews.pro/news/a-question-on-discord-found-two-bugs-in-my-rag-system-fixing-them-found-a-third", "markdown": "https://wpnews.pro/news/a-question-on-discord-found-two-bugs-in-my-rag-system-fixing-them-found-a-third.md", "text": "https://wpnews.pro/news/a-question-on-discord-found-two-bugs-in-my-rag-system-fixing-them-found-a-third.txt", "jsonld": "https://wpnews.pro/news/a-question-on-discord-found-two-bugs-in-my-rag-system-fixing-them-found-a-third.jsonld"}}