Originally published at aideazz.xyz — cross-posted here with canonical link. A newly added data source ran on schedule for a full day, logged a healthy count every hour, and delivered nothing at all downstream. The system reported success, but the output queue was empty. This was not a silent failure; it was a loud success that produced nothing. The root cause was a dedup ledger stamping items as handled before a processing cap was applied. My AI agent queue incident meant 175 items were marked as done without ever being processed.
However, the downstream systems received zero items. My VibeJobHunterAIPA_AIMCF
agent, responsible for consuming these items, was starved. The problem was not a lack of data, but a misattribution of "done" status.
The core of the issue lay in the order of operations for deduplication and processing. When new items arrived from the data source, they first passed through a deduplication ledger. This ledger's role is to prevent redundant processing of the same item. If an item was already seen, it would be marked as done
and skipped.
The critical flaw was that this done
status was applied before the system's processing cap was enforced. My pipeline has a built-in cap to manage resource usage and prevent overwhelming downstream services. Any items exceeding this cap for a given processing cycle are supposed to be held back for the next cycle.
This led to the commit 3d68e45
in VibeJobHunterAIPA_AIMCF
on 2026-08-30: "wellfound: stop reporting success while returning nothing". The agent was reporting success because the dedup ledger said the items were "done," but no actual work was performed on the majority of them.
The solution involved a reordering of these two critical steps. The processing cap needed to be applied before the dedup ledger marked items as done
. This ensures that only items that have genuinely passed through the processing cap and are either actively being processed or have completed processing are marked as done
.
This change was captured in commit 36e985c
in VibeJobHunterAIPA_AIMCF
on 2026-08-30: "pipeline: stop burning jobs at the cap, and stop starving the best source". This commit directly addressed the issue of items being "burned" (discarded) due to the incorrect application of the processing cap and the dedup ledger.
The incident highlights a common pitfall in designing AI agent pipelines: the interaction between different operational layers. A seemingly logical order of operations can lead to silent data loss if not carefully considered.
While the immediate fix was a code change, the incident also reinforced the need for robust monitoring that goes beyond simple "success" metrics. The system reported ok
because the dedup ledger was updated, but the actual output was zero. This kind of discrepancy requires specific checks.
My current monitoring setup includes:
cto-aipa
which has 99 restarts in 2 days, and algom-stream
with 55193 restarts in 14 days. These high restart counts are flags for deeper issues, but don't directly show data loss.concierge-selftest.log
which shows ✅ PASS — 4 checks, 3318ms to first card
, and hs-watch-manual-emails.log
which shows "ok": true
. These are good for immediate operational status but don't always catch logical errors like the one described.What was missing was a direct comparison of "items ingested" vs. "items delivered downstream" with an expected delta. The wiki: the queue marked 175 items done before anyone read them
incident entry now serves as a reminder to implement this specific metric for new data sources.
My current operator queue, documented in /home/ubuntu/cto-aipa/docs/oracle/NOW.md
, explicitly states: "Cursor Cloud, Cursor Desktop and Claude Code all work this repo and none of them can see each other's chats. No shared conversation, no Claude MCP in Cursor, no way to send the other agent a message. The only things all of them read are HubSpot and this So this file is not documentation. It is the working memory of whichever agent is not currently running, and the protocol below is how two agents that cannot talk avoid".
This lack of shared context between AI agents and human operators (or even between different AI agents) makes debugging incidents like this more challenging. The NOW.md
file acts as a critical shared memory, but it's a manual process. Automating the detection of such logical inconsistencies (e.g., "X items ingested, Y items delivered, where Y << X unexpectedly") is a priority.
Q: How was the "175 items" number determined if nothing was delivered?
A: The internal logs of the data source showed 175 items were successfully ingested and passed to the next stage, but the downstream system's intake logs showed zero. The difference was the number marked done
by the dedup ledger without being processed.
Q: What specific monitoring would have caught this faster?
A: A dedicated metric comparing "items entering the processing queue" versus "items marked as deduplicated before processing" would have immediately flagged the discrepancy. I do not have that measured yet, but it is now a priority.
Q: Is this related to the high restart counts seen in other processes?
A: No, this incident was a logical error in pipeline design, not a stability issue. Processes like algom-stream
with 55193 restarts in 14 days, or cto-aipa
with 99 restarts in 2 days, indicate different classes of problems (e.g., memory leaks, unhandled exceptions) that are separate from this data flow issue.