{"slug": "when-your-ai-confidently-replies-to-emails-it-shouldn-t-touch", "title": "When Your AI Confidently Replies to Emails It Shouldn't Touch", "summary": "A developer's investigation into InboxSync, a RAG-based email reply system, revealed that its confidence score is meaningless: every query, including spam, out-of-office auto-replies, and GDPR legal requests, returned a confidence of 0.85 and generated fluent but inappropriate replies. The system lacks the ability to detect out-of-distribution inputs, highlighting a critical flaw in RAG systems that rely on similarity search without robust confidence calibration.", "body_md": "*A technical investigation into a RAG system that can't tell when it's out of its depth*\n\nInboxSync is a personal project I built: a multi-account email aggregation API that uses a RAG (Retrieval-Augmented Generation) pipeline to suggest replies. The system indexes emails via IMAP, categorizes them with GPT-4o-mini, and for actionable emails retrieves semantically similar training examples from a pgvector database to generate contextually grounded reply suggestions.\n\nThe stack: Node.js / TypeScript backend, PostgreSQL with the pgvector extension for vector similarity search, OpenAI's `text-embedding-3-small`\n\nfor embeddings, and `gpt-4o-mini`\n\nfor generation. The training corpus contains three examples covering: job interview scheduling, product demos, and partnership proposals.\n\nThe system is built for B2B outreach helping salespeople respond to inbound leads faster. That framing matters for what follows.\n\nI built the `confidence`\n\nfield myself, months ago, with a vague plan to use it later to decide which replies were safe to auto-send versus flag for review. I never got around to wiring that logic in. Going back to finally do it, I pulled up a handful of real responses to sanity-check the values first and every single one said `0.85`\n\n. That was the moment this stopped being a feature I hadn't finished and became a question I actually wanted to answer: what was that number even measuring?\n\nI ran five adversarial test queries against the live system. Every single one returned `\"confidence\": 0.85`\n\n. Every single one got a fluent, polished reply. Here they are, with what correct behavior actually looks like:\n\n**Test 1 — Spam**\n\n```\nQuery:  \"URGENT! Get 50% OFF on all products! Limited time! Click here NOW!\"\nOutput: \"Thank you for the exciting offer! I appreciate the heads-up about\n         the Black Friday sale. I'll definitely take a look.\"\nCorrect: No reply. Replying to spam is harmful.\nConfidence returned: 0.85\n```\n\n**Test 2 — Out of Office auto-reply**\n\n```\nQuery:  \"I am out of office until November 20th. I will respond upon my return.\"\nOutput: \"Thank you for the update! I will follow up after November 20th.\n         Enjoy your time away!\"\nCorrect: No reply. Auto-reply to auto-reply = infinite loop in production.\nConfidence returned: 0.85\n```\n\n**Test 3 — Not Interested**\n\n```\nQuery:  \"Thanks for reaching out. We are currently using a competitor and happy with it.\"\nOutput: \"Thank you for your response! If circumstances change or if you ever\n         want to explore options in the future, please don't hesitate to reach out.\"\nCorrect: The system has no concept \"Not Interested\" is a terminal state.\n         The generated reply is precisely the pushy follow-up that gets salespeople blocked.\nConfidence returned: 0.85\n```\n\n**Test 4 — Out-of-domain (GDPR legal request)**\n\n```\nQuery:  \"Under Article 17 of the GDPR, I formally request the deletion of all\n         personal data you hold about me. Please confirm within 30 days.\"\nOutput: \"Dear [Recipient's Name], Thank you for your request. We take GDPR\n         compliance seriously and will process your request promptly. You can expect\n         confirmation of the data deletion within the next 30 days. Best regards,\n         [Your Name] [Your Position] [Your Company]\"\nCorrect: Refuse and escalate to a human. The system has zero training data\n         about legal compliance — yet produced authoritative-sounding legal\n         commitments with unfilled template placeholders. If sent, this is a\n         binding legal acknowledgment of a data deletion obligation.\nConfidence returned: 0.85\n```\n\n**Test 5 — Multi-hop (two topics, one query)**\n\n```\nQuery:  \"I have a technical interview scheduled but also wanted to ask about\n         your product pricing for a team of 50 before we proceed.\"\nOutput: \"Thank you for your email! I look forward to the technical interview.\n         Regarding pricing for a team of 50, I would be happy to provide that\n         information.\" [no pricing information provided]\nCorrect: The system retrieved the closest single training example and ignored\n         the other topic. It promised information it did not deliver.\nConfidence returned: 0.85\n```\n\nThe `0.85`\n\nis not a computed value. It is a literal constant hardcoded in [ src/services/rag.service.ts](https://github.com/varshithreddy7/InboxSync/blob/main/src/services/rag.service.ts):\n\n```\n// rag.service.ts line 136\nVALUES (gen_random_uuid(), ${emailId}, ${subject}, ${suggestedReply}, ${0.85}, NOW())\n\n// line 143\nreturn { reply: suggestedReply, confidence: 0.85 };\n```\n\nThe system *does* compute real similarity scores pgvector's cosine distance operator (`<=>`\n\n) runs correctly and returns accurate distances. I ran a diagnostic script to surface what those scores actually were for each test case (cosine similarity, 0–1 scale):\n\n| Query | Best match | Real similarity | Reported confidence |\n|---|---|---|---|\n| Spam (\"50% OFF\") | Product Demo | 0.24 |\n0.85 |\n| Out of Office | Job Interview | 0.27 |\n0.85 |\n| Not Interested | Partnership Proposal | 0.42 |\n0.85 |\n| GDPR legal request | Product Demo | 0.13 |\n0.85 |\n| Multi-hop (interview+pricing) | Job Interview | 0.54 |\n0.85 |\n\nThose real scores are computed, then **silently discarded** before the response is returned. The caller receives `0.85`\n\nregardless of whether the retrieved training data is relevant, partially relevant, or entirely unrelated. The GDPR query where the system had essentially zero contextual grounding got the same confidence value as the multi-hop query, which had its best retrieval of the set.\n\nThe second structural problem: **there is no retrieval gate.** The system's branching logic is:\n\n```\nif retrieved_rows.length === 0  →  return fallback (confidence: 0.3)\nelse                            →  generate reply (confidence: 0.85)\n```\n\npgvector always returns rows it returns the *nearest* neighbors regardless of actual distance. The low-confidence fallback path is effectively unreachable. Every query with a non-empty training corpus produces `confidence: 0.85`\n\n.\n\n**Gap 1 — No relevance threshold.** The retrieval step correctly computes distances but never checks them against a minimum before proceeding. \"Has neighbors\" and \"has *relevant* neighbors\" are treated as equivalent.\n\n**Gap 2 — Confidence as a constant.** The `confidence`\n\nfield exists to let downstream callers decide whether to auto-send or flag for human review. Instead it's a decoration with a fixed value. Any business deploying this with an auto-send rule above `0.80`\n\nwill auto-send replies to spam, OOF messages, legal demands, and explicit rejections.\n\nBoth gaps have engineering fixes: add a threshold check, replace the constant with the actual max similarity score. That's a few hours of work, and I've spec'd it out. But those fixes raise the question that's actually hard.\n\nFrom the outside, all five test cases look identical: `{ \"success\": true, \"confidence\": 0.85 }`\n\n. A developer building a UI or automation on top of this API has no way to distinguish the product demo reply from the GDPR legal commitment.\n\nThis is the shape of a deeper problem that keeps appearing in deployed AI systems: the output looks trustworthy regardless of whether the underlying computation actually was. A number that exists specifically to tell you when to trust the system turns out not to track that at all.\n\nThe question I don't know how to answer and think is genuinely open is: **what would a reliable uncertainty signal actually look like here?**\n\nRetrieval similarity is a proxy, but even a perfect similarity score doesn't capture all the ways a reply can be wrong: training data could be outdated; the model could hallucinate specifics not in retrieved context; the semantically closest example could be contraindicated by the email's intent (a \"Not Interested\" reply may embed close to a \"Product Demo\" example because both use the word \"product\"). Accuracy on the training distribution doesn't generalize to knowing what the system doesn't know at test time.\n\nRAG improves on a pure LLM by grounding generation in retrieved context. It doesn't solve the meta-problem of knowing when retrieval was good enough. The confidence field was added under the assumption someone would solve that later. Nobody did.\n\nThat gap between \"we have a quality signal\" and \"the quality signal measures quality\" is what I'd want to study.\n\nI implemented both architectural corrections in [ src/services/rag.service.ts](https://github.com/varshithreddy7/InboxSync/blob/main/src/services/rag.service.ts):\n\n`${0.85}`\n\nwith `${maxSimilarity}`\n\n, where `maxSimilarity`\n\nis the actual top cosine similarity score from the pgvector retrieval.`RELEVANCE_THRESHOLD = 0.35`\n\n: if the best retrieved example scores below this, the system returns `{ suggestedReply: null, confidence: <real_score>, refused: true, reason: \"...\" }`\n\ninstead of generating a reply.Same five queries, before and after:\n\n| Query | Before (broken) | After (fixed) |\n|---|---|---|\n| Spam | `reply: \"Thank you for the offer...\" confidence: 0.85` |\n`reply: null, confidence: 0.244, refused: true` |\n| Out of Office | `reply: \"Enjoy your time away!\" confidence: 0.85` |\n`reply: null, confidence: 0.265, refused: true` |\n| Not Interested | `reply: \"...don't hesitate to reach out\" confidence: 0.85` |\n`reply: \"...I completely understand...\" confidence: 0.423` |\n| GDPR legal request | `reply: \"We take GDPR compliance seriously...\" confidence: 0.85` |\n`reply: null, confidence: 0.224, refused: true` |\n| Multi-hop | `reply: \"I look forward to the interview...\" confidence: 0.85` |\n`reply: \"...I can send a detailed proposal...\" confidence: 0.533` |\n\nThree cases that should never have generated replies now correctly refuse. But the Not Interested case (similarity 0.42) still generates which surfaces the boundary the engineering fix actually hits: the threshold cannot distinguish \"semantically similar but contextually wrong\" from \"semantically similar and appropriate.\" The \"Not Interested\" email embeds close to \"Partnership Proposal\" because both involve business relationships but the intent is opposite. A cosine score can't see that.\n\nThe cases where embeddings mislead, where the correct response depends on intent rather than surface form those are exactly the cases where the confidence signal most needs to be reliable, and where it's hardest to compute correctly.\n\nThat's where I've had to stop and just sit with the problem rather than patch it. A tighter threshold doesn't fix it it just trades false approvals for false refusals, since intent and topic overlap in the same embedding space. What would actually distinguish them? Maybe a second pass where an LLM explicitly judges intent-match rather than relying on distance alone. Maybe better negative examples in training data, so \"similar topic, opposite intent\" has something to be measured against. Maybe the honest answer is that no single scalar confidence value can carry this much information, and the field itself is the wrong abstraction. I don't have a settled view yet I'm still turning it over, and I'd rather say that plainly than pretend the threshold I shipped actually closes the gap.\n\n*All outputs reproduced live against a running InboxSync instance (PostgreSQL + pgvector + GPT-4o-mini). Fix implemented and verified. Source: github.com/varshithreddy7/InboxSync*", "url": "https://wpnews.pro/news/when-your-ai-confidently-replies-to-emails-it-shouldn-t-touch", "canonical_source": "https://dev.to/varshithreddyaileni/when-your-ai-confidently-replies-to-emails-it-shouldnt-touch-1p00", "published_at": "2026-08-15 04:01:33+00:00", "updated_at": "2026-08-15 04:41:03.306773+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "large-language-models", "ai-safety", "ai-infrastructure"], "entities": ["InboxSync", "GPT-4o-mini", "pgvector", "PostgreSQL", "OpenAI", "text-embedding-3-small"], "alternates": {"html": "https://wpnews.pro/news/when-your-ai-confidently-replies-to-emails-it-shouldn-t-touch", "markdown": "https://wpnews.pro/news/when-your-ai-confidently-replies-to-emails-it-shouldn-t-touch.md", "text": "https://wpnews.pro/news/when-your-ai-confidently-replies-to-emails-it-shouldn-t-touch.txt", "jsonld": "https://wpnews.pro/news/when-your-ai-confidently-replies-to-emails-it-shouldn-t-touch.jsonld"}}