LLMs are basically confident liars until you give them a leash, and that leash is Retrieval-Augmented Generation (
I've managed to stabilize it by implementing a re-ranking step—basically a second AI that filters the search results before they hit the prompt—but it's just adding more latency to an already slow workflow.
RAG). I tried building a custom knowledge base to stop my bot from making up fake documentation, but I spent half my weekend fighting with "context window" errors and results that looked like they were written by a drunk toddler.The logic is simple: don't trust the model's training data for specific facts. Instead, you pull the relevant snippet from a vector database and shove it into the prompt. But the "simple" part is where it all falls apart.
My current disaster looks like this:
Error: ContextWindowExceeded - Input tokens (128k) exceed model limit.
Diagnosis: My chunking strategy is a joke. I'm feeding the LLM entire PDF pages instead of relevant paragraphs.
If you're attempting a RAG deployment from scratch, here is where the actual pain points are:
Chunking Strategy: If your chunks are too small, the AI loses the plot. If they're too big, you hit the token limit or the model gets "lost in the middle" and ignores the actual answer.Embedding Quality: Using a cheap embedding model is like trying to organize a library by the color of the book covers. You'll find "something blue" but not the actual information you need.Retrieval Noise: The top-k results often include one useful paragraph and four completely irrelevant ones that confuse the hell out of the LLM.
I've managed to stabilize it by implementing a re-ranking step—basically a second AI that filters the search results before they hit the prompt—but it's just adding more latency to an already slow workflow.
Still, it beats the alternative of the bot telling my users that our API supports features we haven't even coded yet.
Next My AI Workflow Fail: Getting Catfished by a Bot →
All Replies （3） #
G
Are you using a vector DB or just basic keyword search for the retrieval part?
0
Q
Had this issue until I tuned my chunk size. Overlapping chunks usually fix those gaps.
0
J
Try adding a "don't know" clause to the system prompt. Stops the guessing games immediately.
0