I Stopped Tweaking Prompts. Here's How I Cut LLM Hallucinations to 6%.

LLMs are great at writing code, but ask them to generate strictly formatted Markdown? That's a different story. We spent weeks optimizing our prompts to fix technical hallucinations and structural chaos, but hit a wall. Eventually, we stopped trying to solve it with words alone and built a pipeline using a Judge-Write loop with experience replay.

The result was immediate: content generation accuracy jumped from 77% to 94%.

While building an automated technical documentation system, our Writer Agent kept producing content with SQL syntax errors and logic gaps. It couldn't guarantee strict Markdown compliance, causing frequent crashes in the rendering layer.

The core challenge was maintaining strict data structure rigor without sacrificing speed (latency < 3s) or falling into infinite retry loops. If left unchecked, our online error rate would stay above 20%, triggering over 40 weekly alerts and destroying user trust.

1. Prompt Engineering Failed

Simply increasing prompt complexity (like Chain of Thought) didn't fix structural errors. LLMs still struggle with complex Markdown tables. Asking one model to be purely creative yet strictly rigorous is a losing battle.

2. No Immediate Feedback

The Writer Agent was a one-shot process. If it generated an error, it outputted it directly. There was no mechanism for self-correction or intermediate quality control—like taking an exam without a teacher to grade it.

3. Experience Wasn't Reusable

Every generation was independent. The system couldn't remember which patterns (like specific SQL syntax) were correct, leading to repeated errors. The agent kept falling into the same holes.

We decoupled generation from evaluation by introducing an independent Judge Agent for syntax validation and logic review. If the Writer can't be trusted, we gave it a strict quality control officer.

The Judge-Write Loop:

response = writer_agent.generate(prompt)
return response

max_retries = 3
for i in range(max_retries):
    draft = writer_agent.generate(prompt)
    feedback = judge_agent.evaluate(draft)
    if feedback.is_valid:
        return draft
    else:
        prompt = refine_prompt_with_feedback(prompt, feedback)
raise MaxRetriesExceededError()

Pattern-Based Experience Storage:

Instead of guessing blindly every time, the Writer now references "top student" homework. We extract high-quality code blocks approved by the Judge and store them as patterns in a Vector DB.

messages = [{'role': 'system', 'content': 'You are a writer...'}]

relevant_patterns = memory.search(query=current_topic)
system_prompt = f"You are a writer. Reference these successful patterns: {relevant_patterns}"
messages = [{'role': 'system', 'content': system_prompt}]

Next time your LLM output is full of hallucinations, stop tweaking the prompt. Try giving it a strict Judge instead.