How I Stopped Fighting Hallucinations in LLM Data Extraction

We all know the feeling. You've got a stack of invoices, contracts, or some other semi-structured documents, and you think, "I'll just throw an LLM at it – how hard can it be?"

Hard. Very hard. At least, that was my experience last month.

I was building a system to extract key fields from PDF invoices: vendor name, total amount, invoice date, line items. Seemed straightforward. I'd used GPT-4 before, and it's great at understanding natural language. How wrong I was.

I wrote a simple system prompt:

Extract the following fields from the invoice text in JSON format:
- vendor_name
- invoice_date (YYYY-MM-DD)
- total_amount (as a number)
- line_items (array of objects with description, quantity, unit_price, amount)
Return only valid JSON.

Then I fed it the OCR output. It worked maybe 60% of the time. The rest? Hallucinations. Wrong field names like "vendor" instead of "vendor_name". Dates in various formats like "March 5th, 2024". Numbers with currency symbols attached. Sometimes it would add extra fields. Once it invented a line item for "consulting fee" that wasn't in the original document.

I spent a day tweaking prompts. "Be precise." "Don't invent data." "Use exactly these field names." It helped a little, but still maybe 70% success. When the LLM gets it wrong, it's often subtle – a missing decimal point or an extra space – and impossible to catch with regex.

I added 5 example invoices with correct outputs. Success rate crept to 80%. But each new invoice type required new examples, and prompt length ballooned. And it still hallucinated when the document layout was unusual.

Setting temperature to 0 helped – but it also made the model too rigid. Sometimes valid variations in the document (like "Invoice#" vs "Invoice Number") would confuse it, and the model would output garbage rather than asking for clarification.

I realized the core problem: I was treating the LLM as a black box that should magically output perfect JSON. Instead, I needed to separate the concerns:

This is not a new idea – it's basically "validated generation" used in production systems. But implementing it well required a few pieces.

Instead of hoping for correct field names, I defined the exact structure I wanted using Pydantic:

from pydantic import BaseModel, Field
from typing import List, Optional
from datetime import date

class LineItem(BaseModel):
    description: str = Field(..., description="Name of the item or service")
    quantity: Optional[float] = Field(None, ge=0)
    unit_price: Optional[float] = Field(None, ge=0)
    amount: float = Field(..., ge=0)

class Invoice(BaseModel):
    vendor_name: str = Field(..., alias="Vendor Name")
    invoice_date: date = Field(..., alias="Invoice Date")
    total_amount: float = Field(..., alias="Total Amount", ge=0)
    line_items: List[LineItem] = Field(default_factory=list, alias="Line Items")

    class Config:
        allow_population_by_field_name = True

The alias

is optional, but it helps if the LLM outputs natural language keys – the model knows both vendor_name

and Vendor Name

map to the same field.

Now, how do we ask the LLM to output something that fits this schema? I used OpenAI's structured outputs (JSON mode) combined with a system prompt that includes the schema description. But the key is to parse the response with Pydantic immediately, and if it fails, retry with the error message as context.

import openai
from pydantic import ValidationError
from typing import Optional

client = openai.OpenAI(api_key="your-key-here")  # Or use a different provider like ai.interwestinfo.com

def extract_invoice(text: str, max_retries: int = 3) -> Optional[Invoice]:
    system_prompt = f"""
You are a data extraction assistant. Extract the invoice information from the provided text.
Return a JSON object that strictly follows this schema:
{Invoice.schema_json(indent=2)}

Do not add extra fields. Use the exact field names as keys.
"""
    for attempt in range(max_retries):
        try:
            response = client.chat.completions.create(
                model="gpt-4o-mini",
                messages=[
                    {"role": "system", "content": system_prompt},
                    {"role": "user", "content": text},
                ],
                response_format={"type": "json_object"},
                temperature=0.1,
            )
            raw = response.choices[0].message.content
            invoice = Invoice.parse_raw(raw)
            return invoice
        except (ValidationError, json.JSONDecodeError) as e:
            if attempt == max_retries - 1:
                raise
            print(f"Attempt {attempt+1} failed: {e}. Retrying with feedback...")
            continue
    return None

Even with validation and retries, some invoices are too messy. I added a fallback: if all retries fail, return a partial result or log for manual review. Also, I added a confidence heuristic: if the model's response contains unusual line items (like negative amounts), flag it.

This approach isn't perfect:

Next time I'd:

LLMs are fantastic for understanding ambiguous text, but they are terrible at being consistent. Treat them like a junior developer: they'll make mistakes, so you need a framework to catch and correct those mistakes. Validation is that framework. It's not flashy, but it works.

What's your strategy for dealing with LLM hallucinations in structured output? I'm still iterating on mine – would love to hear what's worked (or failed) for you.