{"slug": "your-llm-keeps-making-the-same-extraction-mistake-here-s-how-to-make-it-learn", "title": "Your LLM Keeps Making the Same Extraction Mistake. Here's How to Make It Learn.", "summary": "An AWS engineer open-sourced a sample pipeline, sample-prompt-correction-memory, that captures human QA corrections to LLM document extraction and reuses them so recurring field errors are not repeated. The system routes fields through three tiers — deterministic rules synthesized from repeated corrections, Amazon Bedrock Claude Haiku 4.5 with confidence scoring, and Claude Sonnet 4.5 few-shot re-extraction using retrieved past corrections — shifting traffic toward the zero-cost rule tier over time without retraining or redeployment.", "body_md": "[>](https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fqhubyrft3pyfvanjeee2.png) **TL;DR** — I open-sourced an AWS sample that makes LLM document extraction *learn from human corrections*. Each QA fix is captured once and reused, so the pipeline gets **more accurate and cheaper over time** — with no retraining and no redeployment. Repo: [github.com/aws-samples/sample-prompt-correction-memory](https://github.com/aws-samples/sample-prompt-correction-memory). You can run the whole self-healing loop locally in 30 seconds, no AWS account needed.\n\nIf you've built an LLM extraction pipeline, you know this pain.\n\nYour QA analyst opens today's invoice. The model extracted `payment_terms` as **\"quarterly.\"** That's wrong — \"quarterly\" is the *billing frequency*; the actual payment term is **Net 30**. The analyst corrects it. Done.\n\nTomorrow, an identical invoice from the same vendor arrives. The model extracts... **\"quarterly\"** again. Same mistake. The analyst corrects it again.\n\nThe correction *evaporated.* It fixed one document and taught the system nothing.\n\nMultiply that across thousands of documents and dozens of recurring error patterns, and you get the two options most teams settle for:\n\nThere's a third way that needs neither.\n\nWhat if every human correction became a permanent, reusable asset?\n\nThat's the pattern behind [`sample-prompt-correction-memory`](https://github.com/aws-samples/sample-prompt-correction-memory). Instead of throwing corrections away, it stores them and feeds them back into future extractions — so the same mistake is never made twice.\n\nIt works as **three tiers**, and the magic is that traffic gradually shifts from the expensive tier to the free one:\n\nWhen the same correction pattern recurs enough times, the system synthesizes a **deterministic rule** (a regex, a lookup, a normalization) that handles that field with zero LLM cost and sub-millisecond latency. \"Net 30\" → `30` becomes a rule; it never needs a model again.\n\nFields with no matching rule go to **Amazon Bedrock (Claude Haiku 4.5)**. Every field comes back with a **confidence score**. If confidence clears the threshold, accept it and move on.\n\nIf confidence is *below* threshold, the system retrieves the most relevant past corrections, formats them as **few-shot examples**, and re-extracts with a stronger model (** Claude Sonnet 4.5**). The model learns from the specific mistakes it made before — in-context, no training.\n\nEvery QA correction flows into a **correction log** (DynamoDB) that powers both the self-healing retrieval *and* the rule graduation. So the more the system is used, the smarter and cheaper it gets.\n\nThe clearest fit is **high-repetition, back-office document processing** — accounts payable, contracts, compliance filings, supply-chain docs — where the same field errors recur and analyst time is the real expense.\n\nFully serverless, deployed with AWS SAM:\n\n```\nDocument ──► S3 ──► EventBridge ──► SQS ──► Lambda (extract)\n                                              │\n                          ┌───────────────────┼───────────────────┐\n                          ▼                    ▼                   ▼\n                   Tier 1: Rules       Tier 2: Bedrock      Tier 3: Self-heal\n                   (deterministic)     (Claude Haiku 4.5)   (Claude Sonnet 4.5)\n                          ▲                                        │\n                          │                                        ▼\n                          └──────── Correction Log (DynamoDB) ◄────┘\n                                    (grows from QA feedback)\n```\n\nQA corrections are uploaded as JSON to an S3 `corrections/` prefix; a second Lambda validates and ingests them into the correction log. Everything is encrypted with a customer-managed KMS key, and IAM is scoped to the specific Bedrock model ARNs the sample uses.\n\nThe quickstart runs the entire self-healing loop with a mocked Bedrock client, so you can watch the mechanism without deploying anything:\n\n```\ngit clone https://github.com/aws-samples/sample-prompt-correction-memory\ncd sample-prompt-correction-memory\npip install -e \".[dev]\"\npython examples/quickstart.py\n```\n\nYou'll see output like this:\n\n```\nStep 1: Initial extraction (no correction memory)\n  Field:      effective_date\n  Value:      March 2024        Confidence: 0.55   ❌ NO\n\nStep 2: Self-healing triggers (0.55 < threshold 0.70)\n  Retrieving corrections for 'effective_date'...\n  Found: \"March 2024\" → \"2024-03-01\"\n\nStep 3: Re-extraction with correction memory\n  Value:       2024-03-01       Confidence: 0.95   ✓ YES\n  Self-Healed: True\n```\n\nLow-confidence extraction → retrieve past correction → re-extract → correct answer. No retraining. No redeployment.\n\nWith the AWS CLI, SAM CLI, and Bedrock model access for Claude Haiku 4.5 + Sonnet 4.5:\n\n```\nmake deploy    # S3, DynamoDB, Lambda, EventBridge, SQS, KMS\nmake seed      # load sample corrections into the correction log\nmake trigger   # upload a sample document → triggers real extraction\nmake verify    # read back results (fields, confidence, self_healed)\nmake destroy   # empty buckets + delete the stack\n```\n\n`make verify` prints each extracted field with its confidence and whether self-healing kicked in — your proof it works end-to-end.\n\nThis sample provides **semantic** correction memory (what was extracted wrong, and why). Its companion, [`sample-textract-field-memory`](https://github.com/aws-samples/sample-textract-field-memory), provides **spatial** memory (where fields appear on a document layout). Together they form a dual memory for document pipelines: use the cheap spatial lookup when you're confident where a field is, and fall back to self-healing extraction when you're not.\n\nBecause it's more useful when it's used well:\n\nThe repo is open source under `aws-samples`, MIT-0 licensed, with a benchmark suite, an interactive dashboard, and an offline test harness.\n\n👉 [github.com/aws-samples/sample-prompt-correction-memory](https://github.com/aws-samples/sample-prompt-correction-memory)\n\nWhat repetitive extraction error is your team still fixing by hand? I'd love to hear which patterns you'd want a system like this to learn.\n\n*This is an open-source AWS sample intended for demonstration and non-production use.*", "url": "https://wpnews.pro/news/your-llm-keeps-making-the-same-extraction-mistake-here-s-how-to-make-it-learn", "canonical_source": "https://dev.to/avneet_bansal_a65b3f31fc4/your-llm-keeps-making-the-same-extraction-mistake-heres-how-to-make-it-learn-5bjb", "published_at": "2026-10-03 01:32:58+00:00", "updated_at": "2026-10-03 01:37:31.236769+00:00", "lang": "en", "topics": ["large-language-models", "ai-agents", "ai-infrastructure", "mlops", "developer-tools"], "entities": ["AWS", "Amazon Bedrock", "Claude Haiku 4.5", "Claude Sonnet 4.5", "Amazon DynamoDB", "AWS Lambda", "AWS SAM", "sample-prompt-correction-memory"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/your-llm-keeps-making-the-same-extraction-mistake-here-s-how-to-make-it-learn", "markdown": "https://wpnews.pro/news/your-llm-keeps-making-the-same-extraction-mistake-here-s-how-to-make-it-learn.md", "text": "https://wpnews.pro/news/your-llm-keeps-making-the-same-extraction-mistake-here-s-how-to-make-it-learn.txt", "jsonld": "https://wpnews.pro/news/your-llm-keeps-making-the-same-extraction-mistake-here-s-how-to-make-it-learn.jsonld"}}