# Your LLM Keeps Making the Same Extraction Mistake. Here's How to Make It Learn. > Source: > Published: 2026-10-03 01:32:58+00:00 [>](https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fqhubyrft3pyfvanjeee2.png) **TL;DR** — I open-sourced an AWS sample that makes LLM document extraction *learn from human corrections*. Each QA fix is captured once and reused, so the pipeline gets **more accurate and cheaper over time** — with no retraining and no redeployment. Repo: [github.com/aws-samples/sample-prompt-correction-memory](https://github.com/aws-samples/sample-prompt-correction-memory). You can run the whole self-healing loop locally in 30 seconds, no AWS account needed. If you've built an LLM extraction pipeline, you know this pain. Your QA analyst opens today's invoice. The model extracted `payment_terms` as **"quarterly."** That's wrong — "quarterly" is the *billing frequency*; the actual payment term is **Net 30**. The analyst corrects it. Done. Tomorrow, an identical invoice from the same vendor arrives. The model extracts... **"quarterly"** again. Same mistake. The analyst corrects it again. The correction *evaporated.* It fixed one document and taught the system nothing. Multiply that across thousands of documents and dozens of recurring error patterns, and you get the two options most teams settle for: There's a third way that needs neither. What if every human correction became a permanent, reusable asset? That's the pattern behind [`sample-prompt-correction-memory`](https://github.com/aws-samples/sample-prompt-correction-memory). Instead of throwing corrections away, it stores them and feeds them back into future extractions — so the same mistake is never made twice. It works as **three tiers**, and the magic is that traffic gradually shifts from the expensive tier to the free one: When the same correction pattern recurs enough times, the system synthesizes a **deterministic rule** (a regex, a lookup, a normalization) that handles that field with zero LLM cost and sub-millisecond latency. "Net 30" → `30` becomes a rule; it never needs a model again. Fields with no matching rule go to **Amazon Bedrock (Claude Haiku 4.5)**. Every field comes back with a **confidence score**. If confidence clears the threshold, accept it and move on. If confidence is *below* threshold, the system retrieves the most relevant past corrections, formats them as **few-shot examples**, and re-extracts with a stronger model (** Claude Sonnet 4.5**). The model learns from the specific mistakes it made before — in-context, no training. Every QA correction flows into a **correction log** (DynamoDB) that powers both the self-healing retrieval *and* the rule graduation. So the more the system is used, the smarter and cheaper it gets. The clearest fit is **high-repetition, back-office document processing** — accounts payable, contracts, compliance filings, supply-chain docs — where the same field errors recur and analyst time is the real expense. Fully serverless, deployed with AWS SAM: ``` Document ──► S3 ──► EventBridge ──► SQS ──► Lambda (extract) │ ┌───────────────────┼───────────────────┐ ▼ ▼ ▼ Tier 1: Rules Tier 2: Bedrock Tier 3: Self-heal (deterministic) (Claude Haiku 4.5) (Claude Sonnet 4.5) ▲ │ │ ▼ └──────── Correction Log (DynamoDB) ◄────┘ (grows from QA feedback) ``` QA corrections are uploaded as JSON to an S3 `corrections/` prefix; a second Lambda validates and ingests them into the correction log. Everything is encrypted with a customer-managed KMS key, and IAM is scoped to the specific Bedrock model ARNs the sample uses. The quickstart runs the entire self-healing loop with a mocked Bedrock client, so you can watch the mechanism without deploying anything: ``` git clone https://github.com/aws-samples/sample-prompt-correction-memory cd sample-prompt-correction-memory pip install -e ".[dev]" python examples/quickstart.py ``` You'll see output like this: ``` Step 1: Initial extraction (no correction memory) Field: effective_date Value: March 2024 Confidence: 0.55 ❌ NO Step 2: Self-healing triggers (0.55 < threshold 0.70) Retrieving corrections for 'effective_date'... Found: "March 2024" → "2024-03-01" Step 3: Re-extraction with correction memory Value: 2024-03-01 Confidence: 0.95 ✓ YES Self-Healed: True ``` Low-confidence extraction → retrieve past correction → re-extract → correct answer. No retraining. No redeployment. With the AWS CLI, SAM CLI, and Bedrock model access for Claude Haiku 4.5 + Sonnet 4.5: ``` make deploy # S3, DynamoDB, Lambda, EventBridge, SQS, KMS make seed # load sample corrections into the correction log make trigger # upload a sample document → triggers real extraction make verify # read back results (fields, confidence, self_healed) make destroy # empty buckets + delete the stack ``` `make verify` prints each extracted field with its confidence and whether self-healing kicked in — your proof it works end-to-end. This sample provides **semantic** correction memory (what was extracted wrong, and why). Its companion, [`sample-textract-field-memory`](https://github.com/aws-samples/sample-textract-field-memory), provides **spatial** memory (where fields appear on a document layout). Together they form a dual memory for document pipelines: use the cheap spatial lookup when you're confident where a field is, and fall back to self-healing extraction when you're not. Because it's more useful when it's used well: The repo is open source under `aws-samples`, MIT-0 licensed, with a benchmark suite, an interactive dashboard, and an offline test harness. 👉 [github.com/aws-samples/sample-prompt-correction-memory](https://github.com/aws-samples/sample-prompt-correction-memory) What repetitive extraction error is your team still fixing by hand? I'd love to hear which patterns you'd want a system like this to learn. *This is an open-source AWS sample intended for demonstration and non-production use.*