{"slug": "spatial-memory-vs-correction-memory-i-benchmarked-both-against-a-plain-llm", "title": "Spatial Memory vs Correction Memory: I Benchmarked Both Against a Plain LLM", "summary": "A developer built a benchmark comparing two open-source document-extraction memory libraries against a plain LLM baseline on 120 synthetic invoices and forms using Claude Haiku 4.5 on Amazon Bedrock. Spatial memory cut LLM calls from 600 to 30 and p50 latency from 3.3 seconds to a memory read, while correction memory left call counts unchanged but reduced the repeated-error rate from 0.185 to 0.012; both raised field accuracy from 0.815 to about 0.99. The developer concluded the two mechanisms are complementary rather than competing, with spatial memory acting as a cost lever and correction memory as a quality lever.", "body_md": "Earlier this year I wrote about two small open-source libraries for document extraction. The first, [Give Your Document OCR Pipeline a Memory](https://dev.to/avneet_bansal_a65b3f31fc4/give-your-document-ocr-pipeline-a-memory-59l1), introduced [`textract-field-memory`](https://github.com/aws-samples/sample-textract-field-memory), which remembers where fields sit on a recurring layout. The second, [Your LLM Keeps Making the Same Extraction Mistake](https://dev.to/avneet_bansal_a65b3f31fc4/your-llm-keeps-making-the-same-extraction-mistake-heres-how-to-make-it-learn-5bjb), introduced [`sample-prompt-correction-memory`](https://github.com/aws-samples/sample-prompt-correction-memory), which turns a human correction into something the pipeline keeps.\n\nEach post made its own case. Neither one answered the question readers kept asking me: if you can only add one, which one, and what does each actually buy you against a plain LLM?\n\nSo I built a small benchmark that runs both mechanisms against the same document stream, next to a no-memory baseline, and measured them on the same five metrics. The short answer is that they are not competitors. One is a cost lever and the other is a quality lever.\n\n**TL;DR.** On 120 synthetic invoices and forms with Claude Haiku 4.5 on Amazon Bedrock: spatial memory cut LLM calls from **600 to 30** (about 95%) and p50 latency from **3.3 s to a memory read**. Correction memory left the call count alone but cut the repeated-error rate from **0.185 to 0.012**. Both raised field accuracy from **0.815 to about 0.99**.\n\nThree arms process the identical seeded stream, in the same order.\n\n**Baseline.** Plain LLM extraction on every field of every document. No memory. This is the reference for cost and for how often a mistake comes back.\n\n**+Spatial memory.** Models `textract-field-memory`. For the first three documents of a type, it calls the model and records where each field sits. Once a field's position is stable within tolerance, it reads the value from the learned region and skips the model call.\n\n**+Correction memory.** Models `sample-prompt-correction-memory`. It always calls the model. The first time a field comes back wrong, a simulated reviewer corrects it, and the arm stores the correction as a generalizing transform (for example, \"undo the day/month swap\"), not as a frozen value. Later documents of that type get the transform applied.\n\nThe dataset is 120 synthetic documents, 60 invoices and 60 forms, five fields each (invoice number, date, vendor, total, tax). Two fields per type carry a deterministic, systematic error, the kind a stateless pipeline repeats forever: a date with day and month swapped, and an amount with a stray leading digit (`12.34` read as `112.34`).\n\nThe five metrics were written down in a test plan before any run: field accuracy, LLM calls, token cost, latency, and repeated-error rate. That last one is the share of later occurrences of a field that still show the same error after it was first corrected.\n\nReal-model run: Amazon Bedrock, `us.anthropic.claude-haiku-4-5-20251001-v1:0`, `us-west-2`, seed 42.\n\n| Arm | Field accuracy | LLM calls | Calls/doc | Token cost (modeled) | p50 latency | p95 latency | Repeated-error rate | \n|---|---|---|---|---|---|---|---|\n| Baseline | 0.815 | 600 | 5.00 | $0.0556 | 3,331 ms | 5,592 ms | 0.185 | \n| +Spatial memory | 0.990 | 30 | 0.25 | $0.0028 | ~0 ms | 156 ms | 0.007 | \n| +Correction memory | 0.988 | 600 | 5.00 | $0.0556 | 3,218 ms | 4,243 ms | 0.012 | \n\nBy document type, the baseline scored 0.80 on invoices and 0.83 on forms. Spatial memory reached 0.99 on both. Correction memory reached 1.00 on invoices and 0.98 on forms.\n\nThe cost column is small because the run is small. The ratio is what matters: spatial memory ran at about 5% of the baseline's modeled token cost.\n\n**Spatial memory is the cost story.** The 30 calls it made are exactly the warmup: 3 documents, times 5 fields, times 2 document types. After that, every field on every document was a lookup. That is why p50 latency collapses to a memory read and p95 drops to 156 ms (the tail is the warmup documents). On a stable, high-volume layout, the model is doing work the layout already answered.\n\n**Correction memory is the quality story.** It does not save a single call, because it still runs the model on every field. What it changes is what happens after the first mistake. In the baseline, the date swap and the stray digit come back on every later document, which is where the 0.185 repeated-error rate comes from. With correction memory, each systematic error is paid for once, by one review, and then it stops. The rate drops to 0.012.\n\n**They fix different failure modes.** Spatial memory removes redundant calls on fields the layout pins down. Correction memory removes errors the model keeps making on calls that still happen. A pipeline that has both problems needs both layers, and a combined arm is the obvious next configuration to measure.\n\nThe OCR memory post ended with a direct ask: the stress tests there were operational numbers, with no labeled benchmark and no comparison against a baseline. This benchmark is a first step on that. It has ground-truth values, a fixed baseline, and pre-registered metrics.\n\nThe correction memory post described a three-tier design: graduated rules at zero cost, a confidence-gated LLM tier, and few-shot self-healing for low-confidence fields. This benchmark isolates one property of that design, that a correction applied once stops the same mistake from recurring, and measures it on a stream.\n\nPut together, the two posts describe two separate places a pipeline wastes effort. This post puts numbers on both, side by side.\n\nThese caveats matter more than the headline, so read them before quoting a number.\n\n**The arms model the libraries, they do not import them.** Each arm is a self-contained reimplementation of the library's core mechanism, so the benchmark runs with no install and no network. The published libraries do more (IoU and center-distance scoring, template identification, drift detection, rule graduation, few-shot retry). Swapping the real libraries into the arms is the next step.\n\n**The spatial arm's accuracy is close to an upper bound.** When spatial confidence is high, the arm returns the value at the learned position, and on synthetic data that value is the ground truth. In a real pipeline, that read still depends on the OCR being right in that box. The defensible claim from this arm is the 95% call reduction. The accuracy gain says that on a stable layout, a positional read avoids the model's occasional misread, assuming the OCR is clean.\n\n**The correction arm knows two error patterns.** Its transform learner recognizes the day/month swap and the stray leading digit, because those are the errors the generator injects. A real correction log is messier, and the [first post](https://dev.to/avneet_bansal_a65b3f31fc4/your-llm-keeps-making-the-same-extraction-mistake-heres-how-to-make-it-learn-5bjb) covers how the library handles that with few-shot retrieval instead of hand-written transforms.\n\n**The errors are synthetic.** In Bedrock mode, the model reads a pre-noised OCR span, so the baseline's errors come from the injected noise model, not from Haiku misreading a clean document. The accuracy and repeated-error deltas are properties of that noise model on this set. Treat them as directional.\n\n**Cost is modeled.** Measured input and output tokens from the Bedrock usage block, times the published Haiku 4.5 rate ($1.00 per 1M input tokens, $5.00 per 1M output tokens). It is not a bill.\n\n**It is small.** 120 documents, two types, one seed for the real-model run.\n\nThe harness is small enough that the core of each arm fits on screen. This is the decision the spatial arm makes for every field:\n\n``` python\ndef confident(memory, key, bbox, warmup=3, tol=0.02):\n    \"\"\"True once a (doc_type, field) has a stable learned position.\"\"\"\n    if key not in memory:\n        return False\n    learned_bbox, seen = memory[key]\n    if seen < warmup:\n        return False\n    return all(abs(a - b) <= tol for a, b in zip(learned_bbox, bbox))\n\n# per field:\nif confident(memory, (doc_type, field_name), field_bbox):\n    value = read_region(field_bbox)        # no LLM call\nelse:\n    value = call_llm(field)                # warmup or position moved\nobserve(memory, (doc_type, field_name), field_bbox)\n```\n\nAnd this is the correction arm. The important part is that the stored correction is a transform that inverts the systematic error, not the corrected value itself, so it applies to the next document regardless of its specific date or amount:\n\n``` python\ndef learn_transform(raw, truth):\n    # Day/month swap: 2026-07-03 read as 2026-03-07\n    if raw.count(\"-\") == 2 and truth.count(\"-\") == 2:\n        ry, rm, rd = raw.split(\"-\")\n        ty, tm, td = truth.split(\"-\")\n        if ry == ty and rm == td and rd == tm:\n            return lambda v: \"-\".join([v.split(\"-\")[0], v.split(\"-\")[2], v.split(\"-\")[1]])\n    # Stray leading digit: 12.34 read as 112.34\n    if \".\" in raw and raw[1:] == truth:\n        return lambda v: v[1:]\n    return lambda v: v   # not systematic, do not transform\n\n# per field:\nraw = call_llm(field)\nif key in corrections:\n    value = corrections[key](raw)          # apply what review taught us\nelif raw != truth and not reviewed[key]:\n    corrections[key] = learn_transform(raw, truth)   # one human review\n    reviewed[key] = True\n    value = truth\nelse:\n    value = raw\n```\n\nThe Bedrock call itself is a single-field Converse request at temperature 0 with `maxTokens` 64, asking for the value only. Token counts come from the response `usage` block, which is what drives the cost column.\n\nThe harness is not public, but nothing in it is hard to rebuild, and the method is the useful part. Generate a few hundred synthetic documents from two or three fixed templates, with ground-truth values and bounding boxes per field. Inject one or two systematic errors per type (a date swap and a stray digit are enough to start). Run every arm over the same seeded stream in the same order, log every field, and compute the metrics from the logs afterwards rather than inside the arms. That last step keeps the arms simple and makes each number auditable against the raw records.\n\nI also ran the same harness with a deterministic stub in place of the model, with no AWS account, to check the metric math. It showed the same structure as the Bedrock run: the baseline at 0.803 accuracy and a 0.197 repeated-error rate, spatial memory at 30 calls and 0.990 accuracy, and correction memory at 1.000 accuracy with no repeated errors. When the stub and the real model agree on the shape, the shape comes from the mechanism and not from one model's behavior.\n\nThree things are on the list. A combined arm, spatial memory first and correction memory on whatever still reaches the model. The real libraries swapped in for the reimplemented mechanisms. And document types where layouts drift, which is where spatial memory should hand back to the model and where `textract-field-memory`'s drift detection earns its place.\n\nIf you run document pipelines and have a layout-heavy, correction-heavy workload, I would like to hear which of the two problems costs you more. That answer decides which layer to add first.\n\nLibraries: [`textract-field-memory`](https://github.com/aws-samples/sample-textract-field-memory) and `sample-prompt-correction-memory`", "url": "https://wpnews.pro/news/spatial-memory-vs-correction-memory-i-benchmarked-both-against-a-plain-llm", "canonical_source": "https://dev.to/avneet_bansal_a65b3f31fc4/spatial-memory-vs-correction-memory-i-benchmarked-both-against-a-plain-llm-3c39", "published_at": "2026-11-05 04:16:24+00:00", "updated_at": "2026-11-05 04:26:11.927021+00:00", "lang": "en", "topics": ["ai-agents", "large-language-models", "ai-tools", "mlops", "artificial-intelligence"], "entities": ["Amazon Bedrock", "Claude Haiku 4.5", "Anthropic", "textract-field-memory", "sample-prompt-correction-memory", "AWS"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/spatial-memory-vs-correction-memory-i-benchmarked-both-against-a-plain-llm", "markdown": "https://wpnews.pro/news/spatial-memory-vs-correction-memory-i-benchmarked-both-against-a-plain-llm.md", "text": "https://wpnews.pro/news/spatial-memory-vs-correction-memory-i-benchmarked-both-against-a-plain-llm.txt", "jsonld": "https://wpnews.pro/news/spatial-memory-vs-correction-memory-i-benchmarked-both-against-a-plain-llm.jsonld"}}