{"slug": "how-a-gpt-5-4-harness-scored-above-gpt-6-astra-and-claude-code-on-extractbench", "title": "How a GPT-5.4 harness scored above GPT-6 Astra and Claude Code on ExtractBench", "summary": "A developer's open-source harness, KYNEX, scored 92.97 F1 on LlamaIndex's ExtractBench (370 documents, 4,869 pages), ranking 12th of 53 systems and beating GPT-6 Astra (91.9) and Claude Code with Opus (87.1) while running on GPT-5.4; the LlamaIndex team independently re-ran and verified the result. The harness skips chunking, vector stores and retrieval, parsing each document once into a page-level world model that the model queries through typed tools with provenance, backed by a deterministic admission gate and five ablatable repair rules. The largest single gain came from page-wise enumeration, which lifted the 26,725-row table case from 0.00 to 0.97, and the same harness runs at ABB on 200+ page engineering RFQs.", "body_md": "On LlamaIndex ExtractBench (370 documents, 4,869 pages), my open-source harness **KYNEX** scores **92.97 F1**. It is ranked **12th of 53 systems** on the leaderboard, and the result was **independently re-run and verified by the LlamaIndex team**.\n\nIt runs on **GPT-5.4**. On the same leaderboard:\n\n| System | F1 | \n|---|---|\n| LlamaExtract Agentic Plus (commercial) | 96.2 | \n| Reducto Deep Extract (commercial) | 96.1 | \n| **KYNEX Harness (GPT-5.4)** | **92.97** | \n| GPT-6 Astra | 91.9 | \n| Claude Code (Opus) | 87.1 | \n| Gemini 3.8 Flash | 80.7 | \n\nThe top two are commercial APIs with their own infrastructure. A cheaper model inside a well-built harness still finished above several frontier models.\n\nThis post covers what the harness does differently and the specific failures each part fixes. My takeaway: for structured extraction, **the model's access to the document mattered more than the model itself**.\n\nOne benchmark document asks for every row of a large unclaimed-property table: **26,725 rows**.\n\n`null`, and the score was Neither approach is a prompt problem. Both are **access** problems.\n\nKYNEX has no chunking, no vector store and no retrieval step. The document is parsed once into a page-level \"world model\" with a lexical index. The model then gets typed tools and decides what to read, the way an analyst flips through pages:\n\nEvery tool result carries **provenance** (page, location, snippet), so each answer can be traced back to evidence.\n\n```\ndocuments ──> parse once ──> WorldModel (pages, tables, lexical index)\n                                  │\nschema ──> per-field contracts ───┤\n                                  ▼\n                 Agent (LangGraph, 3 phases)  <── Supervisor\n                   │ calls typed tools          (tracks progress,\n                   │ (text/table/search/        resumes stalls)\n                   │  OCR/vision/Azure DI)\n                   ▼\n           Admission gate (deterministic, outside the model)\n             - numbers must appear literally in the evidence\n             - text needs enough token overlap with the evidence\n                   ▼\n           Repair pipeline (5 rules, each with a kill switch)\n                   ▼\n                JSON output\n```\n\nTwo design choices matter most here:\n\nAfter the agent finishes, five deterministic post-processing rules run. Each one targets a failure I actually saw, each has its own environment-variable **kill switch**, and each was ablated on its own:\n\n| Rule | Failure it fixes | Off → On | \n|---|---|---|\n| Page-wise enumeration | 26,725-row array returned as `null` | 0.00 → 0.97 | \n| Non-record pruning | Contact blocks counted as data rows | 0.23 → 0.83 | \n| Redaction markers | Withheld cells read as real values | 0.47 → 0.88 | \n| Structured table rows | Values drift between columns | 0.57 → 1.00 | \n| OCR fallback | Pages with no text layer | 0.10 → 0.97 | \n\nThe biggest gain is page-wise enumeration. Instead of asking for the whole array at once, the harness probes two pages to learn the row shape. It then enumerates the table page by page and merges the results. The output never has to fit in one generation.\n\nFrontier models are smarter, but in a single pass they all hit the same limits:\n\nNone of these get fixed by more model intelligence. They get fixed by **what the model can see, how its work is split up, and what checks its output**. A harness turns those into engineering problems you can measure, ablate and switch off.\n\nThe same harness runs at ABB on engineering RFQs. It produces RFQ summaries and CPQ configuration sheets from 200+ page technical documents with tens of thousands of table rows.\n\nIf you've worked on extraction from long, table-heavy documents: **did RAG or an agentic, tool-based approach work better for you, and where did it break?** I'd like to hear about the failure cases.", "url": "https://wpnews.pro/news/how-a-gpt-5-4-harness-scored-above-gpt-6-astra-and-claude-code-on-extractbench", "canonical_source": "https://dev.to/nikit117/how-a-gpt-54-harness-scored-above-gpt-6-astra-and-claude-code-on-extractbench-27o5", "published_at": "2026-11-04 10:18:16+00:00", "updated_at": "2026-11-04 10:28:46.241825+00:00", "lang": "en", "topics": ["ai-agents", "large-language-models", "ai-tools", "ai-research"], "entities": ["KYNEX", "LlamaIndex", "ExtractBench", "GPT-5.4", "GPT-6 Astra", "Claude Code", "ABB", "LangGraph"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/how-a-gpt-5-4-harness-scored-above-gpt-6-astra-and-claude-code-on-extractbench", "markdown": "https://wpnews.pro/news/how-a-gpt-5-4-harness-scored-above-gpt-6-astra-and-claude-code-on-extractbench.md", "text": "https://wpnews.pro/news/how-a-gpt-5-4-harness-scored-above-gpt-6-astra-and-claude-code-on-extractbench.txt", "jsonld": "https://wpnews.pro/news/how-a-gpt-5-4-harness-scored-above-gpt-6-astra-and-claude-code-on-extractbench.jsonld"}}