{"slug": "critically-enterprise-agents-dont-understand-highlighting", "title": "Critically, Enterprise Agents Don’t Understand Highlighting", "summary": "A new arXiv paper (2610.30026) found that widely used document converters routinely discard highlighting and text colors, leaving AI agents unable to discern important instructions in enterprise documents. Of the 6,462 policy documents examined, nearly a third contained red text or highlighting and around one in ten used those markings to flag important requirements, while only 1% of the four billion paper documents circulating through global trade each year are digitized and the US National Archive reports just 4.04% of its analog documents are digitized, with 11.97 billion yet to go. The authors state that mature converters recover text, tables and structure faithfully, but the lost emphasis layer may misrepresent an author's intent when pages are reduced to plain text through OCR or vision-language models.", "body_md": "### \n[Anderson's Angle](https://www.unite.ai/series/andersons-angle/)\n\n# Critically, Enterprise Agents Don’t Understand Highlighting\n\n[Add Unite.AI to your preferred sources on Google](https://www.google.com/preferences/source?q=unite.ai)\n\nThere’s a lot of important, unscanned paper in the world, and particularly in the business world. It’s [estimated](https://www.apec.org/press/news-releases/2026/paper-trail-holds-back-digital-trade-processes-as-apec--abac-push-practical-fixes?trk=article-ssr-frontend-pulse_little-text-block#:~:text=Around%20four%20billion%20paper%20documents%20still%20circulate%20through%20global%20trade%20each%20year%2C%20with%20less%20than%20one%20percent%20fully%20digitalized%2E) that only 1% of the four billion paper documents circulating through global trade each year are digitized, leaving the rest beyond the [analytical powers](https://www.unite.ai/doclang-aims-to-become-the-universal-language-for-ai-ready-documents/) of machine learning.\n\nThat’s just a rough appraisal of the problem in the *business* sector; governments also produce [a huge amount](https://www.cps.gov.uk/node/29729#:~:text=Paper%20consumption%20%28A4%20reams) of printed or paper-bound output, and the US National Archive reported this year that [only 4.04%](https://www.archives.gov/findingaid/record-group-explorer) of the analog documents it maintains are digitized, with *11.97 billion* yet to go.\n\nTherefore it’s an ongoing rather than a legacy problem, and I confess that I am part of the problem too:\n\nIn reviewing papers, I print them, because they need a lot of *human* parsing as well as machine summary. Because time is tight, I scrawl large emphases over key sections and potential quotes, so that the shape of the data leaps out at me when it’s time to write. Since I’m not quite OCD enough to bother with highlighters, added emphasis is indicated by diverse symbols, larger text, and other very personalized signals of importance.\n\nBut a lot of people, including a lot of [important people writing important things](https://www.cam.ac.uk/news/churchill-papers-added-to-unescos-list-of-the-worlds-greatest-cultural-treasures#:~:text=The%20page%20is%20covered%20with%20his%20handwritten%20annotations%20in%20red%20and%20blue%20ink), do use color highlighters, and alternate text colors, for emphases.\n\nAccording to the very paper depicted in the earlier photo (the one on the black lectern), there may be documents numbering in the hundreds of millions that contain a layer of such emphasis that document-ingestion systems [such as Docling](https://web.archive.org/web/20261022180450/https:/docling.ai/) are not currently able to see.\n\nThese manual emphases are significant, giving weight to text in a way that isn’t necessarily reflected in the construction of the language. Therefore reducing the content of such a highlighted page to mere text through [OCR](https://www.ibm.com/think/topics/optical-character-recognition) (or [more advanced methods](https://www.unite.ai/why-agentic-document-extraction-is-replacing-ocr-for-smarter-document-automation/), such as [VLMs](https://www.unite.ai/vlm-vs-ocr-document-processing-understanding/)) does not convey all of the author’s intent, and may even misrepresent the purposes and intentions of the document.\n\n## Just the Highlights\n\nThe [new paper](https://arxiv.org/pdf/2610.30026)* finds that widely-used [document converters](https://web.archive.org/web/20260713102239/https:/www.file2markdown.ai/blog/pymupdf4llm-vs-docling) routinely discard highlighting and text colors, so that AI agents remain unable to discern important instructions. Of the 6,462 policy documents examined in the study’s tests, *nearly a third* contained red text or highlighting, and around one in ten used these markings to draw attention to important requirements.\n\nIn regard to enterprise document ingestion agents, the authors<sup>†</sup> state<sup>††</sup>:\n\n*‘Mature converters recover text, tables and structure faithfully, so the pipeline looks healthy.* \n\n*‘We show that one layer is lost by all of them—the meaning an author encodes in how text* looks*—and that the loss makes agents* confidently wrong *rather than visibly incomplete, which is the harder failure to detect in deployment.’*\n\nIn the authors’ test, when AI models were asked to identify important requirements in real policy documents, they performed far better when the original highlighting and text colors were preserved – a finding confirmed across six models, including GPT-5.6 Terra, Claude Opus 5, and Claude Sonnet 5.\n\nEffectively, it’s possible to extract two signals from highlighting: a) the fact that anything was highlighted at all and b) the *color* it was highlighted in. In an instance where multiple colors were used for manual highlighting, a logical rubric would code red *as in some way negative* (as we saw earlier, in Winston Churchill’s corrections to his own speeches):\n\nThe authors suggest that existing pipelines can solve the problem quite simply, by preserving the original highlighting and text colors during document conversion, and adding labels that explain their significance to the AI model. While this requires no additional AI processing, it does roughly double the number of tokens needed to represent each document:\n\n*‘Recovering these cues is deterministic, code-only, and costs no LLM calls; the entire effect reported here comes from parsing alone. It does cost tokens—roughly twice plain text—so it is a deliberate trade of context budget for factuality rather than a free win.’*\n\n## Bold Indifference\n\nOne interesting facet of the study is the extent to which the existing means of emphasis in typed digital documents is, apparently, ineffective through overuse<sup>††</sup>:\n\n*‘The entire effect [i.e., of the new pipeline] is carried by **highlight** [and] [**colour**]. Bold, italic, monospace and strikethrough contribute nothing [measurable], with confidence intervals straddling it.* \n\n*‘This is not because those cues are absent: a single document carries over a thousand bold markers. It is because they are too common to be informative.* \n\n*‘An author who bolds every heading and defined term has not thereby singled anything out, whereas colouring or highlighting a requirement is a deliberate, sparing act. Cue typing pays only for cues rare enough to signal intent.’*\n\n## Tasks and Tests\n\nTo establish the system, the authors scanned 6,462 real policy documents from an unspecified public corpus in .docx format, checking the documents for red text and highlighting, without making any guesses based on the wording. The models were then asked to pick out the requirements that had been marked as especially important, and leave the unmarked ones alone, with the core prompt:\n\n*‘List every requirement in this document that the author has singled out for special attention or elevated priority.’*\n\nFor the main test, 84 documents were selected, containing 332 marked requirements and 4,192 unmarked ones (i.e., requirements without red text or colored highlighting). Each document had to contain at least one marked requirement and three unmarked ones, so the AI couldn’t simply list everything.\n\nThe [ground truth](https://www.ibm.com/think/topics/ground-truth) was established directly from the documents’ formatting, rather than through human judgment or another AI model. Performance was measured using [precision, recall and F1](https://www.unite.ai/precision-vs-recall-vs-f1-score-which-metric-should-you-use/) scores, taking account of both correctly identified requirements and false positives, where unmarked requirements were mistakenly included.\n\nScoring was programmatic, with no LLM judge involved. To count as a match, an answer had to contain at least four consecutive content words from the original requirement, helping to prevent similar but different requirements from being counted as correct.\n\n### Ablating Highlights\n\nA further test was carried out to see whether important requirements could be identified from their wording alone, without highlighting or colors. A text-only classifier trained on examples achieved an [ROC AUC](https://developers.google.com/machine-learning/crash-course/classification/roc-and-auc) score of 0.664 (where 0.5 represents random guessing).\n\nSimpler methods performed no better than chance: a score of 0.508 was recorded when words suggesting urgency were used as clues, while sentence length produced a score of 0.492. These results suggest that very little can be learned about what the highlighter-wielding author *intended*, just from the wording alone.\n\nThis makes sense: presumably, if the language had been adequately forceful, the author would not have felt the need for any *post facto* emphasis.\n\n### Contenders\n\nThe approaches compared comprised the authors’ own parser, *OURS*; *OURS with salience roles*; *OURS with untyped markup*; *[docx2python](https://pypi.org/project/docx2python/)*; Docling; and plain-text conversion.\n\nThe same documents and questions were used for each approach, with only the *way* the documents were converted being changed. The principle comparison covered 76 documents, with each test repeated three times:\n\nAs shown above, the best overall F1 score of 0.690 was achieved by the authors’ system, compared with 0.586 for docx2python; 0.150 for Docling; and just 0.075 for plain text (F1 combines precision and recall into a single measure of accuracy, where 1.0 represents a perfect score).\n\nThe importance of preserving formatting was also demonstrated by the false-positive results: with Docling, unmarked requirements were incorrectly selected at nearly four times the rate recorded for the authors’ system (4.3% versus 1.1%).\n\nA separate test was used to examine whether AI models benefited from being told what the formatting meant, rather than simply being shown the original colors and highlighting, and a small but statistically significant improvement was found when these extra labels were provided. However, most of the benefit had already been obtained by preserving the formatting itself.\n\n### External Pressure\n\nThe researchers conducted an additional assessment using Microsoft’s [Office Comprehension Benchmark](https://arxiv.org/pdf/2607.01245) (OCB), which measures how accurately information can be recovered from Office documents.\n\n326 questions were extracted from 59 Word documents, covering formatting, tables, hyperlinks, comments and footnotes. Both the questions and the scoring criteria were supplied by Microsoft, independently of the study:\n\nAs shown above, an excellent overall score of 0.959 was achieved by the authors’ system, compared with a good score of 0.810 for docx2python, a moderate 0.720 for Docling, and a relatively poor 0.634 for plain text.\n\nPerfect scores were achieved by the new system for metadata, comments and footnotes, while all competing methods scored zero – though these improvements weren’t solely due to preserving highlighting and text colors; and the authors’ system also proved better at recovering information from tables, document layouts, fonts and hyperlinks.\n\n## Conclusion\n\nI tried with little success recently to see if my own manual emphasis of the tables and figures in a PDF science paper could be automated via LLMs and/or Python, and the results were…well, hilarious. Stratifying, quantifying and parsing science papers is now a well-subscribed strand in the literature, and there’s not much room – or hope – for the casual amateur.\n\nAs regards to this paper, the authors’ victory is to have identified a significant emphasis indicator that was being lost in translation. However, the doubling of token spend required by their solution is no joke, at the kind of scale that OCR and document ingestion layers need to operate at. Therefore, the prize here is to have *identified* the issue, I think.\n\n* *Titled ‘What the Author Wrote, What the Agent Reads: Silent Cue Loss in Enterprise Document Ingestion’.*\n\n<sup>†</sup> *All from Amazon.*\n\n<sup>††</sup> *Authors’ emphases, not mine, though I do routinely convert authors’ inline citations to hyperlinks.*\n\n*First published Friday, October 30, 2026*", "url": "https://wpnews.pro/news/critically-enterprise-agents-dont-understand-highlighting", "canonical_source": "https://www.unite.ai/critically-enterprise-agents-dont-understand-highlighting/", "published_at": "2026-10-30 00:00:00+00:00", "updated_at": "2026-10-30 15:20:30.173313+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-agents", "ai-research", "natural-language-processing"], "entities": ["Docling", "US National Archive", "APEC", "arXiv", "IBM"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/critically-enterprise-agents-dont-understand-highlighting", "markdown": "https://wpnews.pro/news/critically-enterprise-agents-dont-understand-highlighting.md", "text": "https://wpnews.pro/news/critically-enterprise-agents-dont-understand-highlighting.txt", "jsonld": "https://wpnews.pro/news/critically-enterprise-agents-dont-understand-highlighting.jsonld"}}