{"slug": "show-hn-ocr-buddy-local-browser-ocr-for-code-formulas-latex-and-tables", "title": "Show HN: OCR Buddy: local browser OCR for code, formulas (LaTeX) and tables", "summary": "OCR Buddy, a free and open-source Chrome extension, provides fully local OCR for code, formulas, and tables using PaddleOCR's PP-OCRv5 on ONNX Runtime Web, ensuring no data leaves the user's machine and eliminating hallucinated text common in generative models. The extension runs entirely in-browser via WebAssembly/WebGPU, supports region, viewport, and full-page capture, and bundles models for offline use.", "body_md": "**Faithful, fully-local OCR for Chrome.** Grab text from anything on screen — a\nregion, the viewport, or a whole scrolling page — code in a paused video, a\nparagraph in a PDF, a formula, a table. Or turn an entire page into clean\nMarkdown for an LLM. No server. No image ever leaves your machine. No\nhallucinated text.\n\n🌐 [ocr-buddy.com](https://ocr-buddy.com) · 🧩 Chrome extension (Manifest V3) ·\n🔓 Free & open source (MIT) · 🛡️ 100% local, privacy-first\n\n*Silent autoplay loops. ▶ Watch in High Quality Video: demo 1 · demo 2.*\n\nModern OCR is dominated by large autoregressive vision-language models. They top\nthe benchmarks — and they **invent fluent, plausible, wrong text** the moment the\npixels get unclear. For most uses that's an annoyance. For code, numbers, prices,\nIDs, or anything you intend to *trust*, a confidently-wrong transcription is worse\nthan no transcription at all. Those models are also far too heavy to run in a\nbrowser tab.\n\nOCR Buddy is built on the opposite bet: **faithfulness over fluency, and the whole\npipeline on your device.** The interesting part is that those two goals don't fight\n— they point at the *same* engineering choices.\n\nHallucination in OCR is largely **architectural**. A generative model predicts the\n*next likely token*, so when the image is ambiguous it falls back on its language\nprior and writes something that reads well but isn't there. The classic OCR family\n— **detection + CTC recognition** — has no such prior. It transcribes the glyphs\nthat are actually present and, when it can't, it fails to blanks or low-confidence\noutput. It never makes up a sentence.\n\nThat family is also small, fast, and runs comfortably in WebAssembly/WebGPU. So:\n\nIn-browserandno-hallucinationare not a tradeoff. Both constraints select the same stack: PaddleOCR'sPP-OCRv5(Apache-2.0) onONNX Runtime Web.\n\nEverything below follows from that one decision.\n\n```\ncontent overlay (drag-select a region)\n      │  rect + devicePixelRatio\n      ▼\nservice worker  (coordinator only — no DOM, no model, no inference)\n      │  captureVisibleTab → crop on an OffscreenCanvas → PNG data URL\n      ▼\noffscreen document  (cross-origin isolated, WebGPU-capable, long-lived)\n      └─ PP-OCRv5 (+ pix2text-mfr for formulas) on ONNX Runtime Web\n      ▼\nside panel  (crop shown beside the result; low-confidence words flagged)\n```\n\nA few choices worth calling out, because each solved a concrete problem:\n\n**The service worker only coordinates.** MV3 service workers are ephemeral and have no DOM. The heavy, warm OCR engine lives in an**offscreen document**— a real page I keep alive, made*cross-origin isolated*(COOP/COEP) so it can use`SharedArrayBuffer`\n\nfor multi-threaded WASM, with WebGPU as the primary backend.**Capture uses** Grabbing a frame off a cross-origin video taints the canvas and the read fails.`chrome.tabs.captureVisibleTab`\n\n, not`<video>`\n\nframe-grabbing.`captureVisibleTab`\n\nreturns clean, composited pixels — so OCR-ing code from a paused YouTube video Just Works.**Models are bundled, not downloaded.** They ship inside the extension, so the default experience is genuinely offline and nothing — not even a model fetch — touches the network at runtime. The one exception is explicit: choosing a**non-Latin language pack** downloads that recognizer once (from the same pinned open-source repo the bundled models come from), caches it locally, and never touches the network for it again.\n\nBuilt with Vite + CRXJS. Requires **Chrome 124+** (WebGPU in workers).\n\nRegion-select is precise, but dragging a box is overkill when you just want\n*everything* on screen — or everything on a long page. So there are three ways to\nchoose what gets OCR'd, all sharing the selected read mode and language:\n\n**Select region**— drag a box over any part of the page (the original flow).** Capture viewport**— OCR everything currently on screen, in one click, no drag.** Capture full page**— scroll-capture the whole page and OCR it. Each viewport is captured as its own tile and OCR'd separately (so text stays above the detector's downscale threshold and remains legible), then the tiles are merged with seam de-duplication. Caveats, stated honestly: it runs the Text/Code pipeline (not Table/Formula); very long pages stop at a tile cap and say so in the result; pages with a sticky header/sidebar may repeat that text across tiles; lazy-loaded content is captured only as far as it has loaded; and complex multi-column layouts (e.g. a wiki article with a side TOC) can interleave in reading order.\n\nA separate one-click action turns the **current page into faithful Markdown**, shown\nin a preview you can copy or download as a `.md`\n\n. Unlike capture, this reads the page's\n**DOM structure** (headings, lists, links, tables, code blocks) — not pixels — so the\noutput is real Markdown, not OCR'd text, and it's clean enough to hand straight to an\nLLM. It's fully local (reads the DOM, no network), keeps the full page (nav/header/\nfooter included), and resolves relative links to absolute.\n\nIt's also **hybrid**: text baked into readable images is OCR'd and inserted right\nafter the image as a clearly-labelled blockquote (`> **Text extracted from image (OCR):** …`\n\n), so it's never silently merged into the prose — you always know which\ntext came from a picture. Cross-origin images can't be read (browser canvas taint),\nso those keep just their `alt`\n\ntext.\n\nYou pick how a region should be read — and you can change your mind *after*\ncapturing, with the **\"Read as\" switcher** in the result view, which re-runs a\ndifferent mode on the same crop without re-selecting.\n\nPlain OCR for code, prose, or any text. The journey here was mostly about\n**faithfully reconstructing layout from geometry**, because the recognizer only\nemits glyphs:\n\n- The Latin recognizer's dictionary has\n**no space token**— so inter-word spacing is reconstructed from the gaps between word boxes, and blank lines in code from vertical gaps. - A\n**per-box** recognition strategy (each detected box on its own crop) keeps real gaps intact; the default per-line strategy merged adjacent words (\"you should\" → \"youshould\"). **Column-aware reading order** clusters boxes into columns by x-gaps and reads column-major, so two-column papers don't come out interleaved.- A\n**Code view** rebuilds indentation from box geometry and syntax-highlights it. - A\n**homoglyph fold** maps stray Greek/Cyrillic look-alikes back to Latin, and a tightly-scoped rule folds an`o`\n\nwedged between digits back to`0`\n\n(`4o0`\n\n→`400`\n\n) — without touching code identifiers like`arg0`\n\nor octal`0o755`\n\n. (Both folds apply only to Latin-script packs — for Cyrillic or Greek packs those glyphs are the real text.) **Language packs.** Latin (EN/IT/FR/DE/ES + ~40 more) is built in. A selector in the panel adds Chinese+Japanese, Cyrillic, East Slavic, Greek, Korean, Thai, Devanagari, Tamil and Telugu — each a PP-OCRv5 recognizer (~8–17 MB) downloaded once on selection and cached for offline use. No Arabic yet: the line/spacing reconstruction assumes left-to-right, and shipping a language it would scramble is the kind of overpromise this project avoids.\n\nThe one place I *had* to use a generative model — there's no CTC equivalent that\nemits structured LaTeX. That reopens the hallucination risk the whole project\navoids, so the design is built around containing it.\n\n- Model:\n(MIT, a TrOCR-style vision encoder-decoder), bundled as a quantized ~23 MB encoder + 30 MB decoder, lazy-loaded only when you actually use Formula mode.[pix2text-mfr](https://huggingface.co/breezedeus/pix2text-mfr) - I run it\n**directly on ONNX Runtime Web**(which the project already bundles) rather than a higher-level library: that model's ONNX export has no merged KV-cache decoder, which breaks the usual generation loop, so the greedy decode is hand-rolled (feed the full sequence each step — O(n²), but a fraction of a second per formula). The image preprocessing and the byte-level tokenizer decode were validated against a reference implementation to floating-point precision before shipping. **The guardrail is visual, not statistical.** The predicted LaTeX is rendered with**KaTeX right beside the source crop**, so a mismatch is obvious at a glance. If KaTeX can't render the output, or the decode degenerates, OCR Buddy** abstains and shows the crop as an image**— it never presents invented LaTeX as if it were read.** Honest limit:**this is a small local model. It's accurate on clean and moderately complex formulas; it can misread dense, low-resolution ones. That's the price of staying in-browser — and exactly why the render-beside-crop check exists.\n\nA single table → a Markdown grid, reconstructed by **pure geometry** from the OCR\nword boxes: rows by vertical position, columns from an x-coverage profile, each\nword placed in its nearest column. No extra model. Because it keys off column\n*alignment* rather than ruled lines, it handles **borderless tables** — which a\nlayout model reads as figures.\n\nThe thought-flow wasn't a straight line. Two experiments shipped and were then removed, on purpose:\n\n**A full \"Document mode\"**(whole-page layout analysis with a PicoDet CDLA model) could parse a real paper into headings, columns, tables and equations in reading order. But page-layout models need a*full page*of context: on a single tight crop they misclassify — a standalone borderless table reads as a*Figure*, a cropped paragraph block gets dropped. Since the tool is used by selecting one region at a time, the mode was unreliable for how people actually reach for it. I removed it (and its 7.4 MB model) and replaced it with the focused, reliable single-region**Formula** and**Table** modes.**A high-level inference library for the formula model.** Its generation loop assumes a KV-cache decoder this model doesn't export, which silently corrupted the output. It would also have bundled a*second*copy of the ONNX runtime. Hand-rolling the decode on the runtime I already ship was both correct and lighter.**An auto-fix for the pipe** A lone vertical bar is visually identical to capital-I, lowercase-L and the digit 1; the recognizer (favouring letters) usually picks one of those, so pipes in code and tables read as`\\|`\n\n→`I`\n\n/`l`\n\nconfusion.`I`\n\n. The tempting fix — fold a standalone`I`\n\n/`l`\n\nback to`\\|`\n\n— was dropped because the glyphs are*genuinely*ambiguous (especially in monospace, where their boxes are identical width), so any rule that catches real pipes also corrupts real`I`\n\n/`l`\n\n/`1`\n\n. Turning someone's variable`l`\n\ninto`\\|`\n\nis exactly the silent corruption this tool refuses to make. A visible, correctable`I`\n\nbeats an invented`\\|`\n\n.\n\nKeeping these out is part of the design: a small, honest tool beats a broad, flaky one.\n\nOne thing that **did** ship from this round of testing: **coloured text on light\nbackgrounds** (a red form error, a blue link) used to be dropped at detection — the\ndetector is tuned for dark, high-contrast glyphs. A background-adaptive contrast\nboost (collapse each pixel to its darkest channel on light backgrounds) pulls weak\ncolour contrast up to strong luminance contrast, with no effect on ordinary\ndark-on-light text or on dark mode. Emoji remain best-effort: an undetected one is\nmarked with a `□`\n\nplaceholder, but one the recognizer boxes anyway can still come out\nas garbage — a Latin model has no glyph for it.\n\nAnti-hallucination isn't a tagline here, it's the feature set:\n\n- The\n**source crop is always shown beside the result**, so you can verify. **Per-word confidence** is exposed; low-confidence words are flagged, not silently trusted.- A blank or ambiguous region yields\n**empty output**— never invented filler. - Formulas are\n**rendered beside the crop** and**abstain to the image** when unsure.\n\nMeasured with `scripts/ocr-image-test.mjs`\n\n(Node, the exact PP-OCRv5 config the\nextension uses) against ground truth on real academic pages:\n\n- A coherent\n**text block**(the normal \"select a region\" workflow) scores**≈ 99.9–100/100** character accuracy. On clean prose it's effectively verbatim — sentences, citations like`[22]`\n\n, tokens like`RoPE-2D`\n\n, all correct. - Capturing a paragraph\n**and an adjacent table together** drops the score — but that's**reading-order** interleaving, not misrecognition; the characters are right, the order isn't. Selecting one region (or using Table mode for the table) restores it. **Equations and tables aren't text**— use Formula and Table modes for those; Text/Code mode flattens them.\n\nIn short: on the content each mode is meant for, accuracy is essentially perfect. I don't claim \"100% OCR of anything\" — that would be the kind of overstatement the project is a reaction against.\n\n**Nothing leaves your device.** No servers, no API calls, no telemetry. The only network use is downloading the extension itself from the store — and, if you explicitly pick a non-Latin language pack, a one-time fetch of that model from the pinned open-source repo (a plain file download; no page content, no image, no telemetry rides along). It's cached locally and offline from then on.**Models are bundled**, so even first-run inference is fully offline.- The selection overlay is passive and does not read page content; the screenshot permission for a site is requested explicitly, per-site, only when needed.\n\n```\nnpm install\nnpm run dev        # Vite + CRXJS with HMR\nnpm run build      # production build → dist/\nnpm run typecheck\n\n# load the unpacked extension:\n#   chrome://extensions → Developer mode → Load unpacked → select dist/\n```\n\nTesting:\n\n```\nnpm run ocr-bench                  # synthetic OCR benchmark (Node/CPU)\nnpm run fetch-test-images          # populate test-images/ (Wikipedia per script, code, LaTeX)\nnode scripts/ocr-image-test.mjs    # score real images in test-images/ vs ground truth\n                                   #   (a <lang>__ filename prefix picks the language pack)\nnpm run verify                     # load the built extension in Chromium (Playwright)\nnpm run verify:ux                  # end-to-end capture flow (Playwright)\n```\n\nAll models are bundled in the extension (`public/models/`\n\n) and run entirely\non-device. Provenance and pinned versions are in `public/models/SOURCE.md`\n\n.\n\n| Model | Role | Source | License |\n|---|---|---|---|\n`PP-OCRv5_mobile_det_infer.onnx` (~4.7 MB) |\nText detection |\n|\n\n`latin_PP-OCRv5_mobile_rec_infer.onnx`\n\n(~8 MB)[ppu-paddle-ocr-models](https://github.com/PT-Perkasa-Pilar-Utama/ppu-paddle-ocr-models)· upstream[PaddleOCR](https://github.com/PaddlePaddle/PaddleOCR)`mfr_decoder.onnx`\n\n(~30 MB) + tokenizer[breezedeus/pix2text-mfr](https://huggingface.co/breezedeus/pix2text-mfr)`mfr_encoder.onnx`\n\n(~23 MB, int8)[Brian314/pix2text-mfr-quantized](https://huggingface.co/Brian314/pix2text-mfr-quantized)Inference runs on [ONNX Runtime Web](https://github.com/microsoft/onnxruntime)\n(WebGPU, with multi-threaded WASM fallback), via\n[ppu-paddle-ocr](https://github.com/PT-Perkasa-Pilar-Utama/ppu-paddle-ocr) for the\nPP-OCRv5 path.\n\nOCR Buddy's own source code is **MIT** (see [ LICENSE](/Fanfulla/OCR-buddy/blob/main/LICENSE)).\n\nIt bundles third-party models and libraries under their own permissive licenses —\nPaddleOCR / PP-OCRv5 (**Apache-2.0**), pix2text-mfr, ppu-paddle-ocr, ONNX Runtime\nand KaTeX (**MIT**), and highlight.js (** BSD-3-Clause**). All are compatible with\nredistribution in an MIT project; there is **no copyleft** anywhere in the stack.\nFull attribution and license texts are in\n[ public/THIRD_PARTY_LICENSES.md](/Fanfulla/OCR-buddy/blob/main/public/THIRD_PARTY_LICENSES.md), which ships with\nthe packaged extension, and model provenance is in\n\n`public/models/SOURCE.md`\n\n.The copyright holder in\n\n`LICENSE`\n\nis set to \"OCR Buddy contributors\" — change it to your name or organization if you prefer.\n\nOCR Buddy stands on excellent open-source work: **PaddleOCR / PP-OCRv5** (Baidu /\nPaddlePaddle), **pix2text-mfr** (breezedeus), **ppu-paddle-ocr** (PT. Perkasa Pilar\nUtama), **ONNX Runtime** (Microsoft), **KaTeX** (Khan Academy), and **highlight.js**.\nThank you.", "url": "https://wpnews.pro/news/show-hn-ocr-buddy-local-browser-ocr-for-code-formulas-latex-and-tables", "canonical_source": "https://github.com/Fanfulla/OCR-buddy", "published_at": "2026-07-07 21:54:50+00:00", "updated_at": "2026-07-07 21:59:33.102366+00:00", "lang": "en", "topics": ["computer-vision", "ai-tools", "developer-tools"], "entities": ["OCR Buddy", "Chrome", "PaddleOCR", "PP-OCRv5", "ONNX Runtime Web", "WebAssembly", "WebGPU", "Manifest V3"], "alternates": {"html": "https://wpnews.pro/news/show-hn-ocr-buddy-local-browser-ocr-for-code-formulas-latex-and-tables", "markdown": "https://wpnews.pro/news/show-hn-ocr-buddy-local-browser-ocr-for-code-formulas-latex-and-tables.md", "text": "https://wpnews.pro/news/show-hn-ocr-buddy-local-browser-ocr-for-code-formulas-latex-and-tables.txt", "jsonld": "https://wpnews.pro/news/show-hn-ocr-buddy-local-browser-ocr-for-code-formulas-latex-and-tables.jsonld"}}