Unlimited-OCR: Parsing a 40-Page PDF in One Pass Without Your GPU Melting

If you have ever built a document-parsing pipeline, you know the ritual. Split the PDF into pages. Run each page through the OCR model. Stitch the outputs back together. Then write a pile of glue code to fix the tables that got cut in half at a page boundary, and the heading that lost its section, and the footnote that ended up orphaned three pages away from its reference.

Baidu's Unlimited-OCR is a bet that you should not have to do any of that. It parses dozens of pages in a single forward pass, and it is MIT licensed.

This post covers what the model actually does differently, and then three concrete ways to get it running: a quick Transformers script, an SGLang server for real throughput, and a Docker image if you would rather not touch a Python environment at all.

Standard transformer decoding has a cost that most OCR benchmarks quietly hide: the KV cache grows with every token you generate.

The KV cache is the model's short-term memory. Every token the model produces gets appended to it so future tokens can attend backwards. For a chatbot writing three paragraphs, this is fine. For an OCR model transcribing a 40-page technical manual into 30,000 tokens of Markdown, it is not. Memory climbs, attention cost climbs with it, and generation gets slower the longer it runs. Somewhere around page eight your throughput graph stops looking like a line and starts looking like a cliff.

The industry workaround has been chunking. Process one page, dump the cache, process the next page, and accept that the model has no idea what came before.

Unlimited-OCR attacks the cache growth directly. The team replaced every attention layer in the decoder with something they call Reference Sliding Window Attention (R-SWA).

The intuition is close to how a human copy-typist works. You do not hold every word you have already typed in your head. You hold the last sentence or two, and you keep glancing back at the source document. R-SWA does the same thing: the decoder keeps a fixed-size window of recently generated tokens, but it retains permanent access to the original image tokens. The KV cache is implemented as a queue with a fixed capacity, so when a new token arrives, the oldest one in the window gets evicted.

The result is a cache size that is constant rather than growing. Memory and per-token latency stay flat whether you are on token 500 or token 30,000.

The architecture is DeepSeek-OCR's DeepEncoder (SAM-ViT-B plus CLIP-L) feeding an MoE decoder. The encoder's aggressive visual token compression is what keeps the image side of the cache small enough for the whole thing to work.

One thing the paper flags that is easy to miss: R-SWA is not OCR-specific. It is a general-purpose attention mechanism for any long-horizon transcription-shaped task, ASR included.

You need an NVIDIA GPU. There is no CPU or Apple Silicon path in the official repo.

The weights are bfloat16, so roughly 6 GB just to load the model, before activations and image tokens. A 12 GB card is a sane floor for single-image work; for multi-page runs at 32K context you will want more headroom. The maintainers tested on Python 3.12.3 with CUDA 12.9.

If you just want to see output before committing to any of this, there is a Hugging Face Space you can drop a file into.

This is the fastest way to a working result. Set up the environment:

python -m venv .venv
source .venv/bin/activate

pip install torch==2.10.0 torchvision==0.25.0 transformers==4.57.1
pip install Pillow==12.1.1 matplotlib==3.10.8 einops==0.8.2
pip install addict==2.4.0 easydict==1.13 pymupdf==1.27.2.2 psutil==7.2.2

Pin these versions. The model ships custom modeling code via trust_remote_code

, and that code is written against these specific releases. Version drift here produces confusing import errors rather than clean failures.

Now a single image:

import torch
from transformers import AutoModel, AutoTokenizer

model_name = 'baidu/Unlimited-OCR'

tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
model = AutoModel.from_pretrained(
    model_name,
    trust_remote_code=True,
    use_safetensors=True,
    torch_dtype=torch.bfloat16,
)
model = model.eval().cuda()

model.infer(
    tokenizer,
    prompt='<image>document parsing.',
    image_file='your_image.jpg',
    output_path='your/output/dir',
    base_size=1024, image_size=640, crop_mode=True,   # gundam mode
    max_length=32768,
    no_repeat_ngram_size=35, ngram_window=128,
    save_results=True,
)

Two details in there matter more than they look:

The prompt must literally start with <image>. It is not decoration, it is the placeholder token the visual features get spliced into. Drop it and you will get output that looks like the model hallucinating a document it never saw.

** no_repeat_ngram_size and ngram_window are load-bearing.** Long-horizon generation on repetitive layouts, think a table of contents or a price list, can send the model into a loop where it happily emits the same row forever. These parameters block any 35-gram from repeating inside a sliding window. Do not remove them because they look like tuning noise.

Single-image inference gives you two configurations, and the naming is not self-explanatory:

Multi-page and PDF paths only support base mode. That is a hard constraint, not a default.

model.infer_multi(
    tokenizer,
    prompt='<image>Multi page parsing.',
    image_files=['page1.png', 'page2.png', 'page3.png'],
    output_path='your/output/dir',
    image_size=1024,
    max_length=32768,
    no_repeat_ngram_size=35, ngram_window=1024,
    save_results=True,
)

Note that ngram_window

jumps from 128 to 1024 here. Across many pages there is legitimately more repeated structure, so the repeat-detection window has to widen to keep up.

There is no direct PDF entry point. You rasterize first, then feed the images to infer_multi

:

import os, tempfile
import fitz  # PyMuPDF

def pdf_to_images(pdf_path, dpi=300):
    doc = fitz.open(pdf_path)
    tmp_dir = tempfile.mkdtemp(prefix='pdf_ocr_')
    mat = fitz.Matrix(dpi / 72, dpi / 72)
    paths = []
    for i, page in enumerate(doc):
        out = os.path.join(tmp_dir, f'page_{i+1:04d}.png')
        page.get_pixmap(matrix=mat).save(out)
        paths.append(out)
    doc.close()
    return paths

model.infer_multi(
    tokenizer,
    prompt='<image>Multi page parsing.',
    image_files=pdf_to_images('your_doc.pdf', dpi=300),
    output_path='your/output/dir',
    image_size=1024,
    max_length=32768,
    no_repeat_ngram_size=35, ngram_window=1024,
    save_results=True,
)

300 DPI is the recommended default. Going lower to save memory costs you small text and table rules, which is usually the exact content you cared about.

Transformers is fine for evaluating the model. For a service handling concurrent requests, run SGLang and talk to it over an OpenAI-compatible API.

Set up the environment. Note that SGLang ships as a local wheel in the repo rather than from PyPI:

uv venv --python 3.12
source .venv/bin/activate

uv pip install wheel/sglang-0.0.0.dev11416+g92e8bb79e-py3-none-any.whl
uv pip install kernels==0.11.7
uv pip install pymupdf==1.27.2.2

Small heads-up: the README prose mentions pinning kernels==0.9.0

while the command block right beneath it installs 0.11.7

. Follow the command block. If you hit kernel-related errors, that mismatch is the first thing to check against the current repo state.

Launch the server:

python -m sglang.launch_server \
    --model baidu/Unlimited-OCR \
    --served-model-name Unlimited-OCR \
    --attention-backend fa3 \
    --page-size 1 \
    --mem-fraction-static 0.8 \
    --context-length 32768 \
    --enable-custom-logit-processor \
    --disable-overlap-schedule \
    --skip-server-warmup \
    --host 0.0.0.0 \
    --port 10000

The flags that are not optional:

--enable-custom-logit-processor

— the no-repeat-ngram processor runs as a custom logit processor. Without this flag, requests referencing it fail.--attention-backend fa3

— FlashAttention 3, which requires Hopper-class hardware. On older GPUs you will need a different backend.--page-size 1

and --disable-overlap-schedule

— R-SWA's cache eviction does not play nicely with the usual paged-attention and overlapped-scheduling optimizations.Then a client. The image goes in as a base64 data URL, standard OpenAI vision format, with a couple of model-specific extras:

import base64, json, os
import requests
from sglang.srt.sampling.custom_logit_processor import (
    DeepseekOCRNoRepeatNGramLogitProcessor,
)

server_url = "http://127.0.0.1:10000"
session = requests.Session()
session.trust_env = False

def encode_image(image_path):
    ext = os.path.splitext(image_path)[1].lower()
    mime = "image/jpeg" if ext in (".jpg", ".jpeg") else f"image/{ext.lstrip('.')}"
    with open(image_path, "rb") as f:
        data = base64.b64encode(f.read()).decode("utf-8")
    return {"type": "image_url", "image_url": {"url": f"data:{mime};base64,{data}"}}

def generate(prompt, image_paths, image_mode, ngram_window):
    content = [{"type": "text", "text": prompt}] + [
        encode_image(p) for p in image_paths
    ]
    payload = {
        "model": "Unlimited-OCR",
        "messages": [{"role": "user", "content": content}],
        "temperature": 0,
        "skip_special_tokens": False,
        "images_config": {"image_mode": image_mode},
        "custom_logit_processor": DeepseekOCRNoRepeatNGramLogitProcessor.to_str(),
        "custom_params": {"ngram_size": 35, "window_size": ngram_window},
        "stream": True,
    }
    response = session.post(
        f"{server_url}/v1/chat/completions",
        headers={"Content-Type": "application/json"},
        data=json.dumps(payload),
        timeout=1200,
        stream=True,
    )
    response.raise_for_status()

    chunks = []
    for line in response.iter_lines(chunk_size=1, decode_unicode=True):
        if not line or not line.startswith("data: "):
            continue
        data = line[len("data: "):]
        if data == "[DONE]":
            break
        delta = json.loads(data)["choices"][0].get("delta", {}).get("content", "")
        if delta:
            print(delta, end="", flush=True)
            chunks.append(delta)
    return "".join(chunks)

generate("document parsing.", ["your_image.jpg"],
         image_mode="gundam", ngram_window=128)

temperature: 0

and skip_special_tokens: False

are both deliberate. This is transcription, not generation, so any sampling randomness is pure downside. And the special tokens carry layout structure you want in the output.

The 1200 second timeout is also not paranoia. Long documents take a while, which is precisely why you want streaming.

The repo ships infer.py

, which starts the SGLang server for you and fires concurrent requests at it:

python infer.py \
    --image_dir ./examples/images \
    --output_dir ./outputs \
    --concurrency 8 \
    --image_mode gundam

python infer.py \
    --pdf ./examples/document.pdf \
    --output_dir ./outputs \
    --concurrency 8 \
    --image_mode gundam

Useful extras: --model_dir

accepts a local path or a Hugging Face ID, --gpu

sets CUDA_VISIBLE_DEVICES

, and --server_log

puts server output somewhere you can read it.

Start concurrency low. Eight is the documented example, but the right number depends on your VRAM and how long your documents are.

If you want to skip environment management entirely:

docker pull vllm/vllm-openai:unlimited-ocr

docker pull vllm/vllm-openai:unlimited-ocr-cu129
docker run --rm --gpus all --network host --ipc host \
    vllm/vllm-openai:unlimited-ocr \
    baidu/Unlimited-OCR \
    --trust-remote-code \
    --logits_processors vllm.model_executor.models.unlimited_ocr:NGramPerReqLogitsProcessor \
    --no-enable-prefix-caching \
    --mm-processor-cache-gb 0

Same story as SGLang regarding the ngram processor, it has to be registered at server start. Prefix caching and the multimodal processor cache are both disabled because R-SWA's fixed-window cache invalidates the assumptions those optimizations make.

Full details are in the official vLLM recipe.

Reach for something else if you need bounding boxes and per-word confidence scores, since this is an end-to-end model producing Markdown, not a detection-plus-recognition pipeline. Same if you are OCRing short receipts or single lines, where a 3B model on a GPU is enormous overkill compared to PaddleOCR or Tesseract. And if you have no NVIDIA GPU, there is no supported path today.

The interesting claim here is not the benchmark number. It is that a fixed-size KV cache made the model both faster and more accurate on long documents, when efficiency work almost always costs you quality somewhere.

If R-SWA generalizes the way the authors suggest, the same trick applies to any task where a model transcribes a long input into a long output while keeping the source in view. Long-form ASR is the obvious next one.