{"slug": "pixel-native-rag-a-practical-guide-to-visual-document-indexing", "title": "Pixel-Native RAG: A Practical Guide to Visual Document Indexing", "summary": "A new tutorial from StarTrail-org introduces PixelRAG, a pixel-native retrieval-augmented generation pipeline that renders web pages and PDFs as images, divides them into overlapping tiles, and generates multimodal embeddings with SigLIP, CLIP, or Qwen3-VL backends, storing vectors in a FAISS index. The system integrates OCR-based BM25 scoring with reciprocal rank fusion, aggregates tile-level evidence into document-level results, and exposes a FastAPI search service, with evaluation using Recall@k and mean reciprocal rank.", "body_md": "In this tutorial, we build a complete[ pixel](https://github.com/StarTrail-org/PixelRAG)-native retrieval-augmented generation pipeline from scratch and examine how document retrieval works without relying on conventional HTML parsing, text extraction, or fixed chunking strategies. We render web pages and PDF documents as images, divide them into overlapping tiles, generate multimodal embeddings with SigLIP, CLIP, or an optional Qwen3-VL backend, and store the resulting vectors in a FAISS index for efficient similarity search. We also strengthen retrieval with OCR-based BM25 scoring and reciprocal rank fusion, aggregate tile-level evidence into document-level results, and expose the system through a FastAPI search service. Along the way, we evaluate retrieval quality using Recall@k and mean reciprocal rank, train a lightweight residual adapter with contrastive learning, visualize retrieved screenshots, and optionally pass the strongest evidence tiles to a vision-language model for grounded answer generation.\n\n``` python\nimport os\nimport sys\nimport io\nimport re\nimport json\nimport time\nimport math\nimport shutil\nimport hashlib\nimport asyncio\nimport logging\nimport argparse\nimport threading\nimport subprocess\nfrom pathlib import Path\nfrom dataclasses import dataclass, field, asdict\nfrom typing import List, Dict, Any, Optional, Tuple\n@dataclass\nclass Config:\n   urls: List[str] = field(default_factory=lambda: [\n       \"https://en.wikipedia.org/wiki/Retrieval-augmented_generation\",\n       \"https://en.wikipedia.org/wiki/Vector_database\",\n       \"https://en.wikipedia.org/wiki/Transformer_(deep_learning_architecture)\",\n       \"https://en.wikipedia.org/wiki/Photosynthesis\",\n       \"https://en.wikipedia.org/wiki/Delhi\",\n   ])\n   include_synthetic_pdf: bool = True\n   tile_width: int = 1024\n   tile_height: int = 1024\n   tile_overlap: int = 128\n   device_scale: float = 1.0\n   max_page_height: int = 24000\n   max_tiles_per_doc: int = 12\n   min_tile_height: int = 200\n   blank_std_threshold: float = 6.0\n   dedup_hamming: int = 4\n   nav_timeout_ms: int = 60000\n   headless_args: List[str] = field(default_factory=lambda: [\n       \"--no-sandbox\", \"--disable-dev-shm-usage\", \"--hide-scrollbars\",\n       \"--disable-gpu\", \"--force-color-profile=srgb\", \"--font-render-hinting=none\",\n   ])\n   backend: str = \"siglip\"\n   model_id: str = \"google/siglip-base-patch16-224\"\n   qwen_model_id: str = \"Qwen/Qwen3-VL-Embedding-2B\"\n   embed_batch_size: int = 8\n   embed_image_size: Optional[int] = None\n   index_dir: str = \"./pixel_index\"\n   ivf_threshold: int = 2000\n   ivf_nprobe: int = 16\n   top_k_tiles: int = 20\n   n_docs: int = 5\n   use_ocr_hybrid: bool = True\n   rrf_k: int = 60\n   dense_weight: float = 1.0\n   sparse_weight: float = 1.0\n   enable_server: bool = True\n   server_port: int = 8000\n   enable_eval: bool = True\n   enable_adapter_train: bool = True\n   enable_vlm_answer: bool = False\n   vlm_model_id: str = \"Qwen/Qwen2.5-VL-3B-Instruct\"\n   show_plots: bool = True\n   work_dir: str = \"./pixelrag_work\"\n   seed: int = 0\nCFG = Config()\nEVAL_QUERIES: List[Tuple[str, str]] = [\n   (\"how do plants convert sunlight into chemical energy\", \"Photosynthesis\"),\n   (\"chlorophyll light dependent reactions\", \"Photosynthesis\"),\n   (\"converting scanned images of text into machine readable characters\", \"Optical_character\"),\n   (\"approximate nearest neighbour search over embeddings\", \"Vector_database\"),\n   (\"self-attention multi-head architecture\", \"Transformer\"),\n   (\"grounding a language model with retrieved documents\", \"Retrieval-augmented\"),\n   (\"capital territory of india red fort\", \"Delhi\"),\n]\nlogging.basicConfig(level=logging.INFO, format=\"%(asctime)s | %(levelname)-7s | %(message)s\",\n                   datefmt=\"%H:%M:%S\")\nlog = logging.getLogger(\"pixelrag\")\nfor noisy in (\"urllib3\", \"PIL\", \"matplotlib\", \"httpx\", \"asyncio\", \"uvicorn.error\"):\n   logging.getLogger(noisy).setLevel(logging.WARNING)\nIN_COLAB = \"google.colab\" in sys.modules\ndef _pip(*pkgs: str) -> None:\n   \"\"\"Install quietly; never explode the notebook on a single bad wheel.\"\"\"\n   cmd = [sys.executable, \"-m\", \"pip\", \"install\", \"-q\", \"--disable-pip-version-check\", *pkgs]\n   subprocess.run(cmd, check=False, stdout=subprocess.DEVNULL, stderr=subprocess.STDOUT)\ndef _have(mod: str) -> bool:\n   import importlib.util\n   return importlib.util.find_spec(mod) is not None\ndef ensure_deps(cfg: Config) -> None:\n   log.info(\"Installing dependencies (first run only, ~2-4 min)...\")\n   wanted = []\n   for mod, pkg in [\n       (\"PIL\", \"pillow\"), (\"numpy\", \"numpy\"), (\"faiss\", \"faiss-cpu\"),\n       (\"fitz\", \"pymupdf\"), (\"transformers\", \"transformers\"),\n       (\"fastapi\", \"fastapi\"), (\"uvicorn\", \"uvicorn\"), (\"requests\", \"requests\"),\n       (\"matplotlib\", \"matplotlib\"), (\"tqdm\", \"tqdm\"), (\"rank_bm25\", \"rank-bm25\"),\n       (\"playwright\", \"playwright\"), (\"sentencepiece\", \"sentencepiece\"),\n   ]:\n       if not _have(mod):\n           wanted.append(pkg)\n   if cfg.use_ocr_hybrid and not _have(\"pytesseract\"):\n       wanted.append(\"pytesseract\")\n   if wanted:\n       _pip(*wanted)\n   if not _have(\"torch\"):\n       log.warning(\"torch not found — installing CPU wheel (Colab normally ships torch).\")\n       _pip(\"torch\", \"torchvision\")\n   if cfg.use_ocr_hybrid and shutil.which(\"tesseract\") is None:\n       log.info(\"Installing tesseract-ocr system package...\")\n       subprocess.run(\"apt-get -qq update && apt-get -qq install -y tesseract-ocr\",\n                      shell=True, check=False,\n                      stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)\n       if shutil.which(\"tesseract\") is None:\n           log.warning(\"tesseract unavailable -> hybrid retrieval will run dense-only.\")\n           cfg.use_ocr_hybrid = False\n   marker = Path(cfg.work_dir) / \".chromium_ok\"\n   if not marker.exists():\n       log.info(\"Downloading Playwright Chromium...\")\n       r = subprocess.run([sys.executable, \"-m\", \"playwright\", \"install\", \"--with-deps\", \"chromium\"],\n                          capture_output=True, text=True)\n       if r.returncode != 0:\n           r = subprocess.run([sys.executable, \"-m\", \"playwright\", \"install\", \"chromium\"],\n                              capture_output=True, text=True)\n       if r.returncode == 0:\n           marker.parent.mkdir(parents=True, exist_ok=True)\n           marker.write_text(\"ok\")\n       else:\n           log.warning(\"Chromium install failed -> falling back to the text renderer.\\n%s\",\n                       (r.stderr or \"\")[-600:])\n   log.info(\"Dependencies ready.\")\ndef run_async(coro):\n   \"\"\"\n   Run a coroutine from a Jupyter/Colab cell.\n   Colab already owns a running event loop, which makes Playwright's *sync*\n   API raise. Rather than monkey-patching with nest_asyncio, we hand the\n   coroutine to a private loop on a private thread — the most robust option.\n   \"\"\"\n   box: Dict[str, Any] = {}\n   def _runner():\n       loop = asyncio.new_event_loop()\n       asyncio.set_event_loop(loop)\n       try:\n           box[\"value\"] = loop.run_until_complete(coro)\n       except BaseException as exc:\n           box[\"error\"] = exc\n       finally:\n           try:\n               loop.run_until_complete(loop.shutdown_asyncgens())\n           finally:\n               loop.close()\n   t = threading.Thread(target=_runner, daemon=True)\n   t.start()\n   t.join()\n   if \"error\" in box:\n       raise box[\"error\"]\n   return box.get(\"value\")\n```\n\nWe define the global configuration, evaluation queries, logging behavior, and runtime settings for the PixelRAG pipeline. We install the required Python and system dependencies, including Playwright, Chromium, Tesseract, FAISS, and transformer libraries. We also create an asynchronous execution helper that allows browser-rendering coroutines to run reliably inside Google Colab and Jupyter environments.\n\n```\n@dataclass\nclass Tile:\n   tile_id: str\n   doc_id: str\n   source: str\n   kind: str\n   page: int\n   seq: int\n   y0: int\n   y1: int\n   path: str\n   ocr_text: str = \"\"\n   title: str = \"\"\ndef _doc_id_from_source(src: str) -> str:\n   tail = src.rstrip(\"/\").split(\"/\")[-1] or src\n   tail = re.sub(r\"\\.(html?|pdf|png|jpg)$\", \"\", tail, flags=re.I)\n   return re.sub(r\"[^A-Za-z0-9_.\\-()]+\", \"_\", tail)[:80] or hashlib.md5(src.encode()).hexdigest()[:10]\ndef _ahash(img, size: int = 8) -> int:\n   \"\"\"64-bit average hash — cheap near-duplicate detection for repeated headers.\"\"\"\n   import numpy as np\n   g = img.convert(\"L\").resize((size, size))\n   a = np.asarray(g, dtype=\"float32\")\n   bits = (a > a.mean()).flatten()\n   out = 0\n   for b in bits:\n       out = (out << 1) | int(b)\n   return out\ndef _hamming(a: int, b: int) -> int:\n   return bin(a ^ b).count(\"1\")\ndef _is_informative(img, cfg: Config) -> bool:\n   \"\"\"Reject blank / solid-colour tiles before they ever reach the GPU.\"\"\"\n   import numpy as np\n   a = np.asarray(img.convert(\"L\"), dtype=\"float32\")\n   return float(a.std()) >= cfg.blank_std_threshold\ndef _save_tile(img, out_dir: Path, name: str) -> str:\n   out_dir.mkdir(parents=True, exist_ok=True)\n   p = out_dir / f\"{name}.png\"\n   img.convert(\"RGB\").save(p, format=\"PNG\", optimize=True)\n   return str(p)\ndef slice_image_to_tiles(img, cfg: Config, *, doc_id: str, source: str, kind: str,\n                        page: int, out_dir: Path, start_seq: int = 0,\n                        seen_hashes: Optional[List[int]] = None,\n                        title: str = \"\") -> List[Tile]:\n   \"\"\"Vertical sliding window with overlap. Used for PDFs and text fallback.\"\"\"\n   from PIL import Image\n   seen_hashes = seen_hashes if seen_hashes is not None else []\n   W, H = img.size\n   if W != cfg.tile_width:\n       new_h = max(1, int(H * cfg.tile_width / W))\n       img = img.resize((cfg.tile_width, new_h))\n       W, H = img.size\n   step = max(1, cfg.tile_height - cfg.tile_overlap)\n   tiles: List[Tile] = []\n   y, seq = 0, start_seq\n   while y < H and (seq - start_seq) < cfg.max_tiles_per_doc:\n       h = min(cfg.tile_height, H - y)\n       if h < cfg.min_tile_height and seq > start_seq:\n           break\n       crop = img.crop((0, y, W, y + h))\n       if _is_informative(crop, cfg):\n           hsh = _ahash(crop)\n           if all(_hamming(hsh, s) > cfg.dedup_hamming for s in seen_hashes):\n               seen_hashes.append(hsh)\n               tid = f\"{doc_id}__p{page}__t{seq}\"\n               tiles.append(Tile(\n                   tile_id=tid, doc_id=doc_id, source=source, kind=kind, page=page,\n                   seq=seq, y0=y, y1=y + h, title=title,\n                   path=_save_tile(crop, out_dir, tid),\n               ))\n               seq += 1\n       y += step\n   return tiles\n_JS_AUTOSCROLL = \"\"\"\nasync () => {\n await new Promise((resolve) => {\n   let y = 0;\n   const timer = setInterval(() => {\n     window.scrollBy(0, 800);\n     y += 800;\n     if (y >= document.body.scrollHeight || y > 40000) {\n       clearInterval(timer);\n       window.scrollTo(0, 0);\n       setTimeout(resolve, 250);\n     }\n   }, 40);\n });\n}\n\"\"\"\n_JS_FLATTEN = \"\"\"\n() => {\n document.querySelectorAll('*').forEach((el) => {\n   const s = getComputedStyle(el);\n   if (s.position === 'fixed' || s.position === 'sticky') el.style.position = 'absolute';\n });\n document.querySelectorAll('[role=\"dialog\"], .cookie, #cookie-banner, .cc-banner')\n   .forEach((el) => el.remove());\n}\n\"\"\"\n_CSS_CLEANUP = \"\"\"\n* { animation: none !important; transition: none !important;\n   scroll-behavior: auto !important; }\nhtml { -webkit-font-smoothing: antialiased; }\nvideo, iframe[src*=\"youtube\"] { visibility: hidden !important; }\n\"\"\"\n_UA = (\"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) \"\n      \"Chrome/124.0 Safari/537.36 PixelRAG-Tutorial/1.0\")\nasync def _render_urls_async(urls: List[str], cfg: Config, out_dir: Path) -> List[Tile]:\n   from playwright.async_api import async_playwright\n   from PIL import Image\n   all_tiles: List[Tile] = []\n   async with async_playwright() as pw:\n       browser = await pw.chromium.launch(headless=True, args=cfg.headless_args)\n       ctx = await browser.new_context(\n           viewport={\"width\": cfg.tile_width, \"height\": cfg.tile_height},\n           device_scale_factor=cfg.device_scale,\n           user_agent=_UA,\n           java_script_enabled=True,\n       )\n       for url in urls:\n           doc_id = _doc_id_from_source(url)\n           page = await ctx.new_page()\n           try:\n               await page.goto(url, wait_until=\"domcontentloaded\", timeout=cfg.nav_timeout_ms)\n               try:\n                   await page.wait_for_load_state(\"networkidle\", timeout=12000)\n               except Exception:\n                   pass\n               await page.evaluate(_JS_AUTOSCROLL)\n               await page.add_style_tag(content=_CSS_CLEANUP)\n               await page.evaluate(_JS_FLATTEN)\n               title = (await page.title()) or doc_id\n               height = await page.evaluate(\n                   \"() => Math.max(document.body.scrollHeight, \"\n                   \"document.documentElement.scrollHeight)\")\n               height = int(min(height, cfg.max_page_height))\n               step = max(1, cfg.tile_height - cfg.tile_overlap)\n               seen: List[int] = []\n               y, seq = 0, 0\n               while y < height and seq < cfg.max_tiles_per_doc:\n                   h = min(cfg.tile_height, height - y)\n                   if h < cfg.min_tile_height and seq > 0:\n                       break\n                   buf = await page.screenshot(\n                       full_page=True, type=\"png\",\n                       clip={\"x\": 0, \"y\": y, \"width\": cfg.tile_width, \"height\": h})\n                   img = Image.open(io.BytesIO(buf)).convert(\"RGB\")\n                   if img.size[0] != cfg.tile_width:\n                       img = img.resize((cfg.tile_width,\n                                         max(1, int(img.size[1] * cfg.tile_width / img.size[0]))))\n                   if _is_informative(img, cfg):\n                       hsh = _ahash(img)\n                       if all(_hamming(hsh, s) > cfg.dedup_hamming for s in seen):\n                           seen.append(hsh)\n                           tid = f\"{doc_id}__p0__t{seq}\"\n                           all_tiles.append(Tile(\n                               tile_id=tid, doc_id=doc_id, source=url, kind=\"web\",\n                               page=0, seq=seq, y0=y, y1=y + h, title=title,\n                               path=_save_tile(img, out_dir, tid)))\n                           seq += 1\n                   y += step\n               log.info(\"  rendered %-34s -> %2d tiles (page %dpx)\", doc_id, seq, height)\n           except Exception as exc:\n               log.warning(\"  FAILED %s (%s)\", url, type(exc).__name__)\n           finally:\n               await page.close()\n       await ctx.close()\n       await browser.close()\n   return all_tiles\ndef render_urls(urls: List[str], cfg: Config, out_dir: Path) -> List[Tile]:\n   \"\"\"Screenshot every URL into tiles; degrade to the text renderer on failure.\"\"\"\n   try:\n       tiles = run_async(_render_urls_async(urls, cfg, out_dir))\n       if tiles:\n           return tiles\n       log.warning(\"Browser produced no tiles — using text-render fallback.\")\n   except Exception as exc:\n       log.warning(\"Playwright unavailable (%s: %s) — using text-render fallback.\",\n                   type(exc).__name__, str(exc)[:160])\n   return [t for u in urls for t in render_url_as_text(u, cfg, out_dir)]\ndef _strip_html(html: str) -> str:\n   html = re.sub(r\"(?is)<(script|style|nav|footer|header|noscript).*?</\\1>\", \" \", html)\n   html = re.sub(r\"(?s)<!--.*?-->\", \" \", html)\n   html = re.sub(r\"(?i)</(p|div|h[1-6]|li|tr|br)>\", \"\\n\", html)\n   text = re.sub(r\"(?s)<[^>]+>\", \" \", html)\n   for a, b in [(\" \", \" \"), (\"&\", \"&\"), (\"<\", \"<\"), (\">\", \">\"), (\"\"\", '\"')]:\n       text = text.replace(a, b)\n   text = re.sub(r\"\\[\\d+\\]\", \"\", text)\n   text = re.sub(r\"[ \\t]+\", \" \", text)\n   return re.sub(r\"\\n{2,}\", \"\\n\", text).strip()\ndef _mono_font(size: int = 20):\n   from PIL import ImageFont\n   for cand in (\"/usr/share/fonts/truetype/dejavu/DejaVuSans.ttf\",\n                \"/usr/share/fonts/truetype/liberation/LiberationSans-Regular.ttf\"):\n       if os.path.exists(cand):\n           return ImageFont.truetype(cand, size)\n   try:\n       import matplotlib.font_manager as fm\n       return ImageFont.truetype(fm.findfont(\"DejaVu Sans\"), size)\n   except Exception:\n       return ImageFont.load_default()\ndef text_to_image(text: str, cfg: Config, title: str = \"\") -> Any:\n   \"\"\"Render plain text onto a tall white canvas — a browser-free stand-in.\"\"\"\n   from PIL import Image, ImageDraw\n   font, tfont = _mono_font(20), _mono_font(30)\n   pad, lh, wrap = 40, 30, max(20, (cfg.tile_width - 80) // 11)\n   lines: List[str] = []\n   for para in text.split(\"\\n\"):\n       para = para.strip()\n       if not para:\n           continue\n       while len(para) > wrap:\n           cut = para.rfind(\" \", 0, wrap)\n           cut = cut if cut > 0 else wrap\n           lines.append(para[:cut])\n           para = para[cut:].lstrip()\n       lines.append(para)\n   lines = lines[:900]\n   height = pad * 2 + 60 + lh * len(lines)\n   img = Image.new(\"RGB\", (cfg.tile_width, max(cfg.tile_height, height)), \"white\")\n   d = ImageDraw.Draw(img)\n   d.text((pad, pad), title[:60], font=tfont, fill=(15, 15, 15))\n   for i, ln in enumerate(lines):\n       d.text((pad, pad + 60 + i * lh), ln, font=font, fill=(35, 35, 35))\n   return img\ndef render_url_as_text(url: str, cfg: Config, out_dir: Path) -> List[Tile]:\n   import requests\n   doc_id = _doc_id_from_source(url)\n   try:\n       r = requests.get(url, timeout=30, headers={\"User-Agent\": _UA})\n       r.raise_for_status()\n       body = _strip_html(r.text)\n       m = re.search(r\"(?is)<title>(.*?)</title>\", r.text)\n       title = m.group(1).strip() if m else doc_id\n   except Exception as exc:\n       log.warning(\"  fetch failed for %s (%s)\", url, type(exc).__name__)\n       return []\n   img = text_to_image(body, cfg, title=title)\n   log.info(\"  text-rendered %-30s -> canvas %dpx\", doc_id, img.size[1])\n   return slice_image_to_tiles(img, cfg, doc_id=doc_id, source=url, kind=\"text\",\n                               page=0, out_dir=out_dir, title=title)\ndef render_pdf(pdf_path: str, cfg: Config, out_dir: Path, dpi: int = 150) -> List[Tile]:\n   import fitz\n   from PIL import Image\n   doc_id = _doc_id_from_source(pdf_path)\n   tiles: List[Tile] = []\n   with fitz.open(pdf_path) as doc:\n       title = (doc.metadata or {}).get(\"title\") or doc_id\n       n_pages = doc.page_count\n       for pno in range(n_pages):\n           pix = doc[pno].get_pixmap(dpi=dpi)\n           img = Image.frombytes(\"RGB\", (pix.width, pix.height), pix.samples)\n           tiles += slice_image_to_tiles(img, cfg, doc_id=doc_id, source=pdf_path,\n                                         kind=\"pdf\", page=pno, out_dir=out_dir,\n                                         title=title)\n   log.info(\"  rendered %-34s -> %2d tiles (%d pages)\", doc_id, len(tiles), n_pages)\n   return tiles\ndef make_synthetic_pdf(path: Path) -> str:\n   \"\"\"A tiny PDF so the tutorial always exercises the PDF path, offline or not.\"\"\"\n   import fitz\n   body = [\n       (\"PixelRAG Internal Note\", 22),\n       (\"\", 12),\n       (\"Why pixel-native retrieval?\", 16),\n       (\"Parsers are per-site glue code. A renderer is one code path for every\", 11),\n       (\"document type: HTML, PDF, scanned fax, spreadsheet export, dashboard.\", 11),\n       (\"\", 11),\n       (\"Tiling policy\", 16),\n       (\"Tiles are 1024x1024 with 128px of vertical overlap. Overlap keeps a\", 11),\n       (\"sentence or table row from being split across two embeddings, which is\", 11),\n       (\"the single biggest source of recall loss in naive screenshot pipelines.\", 11),\n       (\"\", 11),\n       (\"Serving\", 16),\n       (\"FAISS inner-product over L2-normalised vectors equals cosine similarity.\", 11),\n       (\"Tile scores are max-pooled per document so one strong tile can surface\", 11),\n       (\"a long page, mirroring late-interaction retrieval behaviour.\", 11),\n       (\"\", 11),\n       (\"The mitochondria reference is a joke; the overlap advice is not.\", 11),\n   ]\n   doc = fitz.open()\n   page = doc.new_page()\n   y = 72\n   for line, size in body:\n       page.insert_text((72, y), line, fontsize=size, fontname=\"helv\")\n       y += size + 8\n   doc.save(str(path))\n   doc.close()\n   return str(path)\n```\n\nWe create the document-rendering layer that converts web pages, text content, and PDF files into structured image tiles. We capture web pages with Playwright, clean distracting page elements, apply overlapping vertical slicing, and remove blank or duplicate tiles. We also provide text-rendering and synthetic-PDF fallbacks so the pipeline continues to operate when browser rendering or external content is unavailable.\n\n``` php\ndef ocr_tiles(tiles: List[Tile], cfg: Config) -> None:\n   if not cfg.use_ocr_hybrid:\n       return\n   try:\n       import pytesseract\n       from PIL import Image\n   except Exception:\n       log.warning(\"pytesseract missing -> dense-only retrieval.\")\n       cfg.use_ocr_hybrid = False\n       return\n   from tqdm.auto import tqdm\n   t0 = time.time()\n   for t in tqdm(tiles, desc=\"OCR\", unit=\"tile\"):\n       try:\n           raw = pytesseract.image_to_string(Image.open(t.path), config=\"--psm 6\")\n           t.ocr_text = re.sub(r\"\\s+\", \" \", raw).strip()[:4000]\n       except Exception:\n           t.ocr_text = \"\"\n   log.info(\"OCR over %d tiles in %.1fs\", len(tiles), time.time() - t0)\ndef torch_device() -> str:\n   import torch\n   if torch.cuda.is_available():\n       return \"cuda\"\n   if getattr(torch.backends, \"mps\", None) and torch.backends.mps.is_available():\n       return \"mps\"\n   return \"cpu\"\nclass DualEncoderBackend:\n   \"\"\"\n   SigLIP / CLIP image-text dual encoder.\n   Honest caveat: these encoders were trained on natural images with short\n   captions (64-77 token text towers). They understand a screenshot's *gist*\n   — layout, topic, figures — not its fine print. That is exactly why upstream\n   PixelRAG uses Qwen3-VL-Embedding-2B plus a LoRA trained on screenshots.\n   Sections §9 (OCR hybrid) and §10 (adapter) exist to close part of the gap\n   on hardware that can't host a 2B VLM.\n   \"\"\"\n   def __init__(self, cfg: Config):\n       import torch\n       from transformers import AutoModel, AutoProcessor\n       self.cfg = cfg\n       self.device = torch_device()\n       self.dtype = torch.float16 if self.device == \"cuda\" else torch.float32\n       self.model_id = cfg.model_id if cfg.backend != \"clip\" else \"openai/clip-vit-base-patch32\"\n       log.info(\"Loading embedding model %s on %s (%s)\", self.model_id, self.device,\n                str(self.dtype).replace(\"torch.\", \"\"))\n       self.processor = AutoProcessor.from_pretrained(self.model_id)\n       self.model = AutoModel.from_pretrained(self.model_id, torch_dtype=self.dtype)\n       self.model.to(self.device).eval()\n       self.is_siglip = \"siglip\" in self.model_id.lower()\n       self.dim = int(getattr(self.model.config, \"projection_dim\", 0) or\n                      getattr(self.model.config.text_config, \"hidden_size\", 512))\n       self.name = f\"{'siglip' if self.is_siglip else 'clip'}:{self.model_id}\"\n   @staticmethod\n   def _l2(x):\n       import numpy as np\n       n = np.linalg.norm(x, axis=-1, keepdims=True)\n       return (x / np.clip(n, 1e-12, None)).astype(\"float32\")\n   def embed_images(self, images: List[Any], bs: Optional[int] = None):\n       import torch, numpy as np\n       from tqdm.auto import tqdm\n       bs = bs or self.cfg.embed_batch_size\n       out = []\n       for i in tqdm(range(0, len(images), bs), desc=\"embed:image\", unit=\"batch\"):\n           batch = images[i:i + bs]\n           inputs = self.processor(images=batch, return_tensors=\"pt\")\n           inputs = {k: v.to(self.device, self.dtype if v.is_floating_point() else v.dtype)\n                     for k, v in inputs.items()}\n           with torch.no_grad():\n               feats = self.model.get_image_features(**inputs)\n           out.append(feats.float().cpu().numpy())\n       return self._l2(np.concatenate(out, 0)) if out else np.zeros((0, self.dim), \"float32\")\n   def embed_texts(self, texts: List[str], bs: Optional[int] = None):\n       import torch, numpy as np\n       bs = bs or max(16, self.cfg.embed_batch_size)\n       out = []\n       for i in range(0, len(texts), bs):\n           batch = [t if t.strip() else \" \" for t in texts[i:i + bs]]\n           kw = dict(text=batch, return_tensors=\"pt\", truncation=True)\n           kw.update(padding=\"max_length\", max_length=64) if self.is_siglip else kw.update(padding=True)\n           inputs = self.processor(**kw)\n           inputs = {k: v.to(self.device) for k, v in inputs.items()}\n           with torch.no_grad():\n               feats = self.model.get_text_features(**inputs)\n           out.append(feats.float().cpu().numpy())\n       return self._l2(np.concatenate(out, 0)) if out else np.zeros((0, self.dim), \"float32\")\nclass Qwen3VLEmbeddingBackend:\n   \"\"\"\n   Opt-in backend matching upstream (Qwen/Qwen3-VL-Embedding-2B).\n   Needs a recent transformers (>= 4.57) and ~8 GB of VRAM in fp16. It embeds\n   text and images into one space by mean-pooling the last hidden state of a\n   VLM prompt, which is why it handles dense screenshot text far better than\n   a CLIP-style tower.\n   \"\"\"\n   def __init__(self, cfg: Config):\n       import torch\n       from transformers import AutoModel, AutoProcessor\n       self.cfg = cfg\n       self.device = torch_device()\n       self.dtype = torch.float16 if self.device == \"cuda\" else torch.float32\n       mid = cfg.qwen_model_id\n       log.info(\"Loading %s (this is a large download)...\", mid)\n       self.processor = AutoProcessor.from_pretrained(mid, trust_remote_code=True)\n       self.model = AutoModel.from_pretrained(mid, torch_dtype=self.dtype,\n                                              trust_remote_code=True).to(self.device).eval()\n       self.dim = int(self.model.config.hidden_size)\n       self.name = f\"qwen3vl:{mid}\"\n   def _pool(self, hidden, mask):\n       import torch\n       m = mask.unsqueeze(-1).to(hidden.dtype)\n       return (hidden * m).sum(1) / m.sum(1).clamp(min=1e-6)\n   def _encode(self, **proc_kwargs):\n       import torch, numpy as np\n       inputs = self.processor(return_tensors=\"pt\", padding=True, **proc_kwargs)\n       inputs = {k: (v.to(self.device) if hasattr(v, \"to\") else v) for k, v in inputs.items()}\n       with torch.no_grad():\n           out = self.model(**inputs, output_hidden_states=True)\n       hidden = out.hidden_states[-1] if getattr(out, \"hidden_states\", None) is not None \\\n           else out.last_hidden_state\n       vec = self._pool(hidden, inputs[\"attention_mask\"]).float().cpu().numpy()\n       return DualEncoderBackend._l2(vec)\n   def embed_images(self, images: List[Any], bs: Optional[int] = None):\n       import numpy as np\n       from tqdm.auto import tqdm\n       bs = bs or max(1, self.cfg.embed_batch_size // 4)\n       chunks = []\n       for i in tqdm(range(0, len(images), bs), desc=\"embed:image\", unit=\"batch\"):\n           batch = images[i:i + bs]\n           prompt = [\"Describe this document screenshot for retrieval.\"] * len(batch)\n           chunks.append(self._encode(text=prompt, images=batch))\n       return np.concatenate(chunks, 0)\n   def embed_texts(self, texts: List[str], bs: Optional[int] = None):\n       import numpy as np\n       bs = bs or 8\n       chunks = [self._encode(text=[f\"Query: {t}\" for t in texts[i:i + bs]])\n                 for i in range(0, len(texts), bs)]\n       return np.concatenate(chunks, 0) if chunks else np.zeros((0, self.dim), \"float32\")\ndef build_backend(cfg: Config):\n   if cfg.backend == \"qwen3vl\":\n       try:\n           return Qwen3VLEmbeddingBackend(cfg)\n       except Exception as exc:\n           log.warning(\"Qwen3-VL backend failed (%s: %s) -> falling back to SigLIP.\",\n                       type(exc).__name__, str(exc)[:200])\n           cfg.backend = \"siglip\"\n   return DualEncoderBackend(cfg)\ndef embed_tiles(tiles: List[Tile], backend, cfg: Config):\n   from PIL import Image\n   import numpy as np\n   vecs = []\n   bs = cfg.embed_batch_size\n   for i in range(0, len(tiles), bs):\n       imgs = [Image.open(t.path).convert(\"RGB\") for t in tiles[i:i + bs]]\n       vecs.append(backend.embed_images(imgs, bs=bs))\n       for im in imgs:\n           im.close()\n   return np.concatenate(vecs, 0) if vecs else np.zeros((0, backend.dim), \"float32\")\n```\n\nWe extract OCR text from each rendered tile to support sparse retrieval and automatic training-pair generation. We implement SigLIP, CLIP, and Qwen3-VL embedding backends that place text queries and document screenshots within a shared vector space. We then process the tile images in batches and generate normalized embeddings that are ready for similarity indexing.\n\n```\nclass PixelIndex:\n   \"\"\"\n   Inner-product FAISS index over L2-normalised vectors (== cosine similarity).\n   Flat below `ivf_threshold` vectors (exact, no training); IVF above it\n   (sub-linear, needs training + nprobe tuning). Raw vectors are also kept in\n   memory so §10 can re-project them after adapter training without re-running\n   the encoder.\n   \"\"\"\n   def __init__(self, dim: int, cfg: Config):\n       self.dim, self.cfg = dim, cfg\n       self.index = None\n       self.metas: List[Dict[str, Any]] = []\n       self.vectors = None\n       self._bm25 = None\n       self._bm25_corpus: List[List[str]] = []\n   def build(self, vectors, tiles: List[Tile]) -> \"PixelIndex\":\n       import faiss, numpy as np\n       vectors = np.ascontiguousarray(vectors.astype(\"float32\"))\n       n = vectors.shape[0]\n       if n == 0:\n           raise RuntimeError(\"No vectors to index — did rendering produce any tiles?\")\n       if n >= self.cfg.ivf_threshold:\n           nlist = max(4, min(4096, int(4 * math.sqrt(n))))\n           quant = faiss.IndexFlatIP(self.dim)\n           base = faiss.IndexIVFFlat(quant, self.dim, nlist, faiss.METRIC_INNER_PRODUCT)\n           base.train(vectors)\n           base.nprobe = self.cfg.ivf_nprobe\n           log.info(\"FAISS IndexIVFFlat  n=%d nlist=%d nprobe=%d\", n, nlist, base.nprobe)\n       else:\n           base = faiss.IndexFlatIP(self.dim)\n           log.info(\"FAISS IndexFlatIP   n=%d dim=%d (exact search)\", n, self.dim)\n       self.index = faiss.IndexIDMap2(base)\n       self.index.add_with_ids(vectors, np.arange(n).astype(\"int64\"))\n       self.vectors = vectors\n       self.metas = [asdict(t) for t in tiles]\n       self._fit_bm25()\n       return self\n   def _fit_bm25(self) -> None:\n       if not self.cfg.use_ocr_hybrid:\n           return\n       try:\n           from rank_bm25 import BM25Okapi\n       except Exception:\n           return\n       self._bm25_corpus = [re.findall(r\"[a-z0-9]+\", (m.get(\"ocr_text\", \"\") + \" \" +\n                                                      m.get(\"title\", \"\")).lower())\n                            for m in self.metas]\n       if any(self._bm25_corpus):\n           self._bm25 = BM25Okapi([c or [\"_\"] for c in self._bm25_corpus])\n           log.info(\"BM25 fitted over OCR sidecar (%d docs)\", len(self._bm25_corpus))\n   def search_dense(self, qvecs, k: int):\n       import numpy as np\n       scores, ids = self.index.search(np.ascontiguousarray(qvecs.astype(\"float32\")), k)\n       return scores, ids\n   def search_sparse(self, query: str, k: int) -> List[Tuple[int, float]]:\n       if self._bm25 is None:\n           return []\n       import numpy as np\n       toks = re.findall(r\"[a-z0-9]+\", query.lower())\n       if not toks:\n           return []\n       s = np.asarray(self._bm25.get_scores(toks))\n       top = np.argsort(-s)[:k]\n       return [(int(i), float(s[i])) for i in top if s[i] > 0]\n   def save(self, out_dir: str) -> None:\n       import faiss, numpy as np\n       p = Path(out_dir)\n       p.mkdir(parents=True, exist_ok=True)\n       faiss.write_index(self.index, str(p / \"tiles.faiss\"))\n       np.save(p / \"vectors.npy\", self.vectors)\n       (p / \"metas.jsonl\").write_text(\"\\n\".join(json.dumps(m) for m in self.metas))\n       (p / \"manifest.json\").write_text(json.dumps(\n           {\"dim\": self.dim, \"n\": len(self.metas), \"created\": time.time(),\n            \"config\": asdict(self.cfg)}, indent=2))\n       log.info(\"Index saved to %s (%d tiles)\", p.resolve(), len(self.metas))\n   @classmethod\n   def load(cls, out_dir: str, cfg: Config) -> \"PixelIndex\":\n       import faiss, numpy as np\n       p = Path(out_dir)\n       man = json.loads((p / \"manifest.json\").read_text())\n       obj = cls(man[\"dim\"], cfg)\n       obj.index = faiss.read_index(str(p / \"tiles.faiss\"))\n       obj.vectors = np.load(p / \"vectors.npy\")\n       obj.metas = [json.loads(l) for l in (p / \"metas.jsonl\").read_text().splitlines() if l]\n       obj._fit_bm25()\n       return obj\n   def reproject(self, new_vectors) -> None:\n       \"\"\"Swap in re-embedded vectors (used after adapter training in §10).\"\"\"\n       tiles = [Tile(**m) for m in self.metas]\n       self.build(new_vectors, tiles)\ndef build_index(cfg: Config) -> Tuple[PixelIndex, Any, List[Tile]]:\n   work = Path(cfg.work_dir)\n   tiles_dir = work / \"tiles\"\n   tiles_dir.mkdir(parents=True, exist_ok=True)\n   log.info(\"=\" * 74)\n   log.info(\"STAGE 1/4  RENDER  (documents -> image tiles)\")\n   log.info(\"=\" * 74)\n   tiles: List[Tile] = render_urls(cfg.urls, cfg, tiles_dir)\n   if cfg.include_synthetic_pdf:\n       pdf_path = make_synthetic_pdf(work / \"pixelrag_note.pdf\")\n       tiles += render_pdf(pdf_path, cfg, tiles_dir)\n   if not tiles:\n       raise RuntimeError(\"Rendering produced zero tiles. Check network access.\")\n   log.info(\"Total tiles: %d across %d documents\",\n            len(tiles), len({t.doc_id for t in tiles}))\n   log.info(\"=\" * 74)\n   log.info(\"STAGE 2/4  OCR SIDECAR  (for hybrid retrieval + pair mining)\")\n   log.info(\"=\" * 74)\n   ocr_tiles(tiles, cfg)\n   log.info(\"=\" * 74)\n   log.info(\"STAGE 3/4  EMBED  (tiles -> vectors)\")\n   log.info(\"=\" * 74)\n   backend = build_backend(cfg)\n   t0 = time.time()\n   vecs = embed_tiles(tiles, backend, cfg)\n   log.info(\"Embedded %d tiles -> %s in %.1fs (%.2f tiles/s)\",\n            vecs.shape[0], vecs.shape, time.time() - t0,\n            vecs.shape[0] / max(1e-6, time.time() - t0))\n   log.info(\"=\" * 74)\n   log.info(\"STAGE 4/4  INDEX  (vectors -> FAISS)\")\n   log.info(\"=\" * 74)\n   index = PixelIndex(vecs.shape[1], cfg).build(vecs, tiles)\n   index.save(cfg.index_dir)\n   return index, backend, tiles\n```\n\nWe construct the PixelIndex class and store the normalized tile embeddings inside a FAISS inner-product index. We support exact flat search for smaller datasets, IVF-based search for larger collections, BM25 indexing over OCR text, and persistent storage of vectors and metadata. We also orchestrate the complete indexing pipeline by rendering documents, running OCR, generating embeddings, building the index, and saving all outputs to disk.\n\n``` python\ndef search(query: str, index: PixelIndex, backend, cfg: Config,\n          n_docs: Optional[int] = None) -> List[Dict[str, Any]]:\n   import numpy as np\n   n_docs = n_docs or cfg.n_docs\n   k = min(cfg.top_k_tiles, len(index.metas))\n   qv = backend.embed_texts([query])\n   dscores, dids = index.search_dense(qv, k)\n   dense = [(int(i), float(s)) for i, s in zip(dids[0], dscores[0]) if i >= 0]\n   fused: Dict[int, float] = {}\n   for rank, (tid, _) in enumerate(dense):\n       fused[tid] = fused.get(tid, 0.0) + cfg.dense_weight / (cfg.rrf_k + rank + 1)\n   sparse = index.search_sparse(query, k) if cfg.use_ocr_hybrid else []\n   for rank, (tid, _) in enumerate(sparse):\n       fused[tid] = fused.get(tid, 0.0) + cfg.sparse_weight / (cfg.rrf_k + rank + 1)\n   dense_lookup = dict(dense)\n   tile_hits = sorted(fused.items(), key=lambda kv: -kv[1])\n   per_doc: Dict[str, Dict[str, Any]] = {}\n   for tid, fscore in tile_hits:\n       m = index.metas[tid]\n       d = per_doc.setdefault(m[\"doc_id\"], {\n           \"doc_id\": m[\"doc_id\"], \"title\": m.get(\"title\") or m[\"doc_id\"],\n           \"source\": m[\"source\"], \"kind\": m[\"kind\"], \"score\": 0.0,\n           \"dense_score\": 0.0, \"tiles\": [],\n       })\n       d[\"score\"] = max(d[\"score\"], fscore)\n       d[\"dense_score\"] = max(d[\"dense_score\"], dense_lookup.get(tid, 0.0))\n       if len(d[\"tiles\"]) < 3:\n           d[\"tiles\"].append({\n               \"tile_id\": m[\"tile_id\"], \"path\": m[\"path\"], \"seq\": m[\"seq\"],\n               \"page\": m[\"page\"], \"y0\": m[\"y0\"], \"y1\": m[\"y1\"],\n               \"rrf\": round(fscore, 6),\n               \"cosine\": round(dense_lookup.get(tid, 0.0), 4),\n               \"snippet\": (m.get(\"ocr_text\", \"\") or \"\")[:220],\n           })\n   return sorted(per_doc.values(), key=lambda d: -d[\"score\"])[:n_docs]\ndef pretty_print(query: str, results: List[Dict[str, Any]]) -> None:\n   print(f\"\\n\\033[1mQ: {query}\\033[0m\")\n   if not results:\n       print(\"   (no hits)\")\n       return\n   for i, r in enumerate(results, 1):\n       print(f\"  {i}. [{r['score']:.4f} rrf | {r['dense_score']:.3f} cos] \"\n             f\"{r['title'][:64]}  ({r['kind']})\")\n       top = r[\"tiles\"][0]\n       print(f\"       tile {top['tile_id']}  y={top['y0']}-{top['y1']}\")\n       if top[\"snippet\"]:\n           print(f\"       \\033[2m{top['snippet'][:150]}...\\033[0m\")\nclass SearchServer:\n   \"\"\"FastAPI + uvicorn on a background thread, mirroring upstream's POST /search.\"\"\"\n   def __init__(self, index: PixelIndex, backend, cfg: Config):\n       from fastapi import FastAPI\n       from pydantic import BaseModel\n       class Query(BaseModel):\n           text: str\n       class SearchRequest(BaseModel):\n           queries: List[Query]\n           n_docs: int = cfg.n_docs\n       app = FastAPI(title=\"PixelRAG (tutorial)\", version=\"1.0\")\n       @app.get(\"/health\")\n       def health():\n           return {\"status\": \"ok\", \"tiles\": len(index.metas),\n                   \"docs\": len({m[\"doc_id\"] for m in index.metas}),\n                   \"backend\": getattr(backend, \"name\", \"unknown\")}\n       @app.post(\"/search\")\n       def do_search(req: SearchRequest):\n           return {\"results\": [\n               {\"query\": q.text, \"docs\": search(q.text, index, backend, cfg, req.n_docs)}\n               for q in req.queries]}\n       self.app, self.cfg = app, cfg\n       self.thread: Optional[threading.Thread] = None\n       self.server = None\n   def start(self) -> bool:\n       import uvicorn, requests\n       config = uvicorn.Config(self.app, host=\"127.0.0.1\", port=self.cfg.server_port,\n                               log_level=\"error\")\n       self.server = uvicorn.Server(config)\n       self.thread = threading.Thread(target=self.server.run, daemon=True)\n       self.thread.start()\n       for _ in range(40):\n           time.sleep(0.25)\n           try:\n               if requests.get(f\"http://127.0.0.1:{self.cfg.server_port}/health\",\n                               timeout=2).ok:\n                   log.info(\"Search API live on http://127.0.0.1:%d\", self.cfg.server_port)\n                   return True\n           except Exception:\n               continue\n       log.warning(\"Server did not come up in time.\")\n       return False\n   def stop(self) -> None:\n       if self.server:\n           self.server.should_exit = True\n       if self.thread:\n           self.thread.join(timeout=5)\n```\n\nWe implement hybrid retrieval by combining dense vector rankings and OCR-based BM25 rankings through reciprocal rank fusion. We aggregate matching tiles into document-level results while retaining the strongest evidence tiles, similarity scores, and OCR snippets for inspection. We also expose the retrieval system through a FastAPI server with health and search endpoints that run on a background Uvicorn thread.\n\n``` python\ndef evaluate(index: PixelIndex, backend, cfg: Config,\n            queries: List[Tuple[str, str]] = EVAL_QUERIES,\n            label: str = \"eval\", quiet: bool = False) -> Dict[str, float]:\n   ranks: List[Optional[int]] = []\n   for q, want in queries:\n       docs = search(q, index, backend, cfg, n_docs=10)\n       hit = next((i for i, d in enumerate(docs) if want.lower() in d[\"doc_id\"].lower()), None)\n       ranks.append(hit)\n       if not quiet:\n           got = docs[0][\"doc_id\"] if docs else \"-\"\n           mark = \"OK \" if hit == 0 else (f\"@{hit + 1}\" if hit is not None else \"MISS\")\n           print(f\"  [{mark:>4}] {q[:56]:<58} -> {got[:32]}\")\n   n = len(ranks)\n   m = {\n       \"recall@1\": sum(r == 0 for r in ranks) / n,\n       \"recall@3\": sum(r is not None and r < 3 for r in ranks) / n,\n       \"recall@5\": sum(r is not None and r < 5 for r in ranks) / n,\n       \"mrr\": sum(1.0 / (r + 1) for r in ranks if r is not None) / n,\n   }\n   print(f\"  \\033[1m{label}\\033[0m  R@1={m['recall@1']:.2f}  R@3={m['recall@3']:.2f}  \"\n         f\"R@5={m['recall@5']:.2f}  MRR={m['mrr']:.3f}\")\n   return m\ndef mine_training_pairs(tiles: List[Tile], max_per_tile: int = 2) -> List[Tuple[str, int]]:\n   \"\"\"Weak supervision: pseudo-queries from a tile's own OCR text / title.\"\"\"\n   import random\n   rng = random.Random(0)\n   pairs: List[Tuple[str, int]] = []\n   for idx, t in enumerate(tiles):\n       text = (t.ocr_text or \"\").strip()\n       cands: List[str] = []\n       if len(text) > 80:\n           words = text.split()\n           for _ in range(max_per_tile):\n               if len(words) <= 14:\n                   break\n               s = rng.randint(0, len(words) - 14)\n               span = \" \".join(words[s:s + rng.randint(8, 14)])\n               if len(span) > 30:\n                   cands.append(span)\n       if t.title:\n           cands.append(t.title)\n       for c in cands[:max_per_tile]:\n           pairs.append((c, idx))\n   return pairs\nclass ResidualAdapter:\n   \"\"\"Shared two-layer residual MLP applied to both query and tile vectors.\"\"\"\n   def __init__(self, dim: int, hidden: int = 512, device: str = \"cpu\"):\n       import torch\n       import torch.nn as nn\n       self.device = device\n       self.net = nn.Sequential(\n           nn.Linear(dim, hidden), nn.GELU(), nn.Linear(hidden, dim)\n       ).to(device)\n       for p in self.net[-1].parameters():\n           torch.nn.init.zeros_(p)\n       self.logit_scale = torch.nn.Parameter(torch.tensor(2.996, device=device))\n       self.dim = dim\n   def forward_t(self, x):\n       import torch\n       y = x + self.net(x)\n       return torch.nn.functional.normalize(y, dim=-1)\n   def apply_np(self, arr):\n       import torch, numpy as np\n       with torch.no_grad():\n           t = torch.from_numpy(np.ascontiguousarray(arr.astype(\"float32\"))).to(self.device)\n           return self.forward_t(t).cpu().numpy().astype(\"float32\")\ndef train_adapter(index: PixelIndex, backend, tiles: List[Tile], cfg: Config,\n                 epochs: int = 12, batch: int = 24, lr: float = 1e-4):\n   import torch, numpy as np\n   pairs = mine_training_pairs(tiles)\n   if len(pairs) < 32:\n       log.warning(\"Only %d mined pairs — skipping adapter training \"\n                   \"(enable OCR or add documents).\", len(pairs))\n       return None\n   log.info(\"Mined %d (pseudo-query, tile) pairs from %d tiles\", len(pairs), len(tiles))\n   q_texts = [p[0] for p in pairs]\n   t_idx = np.array([p[1] for p in pairs], dtype=\"int64\")\n   log.info(\"Pre-embedding pseudo-queries (frozen encoder, done once)...\")\n   Q = torch.from_numpy(backend.embed_texts(q_texts))\n   V = torch.from_numpy(index.vectors)\n   device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n   ad = ResidualAdapter(index.dim, device=device)\n   Q, V = Q.to(device), V.to(device)\n   opt = torch.optim.AdamW(list(ad.net.parameters()) + [ad.logit_scale], lr=lr, weight_decay=1e-2)\n   n = len(pairs)\n   doc_ids = torch.from_numpy(t_idx).to(device)\n   for ep in range(epochs):\n       perm = torch.randperm(n, device=device)\n       total, steps = 0.0, 0\n       for i in range(0, n, batch):\n           sel = perm[i:i + batch]\n           if sel.numel() < 4:\n               continue\n           qb = ad.forward_t(Q[sel])\n           docs = doc_ids[sel]\n           vb = ad.forward_t(V[docs])\n           logits = ad.logit_scale.exp().clamp(max=100) * qb @ vb.T\n           same = docs[:, None] == docs[None, :]\n           eye = torch.eye(len(sel), dtype=torch.bool, device=device)\n           logits = logits.masked_fill(same & ~eye, float(\"-inf\"))\n           labels = torch.arange(len(sel), device=device)\n           loss = 0.5 * (torch.nn.functional.cross_entropy(logits, labels) +\n                         torch.nn.functional.cross_entropy(logits.T, labels))\n           opt.zero_grad()\n           loss.backward()\n           torch.nn.utils.clip_grad_norm_(ad.net.parameters(), 1.0)\n           opt.step()\n           total += loss.detach().item()\n           steps += 1\n       if ep % 3 == 0 or ep == epochs - 1:\n           log.info(\"  epoch %2d/%d  InfoNCE loss %.4f\", ep + 1, epochs, total / max(steps, 1))\n   return ad\nclass AdaptedBackend:\n   \"\"\"Wraps a frozen backend so queries pass through the trained adapter.\"\"\"\n   def __init__(self, backend, adapter: ResidualAdapter):\n       self.backend, self.adapter = backend, adapter\n       self.dim = backend.dim\n       self.name = f\"{getattr(backend, 'name', 'backend')}+adapter\"\n   def embed_texts(self, texts, bs=None):\n       return self.adapter.apply_np(self.backend.embed_texts(texts, bs=bs))\n   def embed_images(self, images, bs=None):\n       return self.adapter.apply_np(self.backend.embed_images(images, bs=bs))\ndef answer_with_vlm(query: str, results: List[Dict[str, Any]], cfg: Config,\n                   max_tiles: int = 3) -> str:\n   \"\"\"\n   Retrieval returns pixels, so generation must accept pixels. Any VLM works;\n   Qwen2.5-VL-3B is a reasonable Colab-sized default (~7 GB download).\n   \"\"\"\n   try:\n       import torch\n       from PIL import Image\n       from transformers import AutoProcessor, AutoModelForImageTextToText\n   except Exception as exc:\n       return f\"[VLM unavailable: {exc}]\"\n   paths = [t[\"path\"] for r in results for t in r[\"tiles\"]][:max_tiles]\n   if not paths:\n       return \"[no retrieved tiles]\"\n   log.info(\"Loading VLM %s ...\", cfg.vlm_model_id)\n   proc = AutoProcessor.from_pretrained(cfg.vlm_model_id)\n   model = AutoModelForImageTextToText.from_pretrained(\n       cfg.vlm_model_id,\n       torch_dtype=torch.float16 if torch.cuda.is_available() else torch.float32,\n       device_map=\"auto\")\n   images = [Image.open(p).convert(\"RGB\") for p in paths]\n   content = [{\"type\": \"image\"} for _ in images] + [{\"type\": \"text\", \"text\":\n       f\"These are screenshots retrieved for the question. Answer using only what \"\n       f\"is visible, and say so if the answer is not shown.\\n\\nQuestion: {query}\"}]\n   prompt = proc.apply_chat_template([{\"role\": \"user\", \"content\": content}],\n                                     add_generation_prompt=True, tokenize=False)\n   inputs = proc(text=[prompt], images=images, return_tensors=\"pt\").to(model.device)\n   with torch.no_grad():\n       out = model.generate(**inputs, max_new_tokens=256, do_sample=False)\n   text = proc.batch_decode(out[:, inputs[\"input_ids\"].shape[1]:],\n                            skip_special_tokens=True)[0]\n   return text.strip()\ndef show_results(query: str, results: List[Dict[str, Any]], max_tiles: int = 3) -> None:\n   try:\n       import matplotlib.pyplot as plt\n       from PIL import Image\n   except Exception:\n       return\n   tiles = [(r, t) for r in results for t in r[\"tiles\"][:1]][:max_tiles]\n   if not tiles:\n       return\n   fig, axes = plt.subplots(1, len(tiles), figsize=(5 * len(tiles), 6))\n   axes = [axes] if len(tiles) == 1 else list(axes)\n   for ax, (r, t) in zip(axes, tiles):\n       ax.imshow(Image.open(t[\"path\"]))\n       ax.set_title(f\"{r['title'][:34]}\\nrrf={r['score']:.4f} cos={t['cosine']:.3f}\",\n                    fontsize=9)\n       ax.axis(\"off\")\n   fig.suptitle(f\"Q: {query}\", fontsize=12)\n   plt.tight_layout()\n   plt.show()\n```\n\nWe evaluate retrieval quality using Recall@1, Recall@3, Recall@5, and mean reciprocal rank across a small benchmark. We mine pseudo-query and tile pairs from OCR content, train a residual contrastive adapter, and apply the learned transformation to both query and image embeddings. We also support grounded answer generation with a vision-language model and visualize the highest-ranked screenshot tiles with their retrieval scores.\n\n``` php\ndef main(cfg: Config = CFG) -> Dict[str, Any]:\n   Path(cfg.work_dir).mkdir(parents=True, exist_ok=True)\n   ensure_deps(cfg)\n   import numpy as np\n   np.random.seed(cfg.seed)\n   banner = \"\"\"\n   ██████╗ ██╗██╗  ██╗███████╗██╗     ██████╗  █████╗  ██████╗\n   ██╔══██╗██║╚██╗██╔╝██╔════╝██║     ██╔══██╗██╔══██╗██╔════╝\n   ██████╔╝██║ ╚███╔╝ █████╗  ██║     ██████╔╝███████║██║  ███╗\n   ██╔═══╝ ██║ ██╔██╗ ██╔══╝  ██║     ██╔══██╗██╔══██║██║   ██║\n   ██║     ██║██╔╝ ██╗███████╗███████╗██║  ██║██║  ██║╚██████╔╝\n   ╚═╝     ╚═╝╚═╝  ╚═╝╚══════╝╚══════╝╚═╝  ╚═╝╚═╝  ╚═╝ ╚═════╝\n       pixel-native retrieval:  render -> tile -> embed -> FAISS -> serve\n   \"\"\"\n   print(banner)\n   index, backend, tiles = build_index(cfg)\n   print(\"\\n\" + \"=\" * 74)\n   print(\"SEARCH DEMO — text query against a pixel index\")\n   print(\"=\" * 74)\n   demo_queries = [\n       \"how do plants turn light into sugar\",\n       \"what does a vector database store\",\n       \"why use overlapping tiles when screenshotting a page\",\n   ]\n   for q in demo_queries:\n       res = search(q, index, backend, cfg)\n       pretty_print(q, res)\n       if cfg.show_plots:\n           show_results(q, res)\n   metrics_before = None\n   if cfg.enable_eval:\n       print(\"\\n\" + \"=\" * 74)\n       print(\"EVALUATION — baseline\")\n       print(\"=\" * 74)\n       metrics_before = evaluate(index, backend, cfg, label=\"baseline\")\n       if cfg.use_ocr_hybrid:\n           cfg.use_ocr_hybrid = False\n           print(\"\\n  -- ablation: dense only (OCR/BM25 disabled) --\")\n           evaluate(index, backend, cfg, label=\"dense-only\", quiet=True)\n           cfg.use_ocr_hybrid = True\n   active_backend = backend\n   if cfg.enable_adapter_train:\n       print(\"\\n\" + \"=\" * 74)\n       print(\"ADAPTER TRAINING — contrastive head over frozen embeddings\")\n       print(\"=\" * 74)\n       adapter = train_adapter(index, backend, tiles, cfg)\n       if adapter is not None:\n           index.reproject(adapter.apply_np(index.vectors))\n           active_backend = AdaptedBackend(backend, adapter)\n           if cfg.enable_eval:\n               print(\"\\n  -- after adapter --\")\n               after = evaluate(index, active_backend, cfg, label=\"adapted\")\n               if metrics_before:\n                   d = after[\"mrr\"] - metrics_before[\"mrr\"]\n                   print(f\"  MRR delta: {d:+.3f} \"\n                         f\"({'improved' if d > 0 else 'no gain — expected on a corpus this small'})\")\n   server = None\n   if cfg.enable_server:\n       print(\"\\n\" + \"=\" * 74)\n       print(\"SERVE — FastAPI, upstream-compatible POST /search\")\n       print(\"=\" * 74)\n       server = SearchServer(index, active_backend, cfg)\n       if server.start():\n           import requests\n           r = requests.post(f\"http://127.0.0.1:{cfg.server_port}/search\",\n                             json={\"queries\": [{\"text\": \"what is retrieval augmented generation\"}],\n                                   \"n_docs\": 3}, timeout=120)\n           payload = r.json()\n           for res in payload[\"results\"]:\n               print(f\"\\n  POST /search  query={res['query']!r}\")\n               for d in res[\"docs\"]:\n                   print(f\"    - {d['score']:.4f}  {d['title'][:56]}  <{d['source'][:48]}>\")\n           print(\"\\n  Equivalent curl:\")\n           print(f\"    curl -X POST http://127.0.0.1:{cfg.server_port}/search \\\\\")\n           print(\"      -H 'Content-Type: application/json' \\\\\")\n           print(\"      -d '{\\\"queries\\\":[{\\\"text\\\":\\\"capital of india\\\"}],\\\"n_docs\\\":3}'\")\n   if cfg.enable_vlm_answer:\n       print(\"\\n\" + \"=\" * 74)\n       print(\"GENERATION — answering from retrieved pixels\")\n       print(\"=\" * 74)\n       q = \"According to the retrieved screenshots, what is photosynthesis?\"\n       res = search(q, index, active_backend, cfg, n_docs=2)\n       print(answer_with_vlm(q, res, cfg))\n   else:\n       print(\"\\n[i] Set CFG.enable_vlm_answer = True (GPU) to generate answers \"\n             \"directly from the retrieved tiles.\")\n   n_docs = len({m['doc_id'] for m in index.metas})\n   print(\"\\n\" + \"=\" * 74)\n   print(\"DONE\")\n   print(\"=\" * 74)\n   print(f\"  tiles indexed : {len(index.metas)} across {n_docs} documents\")\n   print(f\"  embedding dim : {index.dim}   backend: {getattr(active_backend, 'name', '?')}\")\n   print(f\"  index on disk : {Path(cfg.index_dir).resolve()}\")\n   print(f\"  tiles on disk : {Path(cfg.work_dir).resolve() / 'tiles'}\")\n   print(\"\"\"\n Try next:\n   * CFG.urls  -> point at your own pages, then re-run main()\n   * CFG.backend = \"qwen3vl\"  -> upstream's Qwen3-VL-Embedding-2B (needs a big GPU)\n   * CFG.device_scale = 2.0   -> sharper tiles, better small-text retrieval\n   * CFG.tile_overlap = 256   -> higher recall on prose, more vectors to store\n   * render_pdf(\"/content/your.pdf\", CFG, Path(CFG.work_dir)/\"tiles\")\n   * The real deal:  git clone https://github.com/StarTrail-org/PixelRAG\n                     uv sync --package pixelrag-index && pixelrag-index build\n\"\"\")\n   return {\"index\": index, \"backend\": active_backend, \"tiles\": tiles, \"server\": server,\n           \"search\": lambda q, k=5: pretty_print(q, search(q, index, active_backend, cfg, k))}\nif __name__ == \"__main__\":\n   parser = argparse.ArgumentParser(add_help=False)\n   parser.add_argument(\"--no-server\", action=\"store_true\")\n   parser.add_argument(\"--no-train\", action=\"store_true\")\n   parser.add_argument(\"--backend\", default=None)\n   args, _ = parser.parse_known_args()\n   if args.no_server:\n       CFG.enable_server = False\n   if args.no_train:\n       CFG.enable_adapter_train = False\n   if args.backend:\n       CFG.backend = args.backend\n   STATE = main(CFG)\n```\n\nWe connect every component through the main execution workflow and run the complete PixelRAG tutorial from end to end. We demonstrate search, benchmark the baseline system, compare dense-only retrieval, train the adapter, launch the API, and optionally generate answers from retrieved images. We finally display index statistics, saved output locations, extension options, and command-line controls for disabling the server, training stage, or changing the embedding backend.\n\nIn conclusion, we implemented the complete PixelRAG workflow, from rendering documents into screenshot tiles to retrieving and serving relevant visual evidence through a searchable API. We combined dense vision-language embeddings, OCR-derived sparse retrieval, reciprocal rank fusion, FAISS indexing, document-level score aggregation, and contrastive adapter training within a single runnable pipeline. We also measured the system with retrieval benchmarks and inspected results visually, which allows us to compare configurations instead of relying only on qualitative outputs. By working directly with rendered pixels, we preserved document structure, tables, images, mathematical notation, code blocks, and visual layout that traditional text-only pipelines frequently discard, while creating a flexible foundation that we can extend to private documents, larger corpora, stronger multimodal embedding models, and fully grounded vision-language generation.\n\nCheck out the** FULL CODES here. **Also, feel free to follow us on\n\n**and don’t forget to join our**[Twitter](https://x.com/intent/follow?screen_name=marktechpost)\n\n**and Subscribe to**\n\n[150k+ML SubReddit](https://www.reddit.com/r/machinelearningnews/)**. Wait! are you on telegram?**\n\n[our Newsletter](https://www.aidevsignals.com/)\n\n[now you can join us on telegram as well.](https://t.me/machinelearningresearchnews)Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? [Connect with us](https://forms.gle/wbash1wF6efRj8G58)\n\nSana Hassan, a consulting intern at Marktechpost and dual-degree student at IIT Madras, is passionate about applying technology and AI to address real-world challenges. With a keen interest in solving practical problems, he brings a fresh perspective to the intersection of AI and real-life solutions.", "url": "https://wpnews.pro/news/pixel-native-rag-a-practical-guide-to-visual-document-indexing", "canonical_source": "https://www.marktechpost.com/2026/08/04/pixel-native-rag-a-practical-guide-to-visual-document-indexing/", "published_at": "2026-08-04 22:27:38+00:00", "updated_at": "2026-08-04 22:43:26.389217+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "generative-ai", "ai-tools", "ai-research"], "entities": ["StarTrail-org", "PixelRAG", "SigLIP", "CLIP", "Qwen3-VL", "FAISS", "FastAPI", "BM25"], "alternates": {"html": "https://wpnews.pro/news/pixel-native-rag-a-practical-guide-to-visual-document-indexing", "markdown": "https://wpnews.pro/news/pixel-native-rag-a-practical-guide-to-visual-document-indexing.md", "text": "https://wpnews.pro/news/pixel-native-rag-a-practical-guide-to-visual-document-indexing.txt", "jsonld": "https://wpnews.pro/news/pixel-native-rag-a-practical-guide-to-visual-document-indexing.jsonld"}}