{"slug": "pagesift-1-0-the-rust-web-content-engine-is-feature-complete", "title": "PageSift 1.0: the Rust web content engine is feature-complete", "summary": "A solo developer released PageSift 1.0, a self-hosted Rust web content engine that converts URLs, PDFs and Office files into Markdown and structured data for LLM and agent pipelines, after 25 releases in 15 days. The 1.0 milestone adds a from-scratch OCR pipeline running on a self-written inference runtime and CUDA layer, which scores 0.543 on the olmOCR-bench scanned set versus Docling's 0.410, along with schema-driven selector extraction that relocates elements after site redesigns, block-level change monitoring, and browser actions across its HTTP, MCP and Jina-compatible interfaces. Benchmarks claim 2.4 ms/page for PDF-to-Markdown (20.8x faster than pypdf) and 0.24 ms/page for HTML-to-Markdown (8.1x faster than Turndown), with the release notes documenting known limits and security caveats.", "body_md": "Fifteen days, twenty-five releases. PageSift went public as 0.20 on October 20; today it ships **1.0**.\n\nPageSift is a web content engine written in Rust. It turns URLs, PDFs and Office files into clean Markdown and structured data for AI agents and LLM pipelines — one self-hosted binary with a Firecrawl-compatible HTTP API, a Jina Reader-compatible endpoint (including `/s/`), an MCP server, and Python / Node.js / Rust SDKs.\n\n1.0 is not a speed release — performance landed with the 0.9 architecture rebuild. 1.0 is **feature completion**.\n\n**OCR, built from scratch.** Scanned PDFs were PageSift's documented weak spot. 1.0 ships its own OCR pipeline — text detection, recognition, formula recognition (LaTeX out), layout analysis — running on a self-written inference runtime (safetensors executor; no ONNX Runtime) and a self-written CUDA GPU layer. Models are open weights, downloaded on first use, off by default. On the olmOCR-bench scanned set it scores 0.543 vs Docling (GPU) at 0.410.\n\n**Selector extraction with memory.** Give it a JSON schema; when a site redesign breaks your selectors, it relocates elements using element fingerprints and a statistical posterior — and reports `relocated`, `ambiguous`, or `missing` instead of silently returning wrong data.\n\n**Change monitoring.** A `/v2/monitor` endpoint family with block-level diffing that separates noise changes (prices, dates, counters) from substantive ones.\n\n**Browser actions, sessions, and an agent endpoint** across the HTTP, MCP and Jina surfaces — plus WARC files as input, a hooks + `[[rules]]` system, and per-host secret injection.\n\n`completed`. They now fail loudly, with an errors endpoint. 1.0 extends the rule to the API surface: options the engine cannot honor are rejected (unknown MCP arguments error; unimplementable Jina headers return `UNSUPPORTED_OPTION`), and the release notes carry a Known limits section that lists what isn't tested and what doesn't work — including security caveats.\n| Workload | PageSift 1.0 | Fastest rival | Gap | \n|---|---|---|---|\n| PDF → Markdown, olmOCR-bench 1,269 pages, single thread | 2.4 ms/page | pypdf 50.0 ms/page | ×20.8 | \n| HTML → Markdown, 600 documents | 0.24 ms/page | Turndown 1.9 ms/page | ×8.1 | \n| Main content, Firecrawl 597 pages, single thread p50 | 1.0 ms/page | crw 7.4 ms/page | ×7.3 | \n| Office / e-books, 81 files | 93.6 files/s | Kreuzberg 19.5 | ×4.8 | \n| Wide crawl, 200 sites × 100 pages, unthrottled | 3,500–4,400 pages/s, 280–316 MB peak | Spider 2,000–2,700 pages/s, 5.5 GB | — | \n\nHonest footnotes, because they matter: competitor numbers are the recorded late-October runs, not re-run for this release; and the workloads where PageSift *loses* are listed in the same document (\"Largest gaps\") — e.g. a single-host replay looks slow mostly because of the default 1-second per-host politeness interval, which is policy, not engine speed. Full data and reproduction scripts live in the repo (`docs/benchmarks.md`, `benchmarks/`).\n\nIn late October an external team integrated PageSift into their production system and immediately hit a real bug: the official Node binding required a newer glibc than their server had (2.32). 0.81.5 fixed it — build baseline lowered to glibc 2.28, plus a release gate that fails the release if any artifact exceeds it. A user found it; the project fixed it and made it impossible to reintroduce silently. That's the workflow 1.0 is built around.\n\nFor: teams feeding RAG or agent pipelines who want a self-hosted Firecrawl/Jina alternative without per-page pricing, and anyone who needs web pages, PDFs and Office files in one pipeline.\n\nNot for: people who need a hosted cloud API (PageSift is self-hosted; a hosted version is not part of 1.0), or bypassing hard anti-bot targets (robots.txt and politeness are respected; captchas are reported, not solved).\n\nPageSift is a solo project. It exists because the crawler feeding my other project was too slow and not clean enough, so I wrote my own. 1.0 closes the feature list. What I want next is more production workloads breaking it in interesting ways.\n\nRepo: [https://github.com/YOLOW92/pagesift](https://github.com/YOLOW92/pagesift) — AGPL-3.0, commercial licensing available. Issues and head-to-head benchmarks welcome, good or bad.", "url": "https://wpnews.pro/news/pagesift-1-0-the-rust-web-content-engine-is-feature-complete", "canonical_source": "https://dev.to/yolow92/pagesift-10-the-rust-web-content-engine-is-feature-complete-51m7", "published_at": "2026-11-04 03:44:32+00:00", "updated_at": "2026-11-04 03:55:47.971299+00:00", "lang": "en", "topics": ["ai-tools", "ai-agents", "developer-tools", "ai-infrastructure", "agent-protocols"], "entities": ["PageSift", "Firecrawl", "Jina Reader", "Docling", "pypdf", "Turndown", "Kreuzberg", "Spider"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/pagesift-1-0-the-rust-web-content-engine-is-feature-complete", "markdown": "https://wpnews.pro/news/pagesift-1-0-the-rust-web-content-engine-is-feature-complete.md", "text": "https://wpnews.pro/news/pagesift-1-0-the-rust-web-content-engine-is-feature-complete.txt", "jsonld": "https://wpnews.pro/news/pagesift-1-0-the-rust-web-content-engine-is-feature-complete.jsonld"}}