Fifteen days, twenty-five releases. PageSift went public as 0.20 on October 20; today it ships 1.0.
PageSift is a web content engine written in Rust. It turns URLs, PDFs and Office files into clean Markdown and structured data for AI agents and LLM pipelines — one self-hosted binary with a Firecrawl-compatible HTTP API, a Jina Reader-compatible endpoint (including /s/), an MCP server, and Python / Node.js / Rust SDKs.
1.0 is not a speed release — performance landed with the 0.9 architecture rebuild. 1.0 is feature completion.
OCR, built from scratch. Scanned PDFs were PageSift's documented weak spot. 1.0 ships its own OCR pipeline — text detection, recognition, formula recognition (LaTeX out), layout analysis — running on a self-written inference runtime (safetensors executor; no ONNX Runtime) and a self-written CUDA GPU layer. Models are open weights, downloaded on first use, off by default. On the olmOCR-bench scanned set it scores 0.543 vs Docling (GPU) at 0.410.
Selector extraction with memory. Give it a JSON schema; when a site redesign breaks your selectors, it relocates elements using element fingerprints and a statistical posterior — and reports relocated, ambiguous, or missing instead of silently returning wrong data.
Change monitoring. A /v2/monitor endpoint family with block-level diffing that separates noise changes (prices, dates, counters) from substantive ones.
Browser actions, sessions, and an agent endpoint across the HTTP, MCP and Jina surfaces — plus WARC files as input, a hooks + [[rules]] system, and per-host secret injection.
completed. They now fail loudly, with an errors endpoint. 1.0 extends the rule to the API surface: options the engine cannot honor are rejected (unknown MCP arguments error; unimplementable Jina headers return UNSUPPORTED_OPTION), and the release notes carry a Known limits section that lists what isn't tested and what doesn't work — including security caveats.
| Workload | PageSift 1.0 | Fastest rival | Gap |
|---|---|---|---|
| PDF → Markdown, olmOCR-bench 1,269 pages, single thread | 2.4 ms/page | pypdf 50.0 ms/page | ×20.8 |
| HTML → Markdown, 600 documents | 0.24 ms/page | Turndown 1.9 ms/page | ×8.1 |
| Main content, Firecrawl 597 pages, single thread p50 | 1.0 ms/page | crw 7.4 ms/page | ×7.3 |
| Office / e-books, 81 files | 93.6 files/s | Kreuzberg 19.5 | ×4.8 |
| Wide crawl, 200 sites × 100 pages, unthrottled | 3,500–4,400 pages/s, 280–316 MB peak | Spider 2,000–2,700 pages/s, 5.5 GB | — |
Honest footnotes, because they matter: competitor numbers are the recorded late-October runs, not re-run for this release; and the workloads where PageSift loses are listed in the same document ("Largest gaps") — e.g. a single-host replay looks slow mostly because of the default 1-second per-host politeness interval, which is policy, not engine speed. Full data and reproduction scripts live in the repo (docs/benchmarks.md, benchmarks/).
In late October an external team integrated PageSift into their production system and immediately hit a real bug: the official Node binding required a newer glibc than their server had (2.32). 0.81.5 fixed it — build baseline lowered to glibc 2.28, plus a release gate that fails the release if any artifact exceeds it. A user found it; the project fixed it and made it impossible to reintroduce silently. That's the workflow 1.0 is built around.
For: teams feeding RAG or agent pipelines who want a self-hosted Firecrawl/Jina alternative without per-page pricing, and anyone who needs web pages, PDFs and Office files in one pipeline.
Not for: people who need a hosted cloud API (PageSift is self-hosted; a hosted version is not part of 1.0), or bypassing hard anti-bot targets (robots.txt and politeness are respected; captchas are reported, not solved).
PageSift is a solo project. It exists because the crawler feeding my other project was too slow and not clean enough, so I wrote my own. 1.0 closes the feature list. What I want next is more production workloads breaking it in interesting ways.
Repo: https://github.com/YOLOW92/pagesift — AGPL-3.0, commercial licensing available. Issues and head-to-head benchmarks welcome, good or bad.