# PageSift 1.0: the Rust web content engine is feature-complete

> Source: <https://dev.to/yolow92/pagesift-10-the-rust-web-content-engine-is-feature-complete-51m7>
> Published: 2026-11-04 03:44:32+00:00

Fifteen days, twenty-five releases. PageSift went public as 0.20 on October 20; today it ships **1.0**.

PageSift is a web content engine written in Rust. It turns URLs, PDFs and Office files into clean Markdown and structured data for AI agents and LLM pipelines — one self-hosted binary with a Firecrawl-compatible HTTP API, a Jina Reader-compatible endpoint (including `/s/`), an MCP server, and Python / Node.js / Rust SDKs.

1.0 is not a speed release — performance landed with the 0.9 architecture rebuild. 1.0 is **feature completion**.

**OCR, built from scratch.** Scanned PDFs were PageSift's documented weak spot. 1.0 ships its own OCR pipeline — text detection, recognition, formula recognition (LaTeX out), layout analysis — running on a self-written inference runtime (safetensors executor; no ONNX Runtime) and a self-written CUDA GPU layer. Models are open weights, downloaded on first use, off by default. On the olmOCR-bench scanned set it scores 0.543 vs Docling (GPU) at 0.410.

**Selector extraction with memory.** Give it a JSON schema; when a site redesign breaks your selectors, it relocates elements using element fingerprints and a statistical posterior — and reports `relocated`, `ambiguous`, or `missing` instead of silently returning wrong data.

**Change monitoring.** A `/v2/monitor` endpoint family with block-level diffing that separates noise changes (prices, dates, counters) from substantive ones.

**Browser actions, sessions, and an agent endpoint** across the HTTP, MCP and Jina surfaces — plus WARC files as input, a hooks + `[[rules]]` system, and per-host secret injection.

`completed`. They now fail loudly, with an errors endpoint. 1.0 extends the rule to the API surface: options the engine cannot honor are rejected (unknown MCP arguments error; unimplementable Jina headers return `UNSUPPORTED_OPTION`), and the release notes carry a Known limits section that lists what isn't tested and what doesn't work — including security caveats.
| Workload | PageSift 1.0 | Fastest rival | Gap | 
|---|---|---|---|
| PDF → Markdown, olmOCR-bench 1,269 pages, single thread | 2.4 ms/page | pypdf 50.0 ms/page | ×20.8 | 
| HTML → Markdown, 600 documents | 0.24 ms/page | Turndown 1.9 ms/page | ×8.1 | 
| Main content, Firecrawl 597 pages, single thread p50 | 1.0 ms/page | crw 7.4 ms/page | ×7.3 | 
| Office / e-books, 81 files | 93.6 files/s | Kreuzberg 19.5 | ×4.8 | 
| Wide crawl, 200 sites × 100 pages, unthrottled | 3,500–4,400 pages/s, 280–316 MB peak | Spider 2,000–2,700 pages/s, 5.5 GB | — | 

Honest footnotes, because they matter: competitor numbers are the recorded late-October runs, not re-run for this release; and the workloads where PageSift *loses* are listed in the same document ("Largest gaps") — e.g. a single-host replay looks slow mostly because of the default 1-second per-host politeness interval, which is policy, not engine speed. Full data and reproduction scripts live in the repo (`docs/benchmarks.md`, `benchmarks/`).

In late October an external team integrated PageSift into their production system and immediately hit a real bug: the official Node binding required a newer glibc than their server had (2.32). 0.81.5 fixed it — build baseline lowered to glibc 2.28, plus a release gate that fails the release if any artifact exceeds it. A user found it; the project fixed it and made it impossible to reintroduce silently. That's the workflow 1.0 is built around.

For: teams feeding RAG or agent pipelines who want a self-hosted Firecrawl/Jina alternative without per-page pricing, and anyone who needs web pages, PDFs and Office files in one pipeline.

Not for: people who need a hosted cloud API (PageSift is self-hosted; a hosted version is not part of 1.0), or bypassing hard anti-bot targets (robots.txt and politeness are respected; captchas are reported, not solved).

PageSift is a solo project. It exists because the crawler feeding my other project was too slow and not clean enough, so I wrote my own. 1.0 closes the feature list. What I want next is more production workloads breaking it in interesting ways.

Repo: [https://github.com/YOLOW92/pagesift](https://github.com/YOLOW92/pagesift) — AGPL-3.0, commercial licensing available. Issues and head-to-head benchmarks welcome, good or bad.