{"slug": "one-engine-seven-open-source-projects-pagesift-1-4-and-the-yolow-family", "title": "One Engine, Seven Open-Source Projects: PageSift 1.4 and the yolow Family", "summary": "A developer released PageSift 1.4, completing a refactor that splits the scraper's engine into seven independently versioned MIT-licensed open-source projects, including yolowdoc, yolowsyntax, yolowzstd, yolow64, yolowpdf and yolowhtml. The release adds DataGraph, a ~4,000-line crate that tracks provenance, conflicts and versioning of extracted web data for LLM pipelines, exposed via CLI, HTTP API, an MCP tool and Python/Node SDKs. The developer reports benchmarks including 20.4 GB/s Base64 throughput, 1,900 pages/s PDF-to-Markdown conversion, and 1.000 structural F1 for Markdown extraction.", "body_md": "PageSift started as the ingestion layer of an AI research platform I was building: something had to fetch web pages and documents and turn them into clean Markdown that an LLM pipeline could actually use. It grew into a self-hosted web scraper, crawler and document parser — Firecrawl- and Jina-compatible API, MCP server, Python/Node SDKs, one Rust binary.\n\nOver the 1.x series I kept doing one thing: every time a piece of the engine turned out to be generally useful, I cut it out and gave it its own repository, its own versioning, and an MIT license. PageSift 1.4 finishes that split — and adds the subsystem the extraction was making room for.\n\n**The split, completed.** Wave one (1.2.7) turned the base64, PDF, HTML and content crates into thin shims over standalone libraries. Wave two (1.4.0) does the same for the remaining three: `pagesift-office` → **yolowdoc**, `pagesift-syntax` → **yolowsyntax**, `pagesift-compress` → **yolowzstd**. The engine is now a composition of seven independently versioned open-source projects — and since 1.3, PageSift itself is Apache-2.0 (it was AGPL before).\n\n**DataGraph.** The new crate, `pagesift-graph` (~4,000 lines), is what the cleanup was for. A normal scraper returns text and forgets everything else. DataGraph keeps the receipts:\n\n`sku`) are clustered into It's exposed everywhere PageSift is: CLI (`scrape --graph`, `crawl --graph`, `graph inspect`), the HTTP API (a `graph` slice on scrape responses, plus `/v2/graph/history` and `/v2/graph/conflicts`), the MCP tool `pagesift_graph_query`, and the Python and Node SDKs.\n\nAn LLM pipeline doesn't just need text. It needs to know where a claim came from, what it contradicts, and which version is current. That's the layer DataGraph adds.\n\nA fast Base64 codec with runtime-dispatched SIMD (AVX2/NEON), zero runtime dependencies, streaming and data-URL APIs. The 0.3 release added dedicated short-input kernels and fused hash-table tail processing: around **20.4 GB/s from 8 KiB to 1 MiB**, and short payloads (8–64 bytes) got 1.5–2.8× faster than in 0.2. Short inputs are the common case in real pipelines — tokens, hashes, embedded blobs — so that's where the work went.\n\n`cargo add yolow64` · [GitHub](https://github.com/YOLOW92/yolow64) · [crates.io](https://crates.io/crates/yolow64)\n\nA PDF-to-Markdown library and OCR engine that parses the format itself — no Poppler, no MuPDF, nothing to link against. The workspace includes GPU-accelerated OCR (PP-OCRv6). The README benchmark clocks **1,900 pages/s** on born-digital documents (i7-10700). This is the component that taught the family its hardest rule: every allocation a hostile file can influence is budgeted *before* it happens.\n\n`cargo add yolowpdf` · [GitHub](https://github.com/YOLOW92/yolowpdf) · [crates.io](https://crates.io/crates/yolowpdf)\n\nTwo crates: `yolowhtml-dom`, a zero-copy HTML5 parser over a contiguous arena (` Vec<Node>`, `u32` ids, CSS selectors), and `yolowhtml`, a single-pass main-content extractor that produces Markdown, metadata and publication dates from the same parse. On the README's test corpus it scores **1.000 structural F1** for Markdown across 15 page categories, and **0.945 F1** for publication-date extraction across 20 languages. Both crates are `#![forbid(unsafe_code)]`.\n\n`cargo add yolowhtml` · [GitHub](https://github.com/YOLOW92/yolowhtml) · [crates.io](https://crates.io/crates/yolowhtml)\n\nA Zstandard (RFC 8878) encoder/decoder written from the spec: zero dependencies, `#![forbid(unsafe_code)]`, multi-core encode/decode that emits standard concatenated frames — interoperability is tested against the official zstd CLI. It's built for the small-record end of the world: on 64-byte payloads it decodes at **3,257 MB/s (1.6× C libzstd)** and encodes at **264 MB/s (4.1×)** on an i7-10700, which is exactly the shape of PageSift's cache records. One stated gap: no dictionary support yet.\n\n`cargo add yolowzstd` · [GitHub](https://github.com/YOLOW92/yolowzstd) · [crates.io](https://crates.io/crates/yolowzstd)\n\nA pure-Rust runtime for tree-sitter grammars: the C runtime reimplemented in Rust, with **41 languages'** parse tables converted and embedded — 6.22 MiB total, Zstd-compressed (with yolowzstd, its only runtime dependency) and decompressed lazily per language. It extracts code outlines for 32 symbol-bearing languages. No C FFI, `#![forbid(unsafe_code)]`. Stated limits: no Query DSL and no incremental re-parsing.\n\n`cargo add yolowsyntax` · [GitHub](https://github.com/YOLOW92/yolowsyntax) · [crates.io](https://crates.io/crates/yolowsyntax)\n\nDocuments to Markdown behind a single `convert(bytes, format, opts)`: Office (DOCX/XLSX/PPTX, plus legacy DOC/XLS/PPT and XLSB), OpenDocument, RTF, WordPerfect, HWP/HWPX, e-books (EPUB/MOBI/FB2), email (EML/MBOX/MSG), archives (ZIP/TAR/7Z/GZ/BZ2/XZ — the decompressors are pure Rust too), SQLite, dBASE, CSV and Jupyter notebooks. The per-format parsers go deep — pivot tables, charts, equations — and a `Limits` budget is checked before any work starts.\n\n`cargo add yolowdoc` · [GitHub](https://github.com/YOLOW92/yolowdoc) · [crates.io](https://crates.io/crates/yolowdoc)\n\nThe family is a small lattice, not a pile: PageSift depends on all six libraries; yolowsyntax depends on yolowzstd; yolowdoc builds on both; yolowpdf uses yolow64. The rules are shared across all of them — pure Rust, no C dependencies, unsafe code forbidden in the libraries, and resource budgets enforced before allocation, because every one of these parsers eats untrusted input for a living.\n\nPageSift is the engine. These six are its organs, now living their own lives — each usable without the others, each on crates.io under MIT, with PageSift itself under Apache-2.0.\n\nLinks: [PageSift](https://github.com/YOLOW92/pagesift) · [yolow64](https://github.com/YOLOW92/yolow64) · [yolowpdf](https://github.com/YOLOW92/yolowpdf) · [yolowhtml](https://github.com/YOLOW92/yolowhtml) · [yolowzstd](https://github.com/YOLOW92/yolowzstd) · [yolowsyntax](https://github.com/YOLOW92/yolowsyntax) · [yolowdoc](https://github.com/YOLOW92/yolowdoc)\n\nTry them, break them, and tell me where they break — issues and benchmarks from other machines are the most useful feedback I can get.", "url": "https://wpnews.pro/news/one-engine-seven-open-source-projects-pagesift-1-4-and-the-yolow-family", "canonical_source": "https://dev.to/yolow92/one-engine-seven-open-source-projects-pagesift-14-and-the-yolow-family-o19", "published_at": "2026-11-14 01:37:54+00:00", "updated_at": "2026-11-14 01:50:43.003392+00:00", "lang": "en", "topics": ["ai-tools", "ai-agents", "agent-protocols", "developer-tools", "ai-infrastructure"], "entities": ["PageSift", "DataGraph", "yolowdoc", "yolowzstd", "yolowhtml", "yolowpdf", "yolow64", "MCP"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/one-engine-seven-open-source-projects-pagesift-1-4-and-the-yolow-family", "markdown": "https://wpnews.pro/news/one-engine-seven-open-source-projects-pagesift-1-4-and-the-yolow-family.md", "text": "https://wpnews.pro/news/one-engine-seven-open-source-projects-pagesift-1-4-and-the-yolow-family.txt", "jsonld": "https://wpnews.pro/news/one-engine-seven-open-source-projects-pagesift-1-4-and-the-yolow-family.jsonld"}}