Raw web pages are terrible training data. Nav bars, cookie banners, "related articles" and ads drown the signal, and near-identical syndicated text pollutes the corpus. If you're fine-tuning a domain model — legal reasoning, medical QA, financial analysis — you want clean vertical text with provenance, not a pile of HTML. I built an actor that goes from seed URLs to a token-aware JSONL dataset in one run.
seed URLs → crawl (same-domain BFS) → clean (boilerplate strip)
→ dedupe (exact + near) → token-aware chunk → JSONL with provenance
Three steps matter most:
script
/style
/nav
/footer
/header
/aside
and utility blocks (breadcrumbs, shares, comments, menus). One detail that earned its keep: paragraphs where more than half the links are anchor text get dropped too — those are link farms, not content.Every record carries source
(hostname), url
, domain
(your vertical), title
, and chunk_index
— so you can filter, cite, or re-weight the corpus later.
Not every legal text source works from a datacenter IP. CourtListener sits behind an AWS WAF JavaScript challenge, and Justia 403s datacenter egress — both unusable as plain-HTTP seeds. Cornell LII (law.cornell.edu/supremecourt/text/…
) serves full opinions as clean HTML and works beautifully. The honest move was documenting that in the README and defaulting the examples to LII rather than pretending every source is reachable.
Three LII Supreme Court opinions (Dobbs, Bruen, WV v EPA):
domain=legal
, source law.cornell.edu
, 0 duplicate textsoutput.jsonl
— 443 lines / ~811 KB, plus a SUMMARY
record: 443 chunks, 2 pages, ~173,000 estimated tokens, no failed sourcesmaxChunksPerPage
.👉 Vertical Corpora Builder on Apify Store
While you're here, these might be worth a read: