Building a Vertical Corpus Builder: Clean JSONL Datasets for LLM Fine-Tuning

A developer built a vertical corpus builder that crawls seed URLs, strips boilerplate, dedupes, and chunks content into token-aware JSONL datasets for LLM fine-tuning. The tool, published on Apify Store, defaults to Cornell LII for legal text because sources like CourtListener and Justia block datacenter IPs. Tests on three Supreme Court opinions produced 443 clean chunks with provenance metadata.

Raw web pages are terrible training data. Nav bars, cookie banners, "related articles" and ads drown the signal, and near-identical syndicated text pollutes the corpus. If you're fine-tuning a domain model — legal reasoning, medical QA, financial analysis — you want clean vertical text with provenance, not a pile of HTML. I built an actor that goes from seed URLs to a token-aware JSONL dataset in one run. seed URLs → crawl same-domain BFS → clean boilerplate strip → dedupe exact + near → token-aware chunk → JSONL with provenance Three steps matter most: script / style / nav / footer / header / aside and utility blocks breadcrumbs, shares, comments, menus . One detail that earned its keep: paragraphs where more than half the links are anchor text get dropped too — those are link farms, not content.Every record carries source hostname , url , domain your vertical , title , and chunk index — so you can filter, cite, or re-weight the corpus later. Not every legal text source works from a datacenter IP. CourtListener sits behind an AWS WAF JavaScript challenge, and Justia 403s datacenter egress — both unusable as plain-HTTP seeds. Cornell LII law.cornell.edu/supremecourt/text/… serves full opinions as clean HTML and works beautifully. The honest move was documenting that in the README and defaulting the examples to LII rather than pretending every source is reachable. Three LII Supreme Court opinions Dobbs, Bruen, WV v EPA : domain=legal , source law.cornell.edu , 0 duplicate texts output.jsonl — 443 lines / ~811 KB, plus a SUMMARY record: 443 chunks, 2 pages, ~173,000 estimated tokens, no failed sources maxChunksPerPage .👉 Vertical Corpora Builder on Apify Store While you're here, these might be worth a read: