{"slug": "building-a-vertical-corpus-builder-clean-jsonl-datasets-for-llm-fine-tuning", "title": "Building a Vertical Corpus Builder: Clean JSONL Datasets for LLM Fine-Tuning", "summary": "A developer built a vertical corpus builder that crawls seed URLs, strips boilerplate, dedupes, and chunks content into token-aware JSONL datasets for LLM fine-tuning. The tool, published on Apify Store, defaults to Cornell LII for legal text because sources like CourtListener and Justia block datacenter IPs. Tests on three Supreme Court opinions produced 443 clean chunks with provenance metadata.", "body_md": "Raw web pages are terrible training data. Nav bars, cookie banners, \"related articles\" and ads drown the signal, and near-identical syndicated text pollutes the corpus. If you're fine-tuning a domain model — legal reasoning, medical QA, financial analysis — you want clean vertical text with provenance, not a pile of HTML. I built an actor that goes from seed URLs to a token-aware JSONL dataset in one run.\n\n```\nseed URLs → crawl (same-domain BFS) → clean (boilerplate strip)\n  → dedupe (exact + near) → token-aware chunk → JSONL with provenance\n```\n\nThree steps matter most:\n\n`script`\n\n/`style`\n\n/`nav`\n\n/`footer`\n\n/`header`\n\n/`aside`\n\nand utility blocks (breadcrumbs, shares, comments, menus). One detail that earned its keep: paragraphs where more than half the links are anchor text get dropped too — those are link farms, not content.Every record carries `source`\n\n(hostname), `url`\n\n, `domain`\n\n(your vertical), `title`\n\n, and `chunk_index`\n\n— so you can filter, cite, or re-weight the corpus later.\n\nNot every legal text source works from a datacenter IP. CourtListener sits behind an AWS WAF JavaScript challenge, and Justia 403s datacenter egress — both unusable as plain-HTTP seeds. **Cornell LII** (`law.cornell.edu/supremecourt/text/…`\n\n) serves full opinions as clean HTML and works beautifully. The honest move was documenting that in the README and defaulting the examples to LII rather than pretending every source is reachable.\n\nThree LII Supreme Court opinions (Dobbs, Bruen, WV v EPA):\n\n`domain=legal`\n\n, source `law.cornell.edu`\n\n, 0 duplicate texts`output.jsonl`\n\n— 443 lines / ~811 KB, plus a `SUMMARY`\n\nrecord: 443 chunks, 2 pages, ~173,000 estimated tokens, no failed sources`maxChunksPerPage`\n\n.👉 **Vertical Corpora Builder on Apify Store**\n\nWhile you're here, these might be worth a read:", "url": "https://wpnews.pro/news/building-a-vertical-corpus-builder-clean-jsonl-datasets-for-llm-fine-tuning", "canonical_source": "https://dev.to/darksider4all_afa2428f63d0/building-a-vertical-corpus-builder-clean-jsonl-datasets-for-llm-fine-tuning-2p32", "published_at": "2026-08-19 11:11:46+00:00", "updated_at": "2026-08-19 11:42:19.147555+00:00", "lang": "en", "topics": ["large-language-models", "developer-tools", "ai-infrastructure"], "entities": ["Apify", "Cornell LII", "CourtListener", "Justia", "Dobbs", "Bruen", "WV v EPA"], "alternates": {"html": "https://wpnews.pro/news/building-a-vertical-corpus-builder-clean-jsonl-datasets-for-llm-fine-tuning", "markdown": "https://wpnews.pro/news/building-a-vertical-corpus-builder-clean-jsonl-datasets-for-llm-fine-tuning.md", "text": "https://wpnews.pro/news/building-a-vertical-corpus-builder-clean-jsonl-datasets-for-llm-fine-tuning.txt", "jsonld": "https://wpnews.pro/news/building-a-vertical-corpus-builder-clean-jsonl-datasets-for-llm-fine-tuning.jsonld"}}