Data Scout: Targeted Web Crawling for Domain-Specific Pretraining Corpora

A new arXiv paper (2609.05766v1) presents Data Scout, a targeted web-crawling method that builds domain-specific pretraining corpora by directing an LLM to expand a root topic into a taxonomy and thousands of search queries, then screening returned URLs at the subdomain level with a user-supplied classifier. Using the FineMath classifier as the probe, 21.9% of crawled pages were high-quality math content — 70x the 0.31% rate from filtering a comparable web sample — and 63.2% of those pages are missing from CommonCrawl altogether. Continued pretraining of Llama-3.2-3B on 1.9B Data Scout tokens matched the FineMath corpus on GSM8k, and the authors state the method can in principle apply to any domain with such a classifier.

arXiv:2609.05766v1 Announce Type: new Abstract: The dominant approach to building domain-specific pretraining corpora is to filter large web archives such as CommonCrawl. This works well for popular domains but breaks down for specialized ones, where relevant content is sparse and often beyond the reach of popularity-driven crawlers. We present Data Scout, which inverts this: instead of filtering an archive, it directs a targeted crawl. An LLM expands a root topic into a taxonomy and thousands of search queries; the returned URLs seeds are grouped by subdomain and screened with a user-supplied classifier the probe , admitting each subdomain on the basis of a small sample. This works because relevance has a sharp boundary at the subdomain level: in mathematics, a page is 21x more likely to be relevant than one on a sibling subdomain. With the FineMath classifier as the probe, 21.9% of crawled pages are high-quality math content, 70x the 0.31% rate from filtering a comparable web sample, so the crawl wastes far less effort. But the payoff is not just efficiency: 63.2% of these pages are missing from CommonCrawl altogether, yet just as useful for training. Continued pretraining of Llama-3.2-3B on 1.9B Data Scout tokens matches FineMath corpus on GSM8k. Because the probe is the only domain-specific component, Data Scout can in principle apply to any domain with such a classifier.