Commoncrawl (auto-discovered)

Common Crawl has mirrored selected crawl archives to a Hugging Face Storage Bucket since April 2026, alongside its existing AWS S3 distribution, using the same directory layout such as crawl-data/CC-M…

Common Crawl released the first machine learning embeddings built from its Web Graph, publishing 128-dimensional vectors for 52.9 million web hosts learned from hyperlinks alone. The dataset, trained …

Common Crawl's July 2026 crawl (CC-MAIN-2026-30) recorded 598,298 responses to /llms.txt and /llms-full.txt seed requests, of which 14,191 were empty, leaving 584,107 files analyzed for spec complianc…