Common Crawl Data Stored on a Hugging Face Bucket

Common Crawl has mirrored selected crawl archives to a Hugging Face Storage Bucket since April 2026, alongside its existing AWS S3 distribution, using the same directory layout such as crawl-data/CC-MAIN-2026-17/. The bucket supports S3-like object storage with CDN pre-warming in several regions, and users can access it via the Hugging Face CLI, mount it as a local filesystem with hf-mount, or point existing S3 pipelines at the S3-compatible API. Common Crawl says the Hugging Face hosting lets users apply Hugging Face ecosystem tools, including the fsspec-compatible hf:// protocol and the warcio WARC reader, to large-scale crawl data.

Since April 2026, parts of our crawl archive have been available on a Hugging Face Storage Bucket https://huggingface.co/buckets/commoncrawl/commoncrawl , in addition to being available on AWS S3. Having the data on the Hugging Face HF lets you use the tools available in the HF ecosystem, making large-scale crawl data easier to access. This blog post uses practical examples to show you how to get started. Accessing the data through Hugging Face Storage Buckets Selected crawl archives are mirrored in a Hugging Face Storage Bucket https://huggingface.co/buckets/commoncrawl/commoncrawl , using the same directory layout as S3 e.g. crawl-data/CC-MAIN-2026-17/ . See the bucket README https://huggingface.co/buckets/commoncrawl/commoncrawl for a list of currently available crawls. Storage Buckets provide S3-like object storage on the Hugging Face Hub. You can browse files in the browser, transfer data with the hf CLI, or mount the bucket as a local filesystem. The bucket has CDN pre-warming https://huggingface.co/docs/hub/storage-buckets pre-warming-and-cdn enabled in several regions, which can improve read throughput when your compute runs nearby. For more detail, see the announcement blog post https://commoncrawl.org/blog/april-2026-crawl-archive-now-available-in-a-hugging-face-storage-bucket . Using the Hugging Face CLI Install the Hugging Face CLI https://huggingface.co/docs/huggingface hub/guides/cli , then list or download files: hf buckets list commoncrawl/commoncrawl/crawl-data/CC-MAIN-2026-17 hf buckets cp hf://buckets/commoncrawl/commoncrawl/crawl-data/CC-MAIN-2026-17/segments/.../warc/CC-MAIN-....warc.gz ./local-path/ Mounting the bucket as a local filesystem Install hf-mount https://github.com/huggingface/hf-mount , then mount the bucket. Files are fetched on read, so any local tool can access the data without downloading the full archive first: hf-mount start bucket commoncrawl/commoncrawl /mnt/commoncrawl ls /mnt/commoncrawl/crawl-data/CC-MAIN-2026-17/ See the access patterns guide https://huggingface.co/docs/hub/storage-buckets-access mount-as-a-local-filesystem for backend options and caching. Using an existing S3 pipeline Keep your current code or command and point to the S3-compatible API for buckets. Data will be fetched using the same client but from Hugging Face instead of AWS. See the S3 compatibility documentation https://huggingface.co/docs/hub/storage-buckets-s3 for more info. Reading WARC files from Hugging Face Hugging Face provides file-system-like access to models, datasets, and buckets via the hf:// protocol, i.e., a pythonic fsspec-compatible file interface to the Hugging Face Hub https://huggingface.co/docs/huggingface hub/main/en/guides/hf file system . The WARC reader/writer tool warcio https://github.com/webrecorder/warcio fully supports fsspec https://filesystem-spec.readthedocs.io/en/latest/ . Make sure to install the optional warcio dependencies and the huggingface hub https://pypi.org/project/huggingface-hub/ package. python Read a WARC file via fsspec and hf:// protocol import fsspec from warcio.archiveiterator import ArchiveIterator with fsspec.open 'hf://buckets/commoncrawl/commoncrawl/crawl-data/CC-MAIN-2026-17/segments/1775805908305.14/warc/CC-MAIN-20260410081153-20260410111153-00000.warc.gz', 'rb' as stream: for i, record in enumerate ArchiveIterator stream : if i = 10: break print record.rec type if record.rec type == 'response': print record.rec headers.get header 'WARC-Target-URI' The code will print the first ten WARC record types and the URLs of response records: warcinfo request response http://003ms.ru/catalog/lekarstvennye-sredstva/nervnaya-sistema/antigrippin-312/tabletki-250-mgplus3-mgplus50-mg-dlya-detej-30-shtuk-shipuchie metadata request ... You can also use the warcio CLI directly to read from the HF bucket: Note: This command will trigger a broken pipe error after 10 records due to head -n 10 warcio index hf://buckets/commoncrawl/commoncrawl/crawl-data/CC-MAIN-2026-17/segments/1775805908305.14/warc/CC-MAIN-20260410081153-20260410111153-00000.warc.gz -f offset,content-type,http:content-type,warc-target-uri | head -n 10 Querying the URL index via Hugging Face Common Crawl's URL Index https://commoncrawl.org/url-index previously known as the Columnar Index is one of the indexes available for querying the Common Crawl corpus. As the name suggests, it is an index to the WARC files and URLs in the Common Crawl corpus, stored in a columnar format Apache Parquet . This format is suited to efficient analytical and/or bulk queries of the data, saving time and computing resources. The index files are also available on the Common Crawl HF bucket https://huggingface.co/buckets/commoncrawl/commoncrawl . DuckDB https://duckdb.org/ can be used to query the index without downloading all the data. Index schema Let's first look at the schema of the index: duckdb -c "DESCRIBE FROM read parquet 'https://huggingface.co/buckets/commoncrawl/commoncrawl/resolve/cc-index/table/cc-main/warc/crawl=CC-MAIN-2026-17/subset=warc/part-00000-ee7620e8-27a7-4c74-846b-a92b039dae92.c000.gz.parquet' " The DuckDB CLI will print the schema of the URL index: ┌─────────────────────────────────────────────────────┐ │ Describe │ │ │ │ url surtkey varchar │ │ url varchar │ │ url host name varchar │ │ url host tld varchar │ │ url host 2nd last part varchar │ │ url host 3rd last part varchar │ │ url host 4th last part varchar │ │ url host 5th last part varchar │ │ url host registry suffix varchar │ │ url host registered domain varchar │ │ url host private suffix varchar │ ... Querying the URL index with Python In addition to the CLI, you can also use the DuckDB Python client to run queries: python import duckdb duckdb.sql 'INSTALL httpfs; LOAD httpfs;' The index uses hive-partitioning for crawl and subset which are automatically constructed if you use the S3 index. However, we are using HF via HTTP so we need to manually construct the partitions using glob from the HF API. from huggingface hub import hffs NOTE: We are using only a sample of all files part-00000- index files = list hffs.glob "buckets/commoncrawl/commoncrawl/cc-index/table/cc-main/warc/crawl= /subset= /part-00000- .parquet" print f"Found {len index files :,} index files on HF bucket" Rewrite bucket paths to HTTP URLs http index files = f.replace "buckets/commoncrawl/commoncrawl/", "https://huggingface.co/buckets/commoncrawl/commoncrawl/resolve/" for f in index files Let's see how large our sample is duckdb.sql f"SELECT crawl, subset, COUNT FROM read parquet {http index files r}, hive partitioning = true GROUP BY crawl, subset" .show Time to run the first query This query counts the number of pages per domain within a single top-level domain, .ru Russia : duckdb.sql f"""SELECT COUNT AS count, url host registered domain FROM read parquet {http index files r}, hive partitioning = true WHERE crawl = 'CC-MAIN-2026-17' AND subset = 'warc' AND url host tld = 'ru' GROUP BY url host registered domain HAVING COUNT = 100 ORDER BY count DESC""" .show The last query in the Python script will produce something like this it only represents parts of the crawl since we query only selected index files : ┌────────┬────────────────────────────┐ │ count │ url host registered domain │ │ int64 │ varchar │ ├────────┼────────────────────────────┤ │ 166381 │ yandex.ru │ │ 28067 │ zr.ru │ │ 23968 │ zin.ru │ │ 18272 │ zarplata.ru │ │ 16030 │ zab.ru │ │ 13283 │ yugzone.ru │ │ 12003 │ yapl.ru │ ... Download speed We conducted initial experiments to evaluate the download speed for extracting individual records from the crawl archive. The evaluation involved fetching homepage URLs, i.e., sending many small range requests to the download server. We ran the experiment using CDX Toolkit https://github.com/commoncrawl/cdx toolkit and with various client-server combinations like downloading from S3 or HF bucket to an US-edge or EU-edge client but also to cloud-based clients. The results showed that from S3 to S3 us-east-1 achieved the highest download speed up to 3,500 records/s , whereas from HF to HF was up to 1,500 records/s. Downloading from the HF bucket to an edge client was significantly slower due to rate limiting and the many small range requests < 100 records/s . Note that these experiments provide preliminary findings and should not be treated as a definitive benchmark. We strongly recommend conducting your own measurements before committing to large data transfers. Jupyter Notebooks In our cc-notebooks repository https://github.com/commoncrawl/cc-notebooks/ on GitHub, you can find ready-to-use Jupyter notebooks for the examples in this blog post and more: - cc-index-hf.ipynb https://github.com/commoncrawl/cc-notebooks/blob/main/cc-huggingface/cc-index-hf.ipynb : Common Crawl's URL Index via Hugging Face - s3-hf.ipynb https://github.com/commoncrawl/cc-notebooks/blob/main/cc-huggingface/s3-hf.ipynb : Using Common Crawl data via Hugging Face's S3-compatible gateway - warcio-hf.ipynb https://github.com/commoncrawl/cc-notebooks/blob/main/cc-huggingface/warcio-hf.ipynb : Reading WARC files from Hugging Face We welcome feedback on the Common Crawl Hugging Face Bucket. Please contact us on our Discord https://discord.gg/njaVFh7avF or Google Group https://discord.gg/njaVFh7avF . Malte Ostendorff https://commoncrawl.org/team/malte-ostendorff Erratum: Content is truncated More details https://commoncrawl.org/errata/content-is-truncated