Since April 2026, parts of our crawl archive have been available on a Hugging Face Storage Bucket, in addition to being available on AWS S3. Having the data on the Hugging Face (HF) lets you use the tools available in the HF ecosystem, making large-scale crawl data easier to access. This blog post uses practical examples to show you how to get started.
Accessing the data through Hugging Face Storage Buckets #
Selected crawl archives are mirrored in a Hugging Face Storage Bucket, using the same directory layout as S3 (e.g. crawl-data/CC-MAIN-2026-17/). See the bucket README for a list of currently available crawls. Storage Buckets provide S3-like object storage on the Hugging Face Hub. You can browse files in the browser, transfer data with the hf CLI, or mount the bucket as a local filesystem. The bucket has CDN pre-warming enabled in several regions, which can improve read throughput when your compute runs nearby. For more detail, see the announcement blog post.
Using the Hugging Face CLI
Install the Hugging Face CLI, then list or download files:
hf buckets list commoncrawl/commoncrawl/crawl-data/CC-MAIN-2026-17
hf buckets cp hf://buckets/commoncrawl/commoncrawl/crawl-data/CC-MAIN-2026-17/segments/.../warc/CC-MAIN-....warc.gz ./local-path/
Mounting the bucket as a local filesystem
Install hf-mount, then mount the bucket. Files are fetched on read, so any local tool can access the data without down the full archive first:
hf-mount start bucket commoncrawl/commoncrawl /mnt/commoncrawl
ls /mnt/commoncrawl/crawl-data/CC-MAIN-2026-17/
See the access patterns guide for backend options and caching.
Using an existing S3 pipeline
Keep your current code or command and point to the S3-compatible API for buckets. Data will be fetched using the same client but from Hugging Face instead of AWS. See the S3 compatibility documentation for more info.
Reading WARC files from Hugging Face #
Hugging Face provides file-system-like access to models, datasets, and buckets via the hf:// protocol, i.e., a pythonic fsspec-compatible file interface to the Hugging Face Hub. The WARC reader/writer tool warcio fully supports fsspec. Make sure to install the optional warcio dependencies and the huggingface_hub package.
import fsspec
from warcio.archiveiterator import ArchiveIterator
with fsspec.open('hf://buckets/commoncrawl/commoncrawl/crawl-data/CC-MAIN-2026-17/segments/1775805908305.14/warc/CC-MAIN-20260410081153-20260410111153-00000.warc.gz', 'rb') as stream:
for i, record in enumerate(ArchiveIterator(stream)):
if i >= 10:
break
print(record.rec_type)
if record.rec_type == 'response':
print(record.rec_headers.get_header('WARC-Target-URI'))
The code will print the first ten WARC record types and the URLs of response records:
warcinfo
request
response
http://003ms.ru/catalog/lekarstvennye-sredstva/nervnaya-sistema/antigrippin-312/tabletki-250-mgplus3-mgplus50-mg-dlya-detej-30-shtuk-shipuchie
metadata
request
...
You can also use the warcio CLI directly to read from the HF bucket:
warcio index hf://buckets/commoncrawl/commoncrawl/crawl-data/CC-MAIN-2026-17/segments/1775805908305.14/warc/CC-MAIN-20260410081153-20260410111153-00000.warc.gz -f offset,content-type,http:content-type,warc-target-uri | head -n 10
Querying the URL index via Hugging Face #
Common Crawl's URL Index (previously known as the Columnar Index) is one of the indexes available for querying the Common Crawl corpus. As the name suggests, it is an index to the WARC files and URLs in the Common Crawl corpus, stored in a columnar format (Apache Parquet). This format is suited to efficient analytical and/or bulk queries of the data, saving time and computing resources. The index files are also available on the Common Crawl HF bucket. DuckDB can be used to query the index without down all the data.
Index schema
Let's first look at the schema of the index:
duckdb -c "DESCRIBE FROM read_parquet('https://huggingface.co/buckets/commoncrawl/commoncrawl/resolve/cc-index/table/cc-main/warc/crawl=CC-MAIN-2026-17/subset=warc/part-00000-ee7620e8-27a7-4c74-846b-a92b039dae92.c000.gz.parquet')"
The DuckDB CLI will print the schema of the URL index:
┌─────────────────────────────────────────────────────┐
│ Describe │
│ │
│ url_surtkey varchar │
│ url varchar │
│ url_host_name varchar │
│ url_host_tld varchar │
│ url_host_2nd_last_part varchar │
│ url_host_3rd_last_part varchar │
│ url_host_4th_last_part varchar │
│ url_host_5th_last_part varchar │
│ url_host_registry_suffix varchar │
│ url_host_registered_domain varchar │
│ url_host_private_suffix varchar │
...
Querying the URL index with Python
In addition to the CLI, you can also use the DuckDB Python client to run queries:
import duckdb
duckdb.sql('INSTALL httpfs; LOAD httpfs;')
from huggingface_hub import hffs
index_files = list(hffs.glob("buckets/commoncrawl/commoncrawl/cc-index/table/cc-main/warc/crawl=*/subset=*/part-00000-*.parquet"))
print(f"Found {len(index_files):,} index files on HF bucket")
http_index_files = [f.replace("buckets/commoncrawl/commoncrawl/", "https://huggingface.co/buckets/commoncrawl/commoncrawl/resolve/") for f in index_files]
duckdb.sql(
f"SELECT crawl, subset, COUNT(*) FROM read_parquet({http_index_files!r}, hive_partitioning = true) GROUP BY crawl, subset"
).show()
duckdb.sql(f"""SELECT COUNT(*) AS count,
url_host_registered_domain
FROM read_parquet({http_index_files!r}, hive_partitioning = true)
WHERE crawl = 'CC-MAIN-2026-17' AND subset = 'warc'
AND url_host_tld = 'ru'
GROUP BY url_host_registered_domain
HAVING (COUNT(*) >= 100)
ORDER BY count DESC""").show()
The last query in the Python script will produce something like this (it only represents parts of the crawl since we query only selected index files):
┌────────┬────────────────────────────┐
│ count │ url_host_registered_domain │
│ int64 │ varchar │
├────────┼────────────────────────────┤
│ 166381 │ yandex.ru │
│ 28067 │ zr.ru │
│ 23968 │ zin.ru │
│ 18272 │ zarplata.ru │
│ 16030 │ zab.ru │
│ 13283 │ yugzone.ru │
│ 12003 │ yapl.ru │
...
Download speed #
We conducted initial experiments to evaluate the download speed for extracting individual records from the crawl archive. The evaluation involved fetching homepage URLs, i.e., sending many small range requests to the download server. We ran the experiment using CDX Toolkit and with various client-server combinations like down from S3 or HF bucket to an US-edge or EU-edge client but also to cloud-based clients.
The results showed that from S3 to S3 (us-east-1) achieved the highest download speed (up to 3,500 records/s), whereas from HF to HF was up to 1,500 records/s. Down from the HF bucket to an edge client was significantly slower due to rate limiting and the many small range requests (< 100 records/s). Note that these experiments provide preliminary findings and should not be treated as a definitive benchmark. We strongly recommend conducting your own measurements before committing to large data transfers.
Jupyter Notebooks #
In our cc-notebooks repository on GitHub, you can find ready-to-use Jupyter notebooks for the examples in this blog post and more:
- cc-index-hf.ipynb : Common Crawl's URL Index via Hugging Face
- s3-hf.ipynb : Using Common Crawl data via Hugging Face's S3-compatible gateway
- warcio-hf.ipynb : Reading WARC files from Hugging Face
We welcome feedback on the Common Crawl Hugging Face Bucket. Please contact us on our Discord or Google Group.