{"slug": "common-crawl-data-stored-on-a-hugging-face-bucket", "title": "Common Crawl Data Stored on a Hugging Face Bucket", "summary": "Common Crawl has mirrored selected crawl archives to a Hugging Face Storage Bucket since April 2026, alongside its existing AWS S3 distribution, using the same directory layout such as crawl-data/CC-MAIN-2026-17/. The bucket supports S3-like object storage with CDN pre-warming in several regions, and users can access it via the Hugging Face CLI, mount it as a local filesystem with hf-mount, or point existing S3 pipelines at the S3-compatible API. Common Crawl says the Hugging Face hosting lets users apply Hugging Face ecosystem tools, including the fsspec-compatible hf:// protocol and the warcio WARC reader, to large-scale crawl data.", "body_md": "Since April 2026, parts of our crawl archive have been available on a [Hugging Face Storage Bucket](https://huggingface.co/buckets/commoncrawl/commoncrawl), in addition to being available on AWS S3. Having the data on the Hugging Face (HF) lets you use the tools available in the HF ecosystem, making large-scale crawl data easier to access. This blog post uses practical examples to show you how to get started.\n\n## **Accessing the data through Hugging Face Storage Buckets**\n\nSelected crawl archives are mirrored in a [Hugging Face Storage Bucket](https://huggingface.co/buckets/commoncrawl/commoncrawl), using the same directory layout as S3 (e.g. crawl-data/CC-MAIN-2026-17/). See the [bucket README](https://huggingface.co/buckets/commoncrawl/commoncrawl) for a list of currently available crawls. Storage Buckets provide S3-like object storage on the Hugging Face Hub. You can browse files in the browser, transfer data with the hf CLI, or mount the bucket as a local filesystem. The bucket has [CDN pre-warming](https://huggingface.co/docs/hub/storage-buckets#pre-warming-and-cdn) enabled in several regions, which can improve read throughput when your compute runs nearby. For more detail, see the [announcement blog post](https://commoncrawl.org/blog/april-2026-crawl-archive-now-available-in-a-hugging-face-storage-bucket).\n\n### Using the Hugging Face CLI\n\nInstall the [Hugging Face CLI](https://huggingface.co/docs/huggingface_hub/guides/cli), then list or download files:\n\n```\nhf buckets list commoncrawl/commoncrawl/crawl-data/CC-MAIN-2026-17\nhf buckets cp hf://buckets/commoncrawl/commoncrawl/crawl-data/CC-MAIN-2026-17/segments/.../warc/CC-MAIN-....warc.gz ./local-path/\n```\n\n### Mounting the bucket as a local filesystem\n\nInstall [hf-mount](https://github.com/huggingface/hf-mount), then mount the bucket. Files are fetched on read, so any local tool can access the data without downloading the full archive first:\n\n```\nhf-mount start bucket commoncrawl/commoncrawl /mnt/commoncrawl\nls /mnt/commoncrawl/crawl-data/CC-MAIN-2026-17/\n```\n\nSee the [access patterns guide](https://huggingface.co/docs/hub/storage-buckets-access#mount-as-a-local-filesystem) for backend options and caching.\n\n### Using an existing S3 pipeline\n\nKeep your current code or command and point to the S3-compatible API for buckets. Data will be fetched using the same client but from Hugging Face instead of AWS. See the [S3 compatibility documentation](https://huggingface.co/docs/hub/storage-buckets-s3) for more info.\n\n## **Reading WARC files from Hugging Face**\n\nHugging Face provides file-system-like access to models, datasets, and buckets via the `hf://` protocol, i.e., a [pythonic fsspec-compatible file interface to the Hugging Face Hub](https://huggingface.co/docs/huggingface_hub/main/en/guides/hf_file_system). The WARC reader/writer tool [warcio](https://github.com/webrecorder/warcio) fully supports [fsspec](https://filesystem-spec.readthedocs.io/en/latest/). Make sure to install the optional warcio dependencies and the [huggingface_hub](https://pypi.org/project/huggingface-hub/) package.\n\n``` python\n# Read a WARC file via fsspec and hf:// protocol\nimport fsspec\nfrom warcio.archiveiterator import ArchiveIterator\n\nwith fsspec.open('hf://buckets/commoncrawl/commoncrawl/crawl-data/CC-MAIN-2026-17/segments/1775805908305.14/warc/CC-MAIN-20260410081153-20260410111153-00000.warc.gz', 'rb') as stream:\n    for i, record in enumerate(ArchiveIterator(stream)):\n        if i >= 10:\n            break\n\n        print(record.rec_type)\n\n        if record.rec_type == 'response':\n            print(record.rec_headers.get_header('WARC-Target-URI'))\n```\n\nThe code will print the first ten WARC record types and the URLs of response records:\n\n```\nwarcinfo\nrequest\nresponse\nhttp://003ms.ru/catalog/lekarstvennye-sredstva/nervnaya-sistema/antigrippin-312/tabletki-250-mgplus3-mgplus50-mg-dlya-detej-30-shtuk-shipuchie\nmetadata\nrequest\n...\n```\n\nYou can also use the `warcio` CLI directly to read from the HF bucket:\n\n```\n# Note: This command will trigger a broken pipe error after 10 records due to `head -n 10`\nwarcio index hf://buckets/commoncrawl/commoncrawl/crawl-data/CC-MAIN-2026-17/segments/1775805908305.14/warc/CC-MAIN-20260410081153-20260410111153-00000.warc.gz -f offset,content-type,http:content-type,warc-target-uri | head -n 10\n```\n\n## **Querying the URL index via Hugging Face**\n\n[Common Crawl's URL Index](https://commoncrawl.org/url-index) (previously known as the Columnar Index) is one of the indexes available for querying the Common Crawl corpus. As the name suggests, it is an index to the WARC files and URLs in the Common Crawl corpus, stored in a columnar format (Apache Parquet). This format is suited to efficient analytical and/or bulk queries of the data, saving time and computing resources. The index files are also available on the [Common Crawl HF bucket](https://huggingface.co/buckets/commoncrawl/commoncrawl). [DuckDB](https://duckdb.org/) can be used to query the index without downloading all the data.\n\n### Index schema\n\nLet's first look at the schema of the index:\n\n```\nduckdb -c \"DESCRIBE FROM read_parquet('https://huggingface.co/buckets/commoncrawl/commoncrawl/resolve/cc-index/table/cc-main/warc/crawl=CC-MAIN-2026-17/subset=warc/part-00000-ee7620e8-27a7-4c74-846b-a92b039dae92.c000.gz.parquet')\"\n```\n\nThe DuckDB CLI will print the schema of the URL index:\n\n```\n┌─────────────────────────────────────────────────────┐\n│                      Describe                       │\n│                                                     │\n│ url_surtkey                varchar                  │\n│ url                        varchar                  │\n│ url_host_name              varchar                  │\n│ url_host_tld               varchar                  │\n│ url_host_2nd_last_part     varchar                  │\n│ url_host_3rd_last_part     varchar                  │\n│ url_host_4th_last_part     varchar                  │\n│ url_host_5th_last_part     varchar                  │\n│ url_host_registry_suffix   varchar                  │\n│ url_host_registered_domain varchar                  │\n│ url_host_private_suffix    varchar                  │\n...\n```\n\n### Querying the URL index with Python\n\nIn addition to the CLI, you can also use the DuckDB Python client to run queries:\n\n``` python\nimport duckdb\n\nduckdb.sql('INSTALL httpfs; LOAD httpfs;')\n\n# The index uses hive-partitioning for `crawl` and `subset` which are automatically constructed if you use the S3 index.\n# However, we are using HF via HTTP so we need to manually construct the partitions using glob from the HF API.\n\nfrom huggingface_hub import hffs\n\n# NOTE: We are using only a sample of all files (part-00000-*)\n\nindex_files = list(hffs.glob(\"buckets/commoncrawl/commoncrawl/cc-index/table/cc-main/warc/crawl=*/subset=*/part-00000-*.parquet\"))\n\nprint(f\"Found {len(index_files):,} index files on HF bucket\")\n\n# Rewrite bucket paths to HTTP URLs\nhttp_index_files = [f.replace(\"buckets/commoncrawl/commoncrawl/\", \"https://huggingface.co/buckets/commoncrawl/commoncrawl/resolve/\") for f in index_files]\n\n# Let's see how large our sample is\n\nduckdb.sql(\n    f\"SELECT crawl, subset, COUNT(*) FROM read_parquet({http_index_files!r}, hive_partitioning = true) GROUP BY crawl, subset\"\n).show()\n\n# Time to run the first query! This query counts the number of pages per domain within a single top-level domain, .ru (Russia):\n\nduckdb.sql(f\"\"\"SELECT COUNT(*) AS count,\n       url_host_registered_domain\nFROM read_parquet({http_index_files!r}, hive_partitioning = true) \nWHERE crawl = 'CC-MAIN-2026-17' AND subset = 'warc'\n  AND url_host_tld = 'ru'\nGROUP BY  url_host_registered_domain\nHAVING (COUNT(*) >= 100)\nORDER BY  count DESC\"\"\").show()\n```\n\nThe last query in the Python script will produce something like this (it only represents parts of the crawl since we query only selected index files):\n\n```\n┌────────┬────────────────────────────┐\n│ count  │ url_host_registered_domain │\n│ int64  │          varchar           │\n├────────┼────────────────────────────┤\n│ 166381 │ yandex.ru                  │\n│  28067 │ zr.ru                      │\n│  23968 │ zin.ru                     │\n│  18272 │ zarplata.ru                │\n│  16030 │ zab.ru                     │\n│  13283 │ yugzone.ru                 │\n│  12003 │ yapl.ru                    │\n...\n```\n\n## **Download speed**\n\nWe conducted initial experiments to evaluate the download speed for extracting individual records from the crawl archive. The evaluation involved fetching homepage URLs, i.e., sending many small range requests to the download server. We ran the experiment using [CDX Toolkit](https://github.com/commoncrawl/cdx_toolkit) and with various client-server combinations like downloading from S3 or HF bucket to an US-edge or EU-edge client but also to cloud-based clients. \n\nThe results showed that from S3 to S3 (us-east-1) achieved the highest download speed (up to 3,500 records/s), whereas from HF to HF was up to 1,500 records/s. Downloading from the HF bucket to an edge client was significantly slower due to rate limiting and the many small range requests (< 100 records/s). Note that these experiments provide preliminary findings and should not be treated as a definitive benchmark. We strongly recommend conducting your own measurements before committing to large data transfers.\n\n## **Jupyter Notebooks**\n\nIn our [cc-notebooks repository](https://github.com/commoncrawl/cc-notebooks/) on GitHub, you can find ready-to-use Jupyter notebooks for the examples in this blog post and more:\n\n- [cc-index-hf.ipynb](https://github.com/commoncrawl/cc-notebooks/blob/main/cc-huggingface/cc-index-hf.ipynb) : Common Crawl's URL Index via Hugging Face\n- [s3-hf.ipynb](https://github.com/commoncrawl/cc-notebooks/blob/main/cc-huggingface/s3-hf.ipynb) : Using Common Crawl data via Hugging Face's S3-compatible gateway\n- [warcio-hf.ipynb](https://github.com/commoncrawl/cc-notebooks/blob/main/cc-huggingface/warcio-hf.ipynb) : Reading WARC files from Hugging Face\n\nWe welcome feedback on the Common Crawl Hugging Face Bucket.  Please contact us on our [Discord](https://discord.gg/njaVFh7avF) or [Google Group](https://discord.gg/njaVFh7avF).\n\n[Malte Ostendorff](https://commoncrawl.org/team/malte-ostendorff)\n\n## Erratum:\n\n## Content is truncated\n\n[More details](https://commoncrawl.org/errata/content-is-truncated)", "url": "https://wpnews.pro/news/common-crawl-data-stored-on-a-hugging-face-bucket", "canonical_source": "https://commoncrawl.org/blog/getting-started-with-common-crawl-data-on-hugging-face", "published_at": "2026-09-16 21:42:31+00:00", "updated_at": "2026-09-16 21:54:01.744545+00:00", "lang": "en", "topics": ["ai-crawlers", "structured-data", "ai-infrastructure"], "entities": ["Common Crawl", "Hugging Face", "AWS S3", "hf-mount", "warcio", "fsspec", "huggingface_hub", "CC-MAIN-2026-17"], "alternates": {"html": "https://wpnews.pro/news/common-crawl-data-stored-on-a-hugging-face-bucket", "markdown": "https://wpnews.pro/news/common-crawl-data-stored-on-a-hugging-face-bucket.md", "text": "https://wpnews.pro/news/common-crawl-data-stored-on-a-hugging-face-bucket.txt", "jsonld": "https://wpnews.pro/news/common-crawl-data-stored-on-a-hugging-face-bucket.jsonld"}}