{"slug": "download-speed-way-too-slow", "title": "Download speed way too slow", "summary": "Hugging Face's `huggingface_hub` v1.1.x CLI can be tuned for faster large-file downloads by setting environment variables that raise Xet chunk concurrency and move the cache to an SSD, according to a step-by-step playbook citing Hugging Face's official environment-variable documentation. The guide recommends `HF_XET_HIGH_PERFORMANCE=1` and `HF_XET_NUM_CONCURRENT_RANGE_GETS=24` (up from the default 16, with 24–32 suggested on fast SSD/CPU machines and 8–16 on weaker hardware), plus `HF_HUB_DOWNLOAD_TIMEOUT=60` to replace the 10-second default. For spinning disks it advises `HF_XET_RECONSTRUCT_WRITE_SEQUENTIALLY=1` to avoid thrashing from parallel random writes, and it notes that `hf download --include` with `--local-dir` avoids pulling entire repositories.", "body_md": "Well, for now, the method to set the HF CLI to the absolute fastest mode is currently as follows. Since the method for setting environment variables varies by OS, replace the `export` part according to your environment.\n\nBelow is a focused, step-by-step playbook that actually increases Hugging Face (HF) CLI download speed, with background for each step and concrete commands. The knobs and commands are current as of v1.1.x of `huggingface_hub`.\n\n# \n\nLarge files on the Hub are served via a CDN and, by default, downloaded through the HF CLI using the Rust-based `hf-xet` path. Xet reconstructs files from chunks; this favors SSD/NVMe and high concurrency, and it exposes environment variables you can tune. Route/POP (region) variability also matters, so the same 500 Mb/s line can feel “very fast” or “very slow” depending on path and local I/O. ([Hugging Face](https://huggingface.co/docs/huggingface_hub/main/en/package_reference/environment_variables))\n\n# \n\n**Why:** Eliminates a common disk bottleneck and ensures you have recent CLI/Xet behavior. `HF_HOME` controls where the Hub cache lives. ([Hugging Face](https://huggingface.co/docs/huggingface_hub/main/en/package_reference/environment_variables))\n\n```\n# References:\n# - HF env vars (HF_HOME, HF_HUB_CACHE, timeouts, Xet knobs): https://huggingface.co/docs/huggingface_hub/en/package_reference/environment_variables\n# - CLI install/usage: https://huggingface.co/docs/huggingface_hub/en/guides/cli\npython -m pip install -U \"huggingface_hub\" hf_xet\nexport HF_HOME=\"/fast-ssd/.cache/huggingface\"     # move cache to SSD/NVMe\nexport HF_HUB_DOWNLOAD_TIMEOUT=60                 # reduce spurious timeouts on slow routes\n```\n\n`HF_HOME` sets the base cache dir; `HF_HUB_DOWNLOAD_TIMEOUT` raises the per-request timeout from the 10 s default to something more forgiving. ([Hugging Face](https://huggingface.co/docs/huggingface_hub/main/en/package_reference/environment_variables))\n\n# \n\n**Why:** The Xet backend exposes controls that directly impact throughput. High-perf mode tries to saturate network/CPU; the per-file concurrency (`HF_XET_NUM_CONCURRENT_RANGE_GETS`) increases parallel chunk reads. Defaults are conservative (16). ([Hugging Face](https://huggingface.co/docs/huggingface_hub/main/en/package_reference/environment_variables))\n\n```\n# References (HF_XET_HIGH_PERFORMANCE, HF_XET_NUM_CONCURRENT_RANGE_GETS):\n# https://huggingface.co/docs/huggingface_hub/en/package_reference/environment_variables\nexport HF_XET_HIGH_PERFORMANCE=1\nexport HF_XET_NUM_CONCURRENT_RANGE_GETS=24   # try 24–32 on fast SSD/CPU; 8–16 on weaker machines\n```\n\nThese variables are officially documented; raising concurrency helps if you have headroom. ([Hugging Face](https://huggingface.co/docs/huggingface_hub/main/en/package_reference/environment_variables))\n\n# \n\n**Why:** Parallel random writes that help SSDs can thrash spinning disks. The Xet toggle below switches to sequential writes and is specifically recommended for HDDs. ([Hugging Face](https://huggingface.co/docs/huggingface_hub/main/en/package_reference/environment_variables))\n\n```\n# Reference: HF_XET_RECONSTRUCT_WRITE_SEQUENTIALLY (HDD-only)\n# https://huggingface.co/docs/huggingface_hub/en/package_reference/environment_variables\nexport HF_XET_RECONSTRUCT_WRITE_SEQUENTIALLY=1\n```\n\n# `--include` and `--local-dir`\n\n**Why:** Avoids pulling entire repos and keeps writes local to your target folder. The CLI implements patterns and a direct “download-to-folder” mode. ([Hugging Face](https://huggingface.co/docs/huggingface_hub/en/guides/cli))\n\n```\n# References:\n# - CLI download patterns & --local-dir: https://huggingface.co/docs/huggingface_hub/en/guides/cli\n# - Download guide (hf_hub_download/snapshot_download): https://huggingface.co/docs/huggingface_hub/en/guides/download\nhf download Comfy-Org/Qwen-Image-Edit_ComfyUI \\\n  --include \"split_files/diffusion_models/qwen_image_edit_fp8_e4m3fn.safetensors\" \\\n  --local-dir \"/path/to/ComfyUI/models/diffusion_models\"\n```\n\n# \n\n**Why:** Isolate whether Xet routing/behavior is the issue on your machine/route. You can explicitly disable Xet; set this **before** importing or invoking HF code so it’s honored. ([Hugging Face](https://huggingface.co/docs/huggingface_hub/main/en/package_reference/environment_variables))\n\n```\n# References:\n# - HF_HUB_DISABLE_XET (must be set prior to import/CLI): https://huggingface.co/docs/huggingface_hub/en/package_reference/environment_variables\n# - Note about setting it before use: https://github.com/huggingface/huggingface_hub/issues/3266\nexport HF_HUB_DISABLE_XET=1\nhf download <repo> --include \"<file>\" --local-dir \"<dir>\"\n```\n\n# \n\n**Why:** When a single TCP stream underperforms on your current path, multiple HTTP range connections often achieve higher steady throughput. `aria2c` is a simple, robust option. ([DEV Community](https://dev.to/susumuota/faster-and-more-reliable-hugging-face-downloads-using-aria2-and-gnu-parallel-4f2b))\n\n```\n# Reference/recipe: https://dev.to/susumuota/faster-and-more-reliable-hugging-face-downloads-using-aria2-and-gnu-parallel-4f2b\n# 1) Get the “Copy download link” (?download=true) from the file’s page on the Hub\naria2c -x16 -s16 -j4 -c \\\n \"https://huggingface.co/<repo>/resolve/main/<path/to/large-file>?download=true\"\n# -x connections per server; -s splits per file; -j parallel jobs; -c resume\n```\n\n# \n\n**Why:** The Hub’s CDN and peering vary by POP and time. Toggling VPN on/off or switching to a nearby region often changes anemic 1–2 MB/s to tens of MB/s. Forum reports and timeout traces on `cdn-lfs*.hf.co` illustrate this effect. ([Hugging Face Forums](https://discuss.huggingface.co/t/downloads-intermittently-fail-403-low-bandwidth/140198))\n\n# \n\n**Why:** Many older posts recommend `hf_transfer`. In current `huggingface_hub` v1.x, **`hf_transfer` is removed and `HF_HUB_ENABLE_HF_TRANSFER` is ignored**. Use the Xet knobs (`HF_XET_HIGH_PERFORMANCE`, etc.) instead. ([Hugging Face](https://huggingface.co/docs/huggingface_hub/en/concepts/migration))\n\n## \n\nThis quickly tells you where the bottleneck is on your system/route.\n\n```\n# 0) Prep\npython -m pip install -U \"huggingface_hub\" hf_xet  # https://huggingface.co/docs/huggingface_hub/en/guides/cli\nexport HF_HOME=\"/fast-ssd/.cache/huggingface\"      # https://huggingface.co/docs/huggingface_hub/en/package_reference/environment_variables\n\n# 1) CLI with tuned Xet (expect best performance on SSD)\nexport HF_XET_HIGH_PERFORMANCE=1                   # https://huggingface.co/docs/huggingface_hub/en/package_reference/environment_variables\nexport HF_XET_NUM_CONCURRENT_RANGE_GETS=24\nhf download <repo> --include \"<bigfile>\" --local-dir \"<dir>\"\n\n# 2) Same CLI, but Xet off (if #1 was spiky)\nexport HF_HUB_DISABLE_XET=1                        # set BEFORE running HF code\nhf download <repo> --include \"<bigfile>\" --local-dir \"<dir>\"\n\n# 3) Multi-connection HTTP (expect steady speeds if single-stream path is weak)\naria2c -x16 -s16 -j4 -c \"<direct ?download=true URL>\"  # https://dev.to/susumuota/faster-and-more-reliable-hugging-face-downloads-using-aria2-and-gnu-parallel-4f2b\n```\n\n**Interpretation:**\n\n• #1 fast and stable → keep Xet high-perf + SSD.\n\n• #1 spiky but #2 steady → Xet/route interaction on this box; run with Xet disabled here. ([GitHub](https://github.com/huggingface/huggingface_hub/issues/3266))\n\n• Both slow but `aria2c` fast → single-stream/POP issue; keep CLI for convenience but prefer multi-connection pulls for huge files. ([DEV Community](https://dev.to/susumuota/faster-and-more-reliable-hugging-face-downloads-using-aria2-and-gnu-parallel-4f2b))\n\n• All slow → switch region (VPN/off-VPN) and retry. ([Hugging Face Forums](https://discuss.huggingface.co/t/downloads-intermittently-fail-403-low-bandwidth/140198))\n\n## \n\n- **Downloading whole repos by accident.** Use`--include` to pull only the large file(s) you need. ([Hugging Face](https://huggingface.co/docs/huggingface_hub/en/guides/cli) )\n- **Tiny default timeouts.** Raise`HF_HUB_DOWNLOAD_TIMEOUT` to reduce intermittent read timeouts on`cdn-lfs*.hf.co` . ([Hugging Face](https://huggingface.co/docs/huggingface_hub/en/guides/cli) )\n- **HDD as destination.** If you must, add`HF_XET_RECONSTRUCT_WRITE_SEQUENTIALLY=1` . ([Hugging Face](https://huggingface.co/docs/huggingface_hub/main/en/package_reference/environment_variables) )\n- **Expecting `hf_transfer` to help.** It’s gone in v1.x; use Xet high-perf instead. ([Hugging Face](https://huggingface.co/docs/huggingface_hub/en/concepts/migration) )\n\n## \n\n**Official docs**\n\n- Environment variables (HF_HOME/HF_HUB_CACHE, timeouts, Xet knobs, disable Xet). Useful to confirm every variable above. ([Hugging Face](https://huggingface.co/docs/huggingface_hub/main/en/package_reference/environment_variables) )\n- CLI guide (`hf download` ,`--include` ,`--local-dir` , timeouts). Step-by-step usage examples. ([Hugging Face](https://huggingface.co/docs/huggingface_hub/en/guides/cli) )\n- Migration note: `hf_transfer` removed;`HF_HUB_ENABLE_HF_TRANSFER` ignored; use`HF_XET_HIGH_PERFORMANCE` . ([Hugging Face](https://huggingface.co/docs/huggingface_hub/en/concepts/migration) )\n\n**GitHub issues (real-world behavior)**\n\n- “HF_HUB_DISABLE_XET not disabling unless set before import” — clarifies when the toggle takes effect. ([GitHub](https://github.com/huggingface/huggingface_hub/issues/3266) )\n- “Large downloads stuck/slow with VPN” — intermittent slow/0% cases on big files with CLI. ([GitHub](https://github.com/huggingface/huggingface_hub/issues/2691) )\n\n**Community threads/recipes**\n\n- Intermittent slow speeds, disconnects, and region changes as fixes. Background on route/POP sensitivity. ([Hugging Face Forums](https://discuss.huggingface.co/t/downloads-intermittently-fail-403-low-bandwidth/140198) )\n- Practical `aria2c` recipe for multi-connection, resumable downloads. ([DEV Community](https://dev.to/susumuota/faster-and-more-reliable-hugging-face-downloads-using-aria2-and-gnu-parallel-4f2b) )", "url": "https://wpnews.pro/news/download-speed-way-too-slow", "canonical_source": "https://discuss.huggingface.co/t/download-speed-way-too-slow/169824#post_15", "published_at": "2026-10-08 20:17:58+00:00", "updated_at": "2026-10-08 20:47:07.085358+00:00", "lang": "en", "topics": ["ai-tools", "developer-tools", "ai-infrastructure"], "entities": ["Hugging Face", "huggingface_hub", "hf_xet", "HF_HOME", "HF_HUB_DOWNLOAD_TIMEOUT", "HF_XET_HIGH_PERFORMANCE", "HF_XET_NUM_CONCURRENT_RANGE_GETS", "HF_XET_RECONSTRUCT_WRITE_SEQUENTIALLY"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/download-speed-way-too-slow", "markdown": "https://wpnews.pro/news/download-speed-way-too-slow.md", "text": "https://wpnews.pro/news/download-speed-way-too-slow.txt", "jsonld": "https://wpnews.pro/news/download-speed-way-too-slow.jsonld"}}