{"slug": "huggingface-s-large-file-downloads-keep-stopping-resuming-with-curl-for-reliable", "title": "HuggingFace's Large File Downloads Keep Stopping — Resuming with curl for Reliable Retrieval", "summary": "A developer at Workstyle encountered stalled downloads of large model files from Hugging Face using huggingface_hub, and devised a workaround using curl's stagnation detection and automatic resumption. The solution also constructs a custom Hugging Face cache structure to enable offline model loading with HF_HUB_OFFLINE=1.", "body_md": "📝 Originally published (in Japanese) at\n\n[forge.workstyle.tech].\n\nWhen developing a voice conversion app, I encountered an issue while trying to download the 44.1kHz Seed-VC model set (DiT weights, rmvpe, and BigVGAN vocoder) using `huggingface_hub`\n\n.\n\nWhen attempting to download models using\n\n`huggingface_hub`\n\n, the transfer would stop halfway through large files (hundreds of MB) and never progress.\n\nWhile small `config.json`\n\nfiles would download quickly, larger `.pth`\n\nor `.pt`\n\nfiles would get stuck at a certain point and freeze. This article describes how to use **curl's stagnation detection and automatic resumption** to ensure successful downloads, and create a **custom Hugging Face cache structure** to enable model reading even when `HF_HUB_OFFLINE=1`\n\n.\n\nOn my machine, large file transfers using `hf_hub_download`\n\nwould stall halfway through. The suspected causes include IPv6 route unavailability and connection drops over extended periods. The issues are:\n\nThis isn't a matter of a slow network, but rather large transfers dying under specific conditions, so simply increasing the retry count won't solve the problem.\n\nThe appeal of `huggingface_hub`\n\nlies in its **cache management**, not just as a downloader. `load_custom_model_from_hf`\n\nand `from_pretrained`\n\nassume a specific directory structure (blob entities + symbolic links + refs) and will re-download the model if this structure is broken.\n\nTo address this, I adopted a two-step approach:\n\n`hf_hub`\n\n)Metadata (etag, commit hash, size, and entity URL) can be retrieved from the `huggingface_hub`\n\nAPI to construct the cache structure.\n\nThe key is combining curl options:\n\n```\nsubprocess.run(\n    [\"curl\", \"-L\", \"--fail\", \"-C\", \"-\",\n     \"--retry\", \"50\", \"--retry-delay\", \"3\", \"--retry-all-errors\",\n     \"--speed-time\", \"20\", \"--speed-limit\", \"2000\",   # 20 seconds at 2KB/s or lower, consider transfer failed and retry\n     \"-o\", blob, loc],\n    check=True,\n)\n```\n\nThe roles of each option are as follows:\n\n`--speed-time 20 --speed-limit 2000`\n\n— `-C -`\n\n— `--retry 50 --retry-delay 3 --retry-all-errors`\n\n— Retry after 3 seconds, up to 50 times. Transfers cut off by `--speed-time`\n\nare also retried.`-L`\n\n(follow redirects) and `--fail`\n\n(non-zero exit on HTTP errors).This combination automatically loops through \"stagnation → cut → resume\" and eventually completes the transfer. This was the most reliable method in environments where large files would freeze.\n\nThe core of the Hugging Face cache (e.g., `~/.cache/huggingface/hub`\n\n) has the following structure per repository:\n\n```\nmodels--{org}--{repo}/\n├── blobs/\n│   └── {etag}                      # File entity, named by etag (content hash)\n├── snapshots/\n│   └── {commit_hash}/\n│       └── {filename}              # Symbolic link to blob\n└── refs/\n    └── main                        # Branch name → commit hash text\n```\n\nThe key points are:\n\n`blobs/`\n\nwith etag names`snapshots/{commit}/`\n\ncontains symbolic links`refs/main`\n\ncontains the commit hash`revision=\"main\"`\n\nand allows `hf_hub`\n\nto find the correct snapshot even when `HF_HUB_OFFLINE=1`\n\n.Metadata can be obtained from the `huggingface_hub`\n\nAPI:\n\n``` python\nfrom huggingface_hub import hf_hub_url, get_hf_file_metadata\n\nurl = hf_hub_url(repo_id, filename, revision=revision)\nm = get_hf_file_metadata(url)\netag = (m.etag or \"\").strip('\"')   # Becomes the blob file name\ncommit = m.commit_hash or revision # Becomes the snapshots/ directory name\nsize = m.size                       # Used for download completion detection\nloc = m.location or url             # Entity URL (redirected)\n```\n\nThe rest involves using curl to download the entity to `blobs/{etag}`\n\n, creating a **relative symbolic link** from `snapshots/{commit}/{filename}`\n\n, and writing the commit to `refs/main`\n\n. The relative link ensures the cache remains intact even when moved to another machine.\n\n```\nos.symlink(os.path.relpath(blob, os.path.dirname(snap)), snap)\nwith open(os.path.join(refsdir, \"main\"), \"w\") as f:\n    f.write(commit)\n```\n\nIf the size is known, checking it against the existing blob size and skipping if complete can make re-runs idempotent.\n\nSome repositories, like BigVGAN, contain large unnecessary files for inference (e.g., discriminator or optimizer states). Downloading the entire repository can result in getting stuck with the largest file. **Specify the necessary files explicitly** to avoid this.\n\nFailing to create `refs/main`\n\nalongside `snapshots`\n\nand `blobs`\n\nwill prevent `hf_hub`\n\nfrom resolving the correct commit when `HF_HUB_OFFLINE=1`\n\n, even if the cache exists. Always create the **three-part set (blob, snapshot link, and refs)**.\n\nEven with a cache, `hf_hub`\n\nperforms a HEAD request at startup to check for updates (without re-downloading large files). To ensure complete offline functionality, set `HF_HUB_OFFLINE=1`\n\nor `TRANSFORMERS_OFFLINE=1`\n\n. Conversely, if you want to see updates but prevent downloads, you can leave it online and just allow HEAD requests.\n\n`hf_hub`\n\ntransfers `--speed-time`\n\nand `--speed-limit`\n\nto `-C -`\n\nwith `--retry`\n\nfor resumption.`blobs/{etag}`\n\n+ `snapshots/{commit}/`\n\nwith relative symbolic links + `refs/main`\n\n).`get_hf_file_metadata`\n\n.`refs/main`\n\nis created to enable `HF_HUB_OFFLINE=1`\n\n.", "url": "https://wpnews.pro/news/huggingface-s-large-file-downloads-keep-stopping-resuming-with-curl-for-reliable", "canonical_source": "https://dev.to/orca_forge/huggingfaces-large-file-downloads-keep-stopping-resuming-with-curl-for-reliable-retrieval-43ni", "published_at": "2026-08-04 00:37:44+00:00", "updated_at": "2026-08-04 01:39:11.658643+00:00", "lang": "en", "topics": ["developer-tools", "machine-learning", "ai-infrastructure"], "entities": ["Hugging Face", "curl", "Seed-VC", "Workstyle"], "alternates": {"html": "https://wpnews.pro/news/huggingface-s-large-file-downloads-keep-stopping-resuming-with-curl-for-reliable", "markdown": "https://wpnews.pro/news/huggingface-s-large-file-downloads-keep-stopping-resuming-with-curl-for-reliable.md", "text": "https://wpnews.pro/news/huggingface-s-large-file-downloads-keep-stopping-resuming-with-curl-for-reliable.txt", "jsonld": "https://wpnews.pro/news/huggingface-s-large-file-downloads-keep-stopping-resuming-with-curl-for-reliable.jsonld"}}