HuggingFace's Large File Downloads Keep Stopping — Resuming with curl for Reliable Retrieval

A developer at Workstyle encountered stalled downloads of large model files from Hugging Face using huggingface_hub, and devised a workaround using curl's stagnation detection and automatic resumption. The solution also constructs a custom Hugging Face cache structure to enable offline model loading with HF_HUB_OFFLINE=1.

📝 Originally published in Japanese at forge.workstyle.tech . When developing a voice conversion app, I encountered an issue while trying to download the 44.1kHz Seed-VC model set DiT weights, rmvpe, and BigVGAN vocoder using huggingface hub . When attempting to download models using huggingface hub , the transfer would stop halfway through large files hundreds of MB and never progress. While small config.json files would download quickly, larger .pth or .pt files would get stuck at a certain point and freeze. This article describes how to use curl's stagnation detection and automatic resumption to ensure successful downloads, and create a custom Hugging Face cache structure to enable model reading even when HF HUB OFFLINE=1 . On my machine, large file transfers using hf hub download would stall halfway through. The suspected causes include IPv6 route unavailability and connection drops over extended periods. The issues are: This isn't a matter of a slow network, but rather large transfers dying under specific conditions, so simply increasing the retry count won't solve the problem. The appeal of huggingface hub lies in its cache management , not just as a downloader. load custom model from hf and from pretrained assume a specific directory structure blob entities + symbolic links + refs and will re-download the model if this structure is broken. To address this, I adopted a two-step approach: hf hub Metadata etag, commit hash, size, and entity URL can be retrieved from the huggingface hub API to construct the cache structure. The key is combining curl options: subprocess.run "curl", "-L", "--fail", "-C", "-", "--retry", "50", "--retry-delay", "3", "--retry-all-errors", "--speed-time", "20", "--speed-limit", "2000", 20 seconds at 2KB/s or lower, consider transfer failed and retry "-o", blob, loc , check=True, The roles of each option are as follows: --speed-time 20 --speed-limit 2000 — -C - — --retry 50 --retry-delay 3 --retry-all-errors — Retry after 3 seconds, up to 50 times. Transfers cut off by --speed-time are also retried. -L follow redirects and --fail non-zero exit on HTTP errors .This combination automatically loops through "stagnation → cut → resume" and eventually completes the transfer. This was the most reliable method in environments where large files would freeze. The core of the Hugging Face cache e.g., ~/.cache/huggingface/hub has the following structure per repository: models--{org}--{repo}/ ├── blobs/ │ └── {etag} File entity, named by etag content hash ├── snapshots/ │ └── {commit hash}/ │ └── {filename} Symbolic link to blob └── refs/ └── main Branch name → commit hash text The key points are: blobs/ with etag names snapshots/{commit}/ contains symbolic links refs/main contains the commit hash revision="main" and allows hf hub to find the correct snapshot even when HF HUB OFFLINE=1 .Metadata can be obtained from the huggingface hub API: python from huggingface hub import hf hub url, get hf file metadata url = hf hub url repo id, filename, revision=revision m = get hf file metadata url etag = m.etag or "" .strip '"' Becomes the blob file name commit = m.commit hash or revision Becomes the snapshots/ directory name size = m.size Used for download completion detection loc = m.location or url Entity URL redirected The rest involves using curl to download the entity to blobs/{etag} , creating a relative symbolic link from snapshots/{commit}/{filename} , and writing the commit to refs/main . The relative link ensures the cache remains intact even when moved to another machine. os.symlink os.path.relpath blob, os.path.dirname snap , snap with open os.path.join refsdir, "main" , "w" as f: f.write commit If the size is known, checking it against the existing blob size and skipping if complete can make re-runs idempotent. Some repositories, like BigVGAN, contain large unnecessary files for inference e.g., discriminator or optimizer states . Downloading the entire repository can result in getting stuck with the largest file. Specify the necessary files explicitly to avoid this. Failing to create refs/main alongside snapshots and blobs will prevent hf hub from resolving the correct commit when HF HUB OFFLINE=1 , even if the cache exists. Always create the three-part set blob, snapshot link, and refs . Even with a cache, hf hub performs a HEAD request at startup to check for updates without re-downloading large files . To ensure complete offline functionality, set HF HUB OFFLINE=1 or TRANSFORMERS OFFLINE=1 . Conversely, if you want to see updates but prevent downloads, you can leave it online and just allow HEAD requests. hf hub transfers --speed-time and --speed-limit to -C - with --retry for resumption. blobs/{etag} + snapshots/{commit}/ with relative symbolic links + refs/main . get hf file metadata . refs/main is created to enable HF HUB OFFLINE=1 .