HuggingFace's Large File Downloads Keep Stopping — Resuming with curl for Reliable Retrieval

📝 Originally published (in Japanese) at

[forge.workstyle.tech].

When developing a voice conversion app, I encountered an issue while trying to download the 44.1kHz Seed-VC model set (DiT weights, rmvpe, and BigVGAN vocoder) using huggingface_hub

.

When attempting to download models using

huggingface_hub

, the transfer would stop halfway through large files (hundreds of MB) and never progress.

While small config.json

files would download quickly, larger .pth

or .pt

files would get stuck at a certain point and freeze. This article describes how to use curl's stagnation detection and automatic resumption to ensure successful downloads, and create a custom Hugging Face cache structure to enable model reading even when HF_HUB_OFFLINE=1

.

On my machine, large file transfers using hf_hub_download

would stall halfway through. The suspected causes include IPv6 route unavailability and connection drops over extended periods. The issues are:

This isn't a matter of a slow network, but rather large transfers dying under specific conditions, so simply increasing the retry count won't solve the problem.

The appeal of huggingface_hub

lies in its cache management, not just as a down. load_custom_model_from_hf

and from_pretrained

assume a specific directory structure (blob entities + symbolic links + refs) and will re-download the model if this structure is broken.

To address this, I adopted a two-step approach:

hf_hub

)Metadata (etag, commit hash, size, and entity URL) can be retrieved from the huggingface_hub

API to construct the cache structure.

The key is combining curl options:

subprocess.run(
    ["curl", "-L", "--fail", "-C", "-",
     "--retry", "50", "--retry-delay", "3", "--retry-all-errors",
     "--speed-time", "20", "--speed-limit", "2000",   # 20 seconds at 2KB/s or lower, consider transfer failed and retry
     "-o", blob, loc],
    check=True,
)

The roles of each option are as follows:

--speed-time 20 --speed-limit 2000

— -C -

— --retry 50 --retry-delay 3 --retry-all-errors

— Retry after 3 seconds, up to 50 times. Transfers cut off by --speed-time

are also retried.-L

(follow redirects) and --fail

(non-zero exit on HTTP errors).This combination automatically loops through "stagnation → cut → resume" and eventually completes the transfer. This was the most reliable method in environments where large files would freeze.

The core of the Hugging Face cache (e.g., ~/.cache/huggingface/hub

) has the following structure per repository:

models--{org}--{repo}/
├── blobs/
│   └── {etag}                      # File entity, named by etag (content hash)
├── snapshots/
│   └── {commit_hash}/
│       └── {filename}              # Symbolic link to blob
└── refs/
    └── main                        # Branch name → commit hash text

The key points are:

blobs/

with etag namessnapshots/{commit}/

contains symbolic linksrefs/main

contains the commit hashrevision="main"

and allows hf_hub

to find the correct snapshot even when HF_HUB_OFFLINE=1

.Metadata can be obtained from the huggingface_hub

API:

from huggingface_hub import hf_hub_url, get_hf_file_metadata

url = hf_hub_url(repo_id, filename, revision=revision)
m = get_hf_file_metadata(url)
etag = (m.etag or "").strip('"')   # Becomes the blob file name
commit = m.commit_hash or revision # Becomes the snapshots/ directory name
size = m.size                       # Used for download completion detection
loc = m.location or url             # Entity URL (redirected)

The rest involves using curl to download the entity to blobs/{etag}

, creating a relative symbolic link from snapshots/{commit}/{filename}

, and writing the commit to refs/main

. The relative link ensures the cache remains intact even when moved to another machine.

os.symlink(os.path.relpath(blob, os.path.dirname(snap)), snap)
with open(os.path.join(refsdir, "main"), "w") as f:
    f.write(commit)

If the size is known, checking it against the existing blob size and skipping if complete can make re-runs idempotent.

Some repositories, like BigVGAN, contain large unnecessary files for inference (e.g., discriminator or optimizer states). Down the entire repository can result in getting stuck with the largest file. Specify the necessary files explicitly to avoid this.

Failing to create refs/main

alongside snapshots

and blobs

will prevent hf_hub

from resolving the correct commit when HF_HUB_OFFLINE=1

, even if the cache exists. Always create the three-part set (blob, snapshot link, and refs).

Even with a cache, hf_hub

performs a HEAD request at startup to check for updates (without re-down large files). To ensure complete offline functionality, set HF_HUB_OFFLINE=1

or TRANSFORMERS_OFFLINE=1

. Conversely, if you want to see updates but prevent downloads, you can leave it online and just allow HEAD requests.

hf_hub

transfers --speed-time

and --speed-limit

to -C -

with --retry

for resumption.blobs/{etag}

• snapshots/{commit}/

with relative symbolic links + refs/main

).get_hf_file_metadata

.refs/main

is created to enable HF_HUB_OFFLINE=1

.

source & further reading

dev.to — original article Token Cost Optimization: The Complete Guide to Building Cost-Efficient LLM Applications Stop Sending Your Health Data to the Cloud: Build a Private AI Health Assistant with Llama-3 and MLX OpenAI's Astra Solved 10 Open Math Problems — and the Price Tag Is the Real Story