# HuggingFace's Large File Downloads Keep Stopping — Resuming with curl for Reliable Retrieval

> Source: <https://dev.to/orca_forge/huggingfaces-large-file-downloads-keep-stopping-resuming-with-curl-for-reliable-retrieval-43ni>
> Published: 2026-08-04 00:37:44+00:00

📝 Originally published (in Japanese) at

[forge.workstyle.tech].

When developing a voice conversion app, I encountered an issue while trying to download the 44.1kHz Seed-VC model set (DiT weights, rmvpe, and BigVGAN vocoder) using `huggingface_hub`

.

When attempting to download models using

`huggingface_hub`

, the transfer would stop halfway through large files (hundreds of MB) and never progress.

While small `config.json`

files would download quickly, larger `.pth`

or `.pt`

files would get stuck at a certain point and freeze. This article describes how to use **curl's stagnation detection and automatic resumption** to ensure successful downloads, and create a **custom Hugging Face cache structure** to enable model reading even when `HF_HUB_OFFLINE=1`

.

On my machine, large file transfers using `hf_hub_download`

would stall halfway through. The suspected causes include IPv6 route unavailability and connection drops over extended periods. The issues are:

This isn't a matter of a slow network, but rather large transfers dying under specific conditions, so simply increasing the retry count won't solve the problem.

The appeal of `huggingface_hub`

lies in its **cache management**, not just as a downloader. `load_custom_model_from_hf`

and `from_pretrained`

assume a specific directory structure (blob entities + symbolic links + refs) and will re-download the model if this structure is broken.

To address this, I adopted a two-step approach:

`hf_hub`

)Metadata (etag, commit hash, size, and entity URL) can be retrieved from the `huggingface_hub`

API to construct the cache structure.

The key is combining curl options:

```
subprocess.run(
    ["curl", "-L", "--fail", "-C", "-",
     "--retry", "50", "--retry-delay", "3", "--retry-all-errors",
     "--speed-time", "20", "--speed-limit", "2000",   # 20 seconds at 2KB/s or lower, consider transfer failed and retry
     "-o", blob, loc],
    check=True,
)
```

The roles of each option are as follows:

`--speed-time 20 --speed-limit 2000`

— `-C -`

— `--retry 50 --retry-delay 3 --retry-all-errors`

— Retry after 3 seconds, up to 50 times. Transfers cut off by `--speed-time`

are also retried.`-L`

(follow redirects) and `--fail`

(non-zero exit on HTTP errors).This combination automatically loops through "stagnation → cut → resume" and eventually completes the transfer. This was the most reliable method in environments where large files would freeze.

The core of the Hugging Face cache (e.g., `~/.cache/huggingface/hub`

) has the following structure per repository:

```
models--{org}--{repo}/
├── blobs/
│   └── {etag}                      # File entity, named by etag (content hash)
├── snapshots/
│   └── {commit_hash}/
│       └── {filename}              # Symbolic link to blob
└── refs/
    └── main                        # Branch name → commit hash text
```

The key points are:

`blobs/`

with etag names`snapshots/{commit}/`

contains symbolic links`refs/main`

contains the commit hash`revision="main"`

and allows `hf_hub`

to find the correct snapshot even when `HF_HUB_OFFLINE=1`

.Metadata can be obtained from the `huggingface_hub`

API:

``` python
from huggingface_hub import hf_hub_url, get_hf_file_metadata

url = hf_hub_url(repo_id, filename, revision=revision)
m = get_hf_file_metadata(url)
etag = (m.etag or "").strip('"')   # Becomes the blob file name
commit = m.commit_hash or revision # Becomes the snapshots/ directory name
size = m.size                       # Used for download completion detection
loc = m.location or url             # Entity URL (redirected)
```

The rest involves using curl to download the entity to `blobs/{etag}`

, creating a **relative symbolic link** from `snapshots/{commit}/{filename}`

, and writing the commit to `refs/main`

. The relative link ensures the cache remains intact even when moved to another machine.

```
os.symlink(os.path.relpath(blob, os.path.dirname(snap)), snap)
with open(os.path.join(refsdir, "main"), "w") as f:
    f.write(commit)
```

If the size is known, checking it against the existing blob size and skipping if complete can make re-runs idempotent.

Some repositories, like BigVGAN, contain large unnecessary files for inference (e.g., discriminator or optimizer states). Downloading the entire repository can result in getting stuck with the largest file. **Specify the necessary files explicitly** to avoid this.

Failing to create `refs/main`

alongside `snapshots`

and `blobs`

will prevent `hf_hub`

from resolving the correct commit when `HF_HUB_OFFLINE=1`

, even if the cache exists. Always create the **three-part set (blob, snapshot link, and refs)**.

Even with a cache, `hf_hub`

performs a HEAD request at startup to check for updates (without re-downloading large files). To ensure complete offline functionality, set `HF_HUB_OFFLINE=1`

or `TRANSFORMERS_OFFLINE=1`

. Conversely, if you want to see updates but prevent downloads, you can leave it online and just allow HEAD requests.

`hf_hub`

transfers `--speed-time`

and `--speed-limit`

to `-C -`

with `--retry`

for resumption.`blobs/{etag}`

+ `snapshots/{commit}/`

with relative symbolic links + `refs/main`

).`get_hf_file_metadata`

.`refs/main`

is created to enable `HF_HUB_OFFLINE=1`

.