On serverless GPU, cold start is image pull plus model load. The single biggest win is usually to stop down weights at boot: download them at build time so they ship inside the image and load from local disk instead.
The setup is two lines. Point the cache at a path inside the image, then fetch during the build:
ENV HF_HOME=/opt/huggingface
RUN python -c "from huggingface_hub import snapshot_download; snapshot_download('<repo>', revision='<pinned>')"
snapshot_download
writes into HF_HOME
, that directory becomes an image layer, and at runtime the finds everything already on disk.
Here's the part worth knowing. The resolves weights through HF_HOME
at runtime. So if anything in your runtime environment sets HF_HOME
to a different path — a network volume, a mount, a leftover template variable — the looks there, finds an empty cache, and quietly falls back to down from the network.
Nothing errors. Your image still contains the weights, sitting in a directory nobody reads anymore. You just paid to bake them, and you're paying the download on every cold start anyway. The only symptom is that cold starts went back to what they were before you did the work, which is easy to blame on the platform.
Attach no volume at all. If the worker has no volume, there's no tempting path to point the cache at. In my setup the endpoints run with volume size zero for exactly this reason: everything the worker needs is in the image, so a volume is not an optimization, it's a way to reintroduce the network.
Verify at boot, not by feel. Watch a cold worker's log. If you see download progress for weights, the bake is being bypassed, full stop. A baked image reaching the network for weights is always a misconfiguration, never a normal path.
Treat the cache path as part of the build contract. The Dockerfile that baked the weights and the runtime env that reads them have to agree. That agreement is invisible in both files individually, so write it down where whoever edits the template env will see it.
Baking has real costs: the image gets large, builds take much longer, and updating the model means rebuilding and redistributing the whole thing. If your weights are small or your traffic keeps workers warm, a volume or a runtime fetch may genuinely be the better trade. This is about the case where you already chose to bake — because then paying the build cost and the download cost is the worst of both.