# Running Llama 3.1 405B

> Source: <https://a11ce.com/llama-3.1-405b.html>
> Published: 2026-08-28 20:01:27+00:00

[a11ce.com]/llama-3.1-405b.html

Llama 3.1 405b (base) is no longer hosted by any public inference provider. This page has instructions for running it yourself on an on-demand GPU instance for ~$20/hr and 10 minutes of setup.

`df -h /dev/shm`

.

```
API_KEY=${API_KEY:-$(openssl rand -hex 16)}
echo "💜 api key is $API_KEY"

# NVLS causes problems on some pods so just disable it
export NCCL_NVLS_ENABLE=0

LLAMA=meta-llama/Llama-3.1-405B-FP8
WEIGHTSDIR=/dev/shm/llama

# This is a read-only token for an R2 bucket containing the weights
R2_KEY_ID=41f96e313f23edb542aa2d9f11f27d1f
R2_ACCESS_KEY=1864e92a4c9ac79581489dec9bf2e50b037fc47e35eb581ce46389cacfbdecb4
R2_ENDPOINT=https://47e39f6f60165d6392620a903e13d8b6.r2.cloudflarestorage.com

echo "💜 downloading vllm"
python3 -m pip install vllm fastsafetensors

echo "💜 connecting to r2"
curl -fsSL https://rclone.org/install.sh -o rclone-install.sh
bash rclone-install.sh
rclone config create r2 s3 provider=Cloudflare \
  access_key_id=$R2_KEY_ID \
  secret_access_key=$R2_ACCESS_KEY \
  endpoint=$R2_ENDPOINT \
  acl=private no_check_bucket=true

echo "💜 downloading weights"
mkdir -p $WEIGHTSDIR
rclone copy --transfers 16 --multi-thread-streams 16 --multi-thread-cutoff 64M \
  --progress r2:llama/Llama-3.1-405B-FP8/ "$WEIGHTSDIR"/

echo "💜 starting vllm"
vllm serve "$WEIGHTSDIR" --tensor-parallel-size 4 --load-format fastsafetensors \
  --max-model-len 65536 --allow-deprecated-quantization \
  --served-model-name "$LLAMA" --api-key "$API_KEY"
```

`ssh -N -L 8000:localhost:8000 root@<IP> -p <PORT>`

using the IP and port under "SSH over exposed TCP" to forward
localhost:8000 to the server.`http://localhost:8000/v1/completions`

with the model name
`meta-llama/Llama-3.1-405B-FP8`

and the API key that was
printed at the start of the script. ²²If you lose your API key, ctrl-Z to pause
vllm, run `echo $API_KEY`

, then `fg`

to resume
vllm. For example (on your local machine):

```
curl http://localhost:8000/v1/completions \
  -H "Authorization: Bearer <KEY>" \
  -H "Content-Type: application/json" \
  -d '{"model": "meta-llama/Llama-3.1-405B-FP8", "prompt": "The capital of Poland is", "max_tokens": 32}'
```

`https://<WORDS>.trycloudflare.com/v1/completions`

:

```
curl -sL -o /usr/local/bin/cloudflared https://github.com/cloudflare/cloudflared/releases/latest/download/cloudflared-linux-amd64
chmod +x /usr/local/bin/cloudflared
cloudflared tunnel --url http://localhost:8000 2>&1 | grep --line-buffered -o 'https://[a-z-]*\.trycloudflare\.com'
```

See [the notes page](llama-notes.html) for more details
and other methods.

You can respond to this page on [twitter](https://x.com/oxa11ce/status/2093181373624586591), [mastodon](https://types.pl/@a11ce/117171047422542766), or [bluesky](https://bsky.app/profile/a11ce.bsky.social/post/3mu4grubb6k2v).