Running Llama 3.1 405B

[a11ce.com]/llama-3.1-405b.html

Llama 3.1 405b (base) is no longer hosted by any public inference provider. This page has instructions for running it yourself on an on-demand GPU instance for ~$20/hr and 10 minutes of setup.

df -h /dev/shm

.

API_KEY=${API_KEY:-$(openssl rand -hex 16)}
echo "💜 api key is $API_KEY"

export NCCL_NVLS_ENABLE=0

LLAMA=meta-llama/Llama-3.1-405B-FP8
WEIGHTSDIR=/dev/shm/llama

R2_KEY_ID=41f96e313f23edb542aa2d9f11f27d1f
R2_ACCESS_KEY=1864e92a4c9ac79581489dec9bf2e50b037fc47e35eb581ce46389cacfbdecb4
R2_ENDPOINT=https://47e39f6f60165d6392620a903e13d8b6.r2.cloudflarestorage.com

echo "💜 down vllm"
python3 -m pip install vllm fastsafetensors

echo "💜 connecting to r2"
curl -fsSL https://rclone.org/install.sh -o rclone-install.sh
bash rclone-install.sh
rclone config create r2 s3 provider=Cloudflare \
  access_key_id=$R2_KEY_ID \
  secret_access_key=$R2_ACCESS_KEY \
  endpoint=$R2_ENDPOINT \
  acl=private no_check_bucket=true

echo "💜 down weights"
mkdir -p $WEIGHTSDIR
rclone copy --transfers 16 --multi-thread-streams 16 --multi-thread-cutoff 64M \
  --progress r2:llama/Llama-3.1-405B-FP8/ "$WEIGHTSDIR"/

echo "💜 starting vllm"
vllm serve "$WEIGHTSDIR" --tensor-parallel-size 4 --load-format fastsafetensors \
  --max-model-len 65536 --allow-deprecated-quantization \
  --served-model-name "$LLAMA" --api-key "$API_KEY"

ssh -N -L 8000:localhost:8000 root@<IP> -p <PORT>

using the IP and port under "SSH over exposed TCP" to forward localhost:8000 to the server.http://localhost:8000/v1/completions

with the model name meta-llama/Llama-3.1-405B-FP8

and the API key that was printed at the start of the script. ²²If you lose your API key, ctrl-Z to vllm, run echo $API_KEY

, then fg

to resume vllm. For example (on your local machine):

curl http://localhost:8000/v1/completions \
  -H "Authorization: Bearer <KEY>" \
  -H "Content-Type: application/json" \
  -d '{"model": "meta-llama/Llama-3.1-405B-FP8", "prompt": "The capital of Poland is", "max_tokens": 32}'

https://<WORDS>.trycloudflare.com/v1/completions

:

curl -sL -o /usr/local/bin/cloudflared https://github.com/cloudflare/cloudflared/releases/latest/download/cloudflared-linux-amd64
chmod +x /usr/local/bin/cloudflared
cloudflared tunnel --url http://localhost:8000 2>&1 | grep --line-buffered -o 'https://[a-z-]*\.trycloudflare\.com'

See the notes page for more details and other methods.

You can respond to this page on twitter, mastodon, or bluesky.