Running Llama 3.1 405B

Meta's Llama 3.1 405B model is no longer hosted by any public inference provider, according to a guide on a11ce.com, which provides instructions for running the model on an on-demand GPU instance for about $20 per hour and 10 minutes of setup. The guide details using vLLM with FP8 weights downloaded from a Cloudflare R2 bucket, and setting up a local API endpoint via SSH or Cloudflare Tunnel.

a11ce.com /llama-3.1-405b.html Llama 3.1 405b base is no longer hosted by any public inference provider. This page has instructions for running it yourself on an on-demand GPU instance for ~$20/hr and 10 minutes of setup. df -h /dev/shm . API KEY=${API KEY:-$ openssl rand -hex 16 } echo "💜 api key is $API KEY" NVLS causes problems on some pods so just disable it export NCCL NVLS ENABLE=0 LLAMA=meta-llama/Llama-3.1-405B-FP8 WEIGHTSDIR=/dev/shm/llama This is a read-only token for an R2 bucket containing the weights R2 KEY ID=41f96e313f23edb542aa2d9f11f27d1f R2 ACCESS KEY=1864e92a4c9ac79581489dec9bf2e50b037fc47e35eb581ce46389cacfbdecb4 R2 ENDPOINT=https://47e39f6f60165d6392620a903e13d8b6.r2.cloudflarestorage.com echo "💜 downloading vllm" python3 -m pip install vllm fastsafetensors echo "💜 connecting to r2" curl -fsSL https://rclone.org/install.sh -o rclone-install.sh bash rclone-install.sh rclone config create r2 s3 provider=Cloudflare \ access key id=$R2 KEY ID \ secret access key=$R2 ACCESS KEY \ endpoint=$R2 ENDPOINT \ acl=private no check bucket=true echo "💜 downloading weights" mkdir -p $WEIGHTSDIR rclone copy --transfers 16 --multi-thread-streams 16 --multi-thread-cutoff 64M \ --progress r2:llama/Llama-3.1-405B-FP8/ "$WEIGHTSDIR"/ echo "💜 starting vllm" vllm serve "$WEIGHTSDIR" --tensor-parallel-size 4 --load-format fastsafetensors \ --max-model-len 65536 --allow-deprecated-quantization \ --served-model-name "$LLAMA" --api-key "$API KEY" ssh -N -L 8000:localhost:8000 root@<IP -p <PORT using the IP and port under "SSH over exposed TCP" to forward localhost:8000 to the server. http://localhost:8000/v1/completions with the model name meta-llama/Llama-3.1-405B-FP8 and the API key that was printed at the start of the script. ²²If you lose your API key, ctrl-Z to pause vllm, run echo $API KEY , then fg to resume vllm. For example on your local machine : curl http://localhost:8000/v1/completions \ -H "Authorization: Bearer <KEY " \ -H "Content-Type: application/json" \ -d '{"model": "meta-llama/Llama-3.1-405B-FP8", "prompt": "The capital of Poland is", "max tokens": 32}' https://<WORDS .trycloudflare.com/v1/completions : curl -sL -o /usr/local/bin/cloudflared https://github.com/cloudflare/cloudflared/releases/latest/download/cloudflared-linux-amd64 chmod +x /usr/local/bin/cloudflared cloudflared tunnel --url http://localhost:8000 2 &1 | grep --line-buffered -o 'https:// a-z- \.trycloudflare\.com' See the notes page llama-notes.html for more details and other methods. You can respond to this page on twitter https://x.com/oxa11ce/status/2093181373624586591 , mastodon https://types.pl/@a11ce/117171047422542766 , or bluesky https://bsky.app/profile/a11ce.bsky.social/post/3mu4grubb6k2v .