{"slug": "running-llama-3-1-405b", "title": "Running Llama 3.1 405B", "summary": "Meta's Llama 3.1 405B model is no longer hosted by any public inference provider, according to a guide on a11ce.com, which provides instructions for running the model on an on-demand GPU instance for about $20 per hour and 10 minutes of setup. The guide details using vLLM with FP8 weights downloaded from a Cloudflare R2 bucket, and setting up a local API endpoint via SSH or Cloudflare Tunnel.", "body_md": "[a11ce.com]/llama-3.1-405b.html\n\nLlama 3.1 405b (base) is no longer hosted by any public inference provider. This page has instructions for running it yourself on an on-demand GPU instance for ~$20/hr and 10 minutes of setup.\n\n`df -h /dev/shm`\n\n.\n\n```\nAPI_KEY=${API_KEY:-$(openssl rand -hex 16)}\necho \"💜 api key is $API_KEY\"\n\n# NVLS causes problems on some pods so just disable it\nexport NCCL_NVLS_ENABLE=0\n\nLLAMA=meta-llama/Llama-3.1-405B-FP8\nWEIGHTSDIR=/dev/shm/llama\n\n# This is a read-only token for an R2 bucket containing the weights\nR2_KEY_ID=41f96e313f23edb542aa2d9f11f27d1f\nR2_ACCESS_KEY=1864e92a4c9ac79581489dec9bf2e50b037fc47e35eb581ce46389cacfbdecb4\nR2_ENDPOINT=https://47e39f6f60165d6392620a903e13d8b6.r2.cloudflarestorage.com\n\necho \"💜 downloading vllm\"\npython3 -m pip install vllm fastsafetensors\n\necho \"💜 connecting to r2\"\ncurl -fsSL https://rclone.org/install.sh -o rclone-install.sh\nbash rclone-install.sh\nrclone config create r2 s3 provider=Cloudflare \\\n  access_key_id=$R2_KEY_ID \\\n  secret_access_key=$R2_ACCESS_KEY \\\n  endpoint=$R2_ENDPOINT \\\n  acl=private no_check_bucket=true\n\necho \"💜 downloading weights\"\nmkdir -p $WEIGHTSDIR\nrclone copy --transfers 16 --multi-thread-streams 16 --multi-thread-cutoff 64M \\\n  --progress r2:llama/Llama-3.1-405B-FP8/ \"$WEIGHTSDIR\"/\n\necho \"💜 starting vllm\"\nvllm serve \"$WEIGHTSDIR\" --tensor-parallel-size 4 --load-format fastsafetensors \\\n  --max-model-len 65536 --allow-deprecated-quantization \\\n  --served-model-name \"$LLAMA\" --api-key \"$API_KEY\"\n```\n\n`ssh -N -L 8000:localhost:8000 root@<IP> -p <PORT>`\n\nusing the IP and port under \"SSH over exposed TCP\" to forward\nlocalhost:8000 to the server.`http://localhost:8000/v1/completions`\n\nwith the model name\n`meta-llama/Llama-3.1-405B-FP8`\n\nand the API key that was\nprinted at the start of the script. ²²If you lose your API key, ctrl-Z to pause\nvllm, run `echo $API_KEY`\n\n, then `fg`\n\nto resume\nvllm. For example (on your local machine):\n\n```\ncurl http://localhost:8000/v1/completions \\\n  -H \"Authorization: Bearer <KEY>\" \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\"model\": \"meta-llama/Llama-3.1-405B-FP8\", \"prompt\": \"The capital of Poland is\", \"max_tokens\": 32}'\n```\n\n`https://<WORDS>.trycloudflare.com/v1/completions`\n\n:\n\n```\ncurl -sL -o /usr/local/bin/cloudflared https://github.com/cloudflare/cloudflared/releases/latest/download/cloudflared-linux-amd64\nchmod +x /usr/local/bin/cloudflared\ncloudflared tunnel --url http://localhost:8000 2>&1 | grep --line-buffered -o 'https://[a-z-]*\\.trycloudflare\\.com'\n```\n\nSee [the notes page](llama-notes.html) for more details\nand other methods.\n\nYou can respond to this page on [twitter](https://x.com/oxa11ce/status/2093181373624586591), [mastodon](https://types.pl/@a11ce/117171047422542766), or [bluesky](https://bsky.app/profile/a11ce.bsky.social/post/3mu4grubb6k2v).", "url": "https://wpnews.pro/news/running-llama-3-1-405b", "canonical_source": "https://a11ce.com/llama-3.1-405b.html", "published_at": "2026-08-28 20:01:27+00:00", "updated_at": "2026-08-28 20:18:41.782783+00:00", "lang": "en", "topics": ["large-language-models", "ai-infrastructure", "ai-tools"], "entities": ["Meta", "Llama 3.1 405B", "vLLM", "Cloudflare R2", "Cloudflare Tunnel", "a11ce.com"], "alternates": {"html": "https://wpnews.pro/news/running-llama-3-1-405b", "markdown": "https://wpnews.pro/news/running-llama-3-1-405b.md", "text": "https://wpnews.pro/news/running-llama-3-1-405b.txt", "jsonld": "https://wpnews.pro/news/running-llama-3-1-405b.jsonld"}}