{"slug": "i-ran-a-gpu-inference-app-for-a-month-on-azure-serverless-gpu-here-s-the-actual", "title": "I ran a GPU inference app for a month on Azure serverless GPU. Here's the actual bill.", "summary": "An engineer ran a GPU inference service on Azure Container Apps' serverless GPU for 48 days and found the total bill was around $13 per month, with GPU charges appearing on only 12 days. The biggest cost was not the GPU but the Container Registry storage, which cost about $11 per month due to large CUDA images and accumulated old image tags. The engineer recommends cleaning up old images and warns that leaving min-replicas set to 1 after debugging can cause unexpected charges.", "body_md": "I run a small internal speech-synthesis service on an NVIDIA T4 via Azure Container Apps' serverless GPU, with `min-replicas=0`\n\n. The pitch is that you pay nothing while nobody's using it.\n\nI wanted to know if that's actually true, so I pulled the real numbers out of Azure Cost Management, scoped to just that resource group.\n\nShort answer: **yes, it's true.** But the biggest line item on my bill wasn't the GPU.\n\n```\n[browser] → [Container App (T4, min=0 → max=1)]\n               ├─ web UI      (FastAPI + uvicorn, :8000)\n               └─ inference   (:9880)\n```\n\nIdle for ~5 minutes and it scales to zero replicas. Next request wakes it back up. Cold start is about 5 minutes in my case — a CUDA base image plus loading model weights.\n\nMeasured over 48 days, scoped to one resource group. (Billed in JPY; USD figures are approximate.)\n\n| Service | Per month | Notes |\n|---|---|---|\n| Container Apps (GPU) | $2–7 | Only charged on days I used it |\n| Container Registry | ~$11 | Fixed cost. The biggest line item. |\n| Log Analytics | $0 |\n\nTotal: **around $13/month**, including the month where I was actively building and testing.\n\nHere's the daily distribution. Out of 48 days, **GPU charges appeared on only 12**.\n\n| Charge that day | Days |\n|---|---|\n| $0 (unused) | 36 |\n| $0.01–0.20 | 4 |\n| $0.20–0.80 | 5 |\n| ~$1.05 | 2 |\n| $2.30 | 1 |\n\nA day of real use costs somewhere between four cents and a dollar. An always-on T4 VM of comparable spec runs a few hundred dollars a month. That's close to two orders of magnitude.\n\nThe tradeoff is the cold start. For an internal tool where people know to expect a few minutes on first use, that's an easy trade. For a customer-facing endpoint, it isn't.\n\nThat one outlier day has a boring explanation. I deployed a new image and temporarily forced the app to stay up so I could read the startup logs:\n\n```\n# force it up to inspect logs\naz containerapp update -n $APP -g $RG --image $ACR.azurecr.io/my-app:2 \\\n  --min-replicas 1 --max-replicas 1\n\n# ...and this is the line people forget\naz containerapp update -n $APP -g $RG --min-replicas 0\n```\n\nIn a min=0 setup, the expensive mistake is never the GPU's hourly rate. It's leaving `min-replicas 1`\n\non after a debugging session. Put the scale-back-down step in your deploy checklist.\n\nThe Azure portal showing **\"Status: Running\"** means *the app exists*, not that a replica is up. That confused me early on. The CLI tells the truth:\n\n```\n# empty output = no replicas = not being billed\naz containerapp replica list -n $APP -g $RG -o table\n\n# ScaledToZero or Running\naz containerapp revision list -n $APP -g $RG \\\n  --query \"[?properties.active].properties.runningState\" -o tsv\n```\n\nContainer Registry was ~$11/month against the GPU's $2–7. Two reasons compounding:\n\n**The images are huge.** A CUDA base image plus bundled model weights runs several tens of GB per image. I bundle the weights deliberately — downloading them at startup would add minutes to an already slow cold start.\n\n**Old images never leave.** I tag every deploy (`my-app:1`\n\n, `:2`\n\n, `:3`\n\n…), and nothing removes the old ones. Storage billing is on the total, so the bill creeps up with every deploy.\n\nThe fix is housekeeping. Keep what's deployed plus one for rollback:\n\n```\n# what's actually running\naz containerapp show -n $APP -g $RG \\\n  --query \"properties.template.containers[0].image\" -o tsv\n\n# tags, newest first\naz acr repository show-tags -n $ACR --repository my-app --orderby time_desc -o tsv\n\n# drop the old ones\nfor t in 1 2 3 4 5 6 7 8; do\n  az acr repository delete -n $ACR --image my-app:$t --yes\ndone\n```\n\nI went from 10 images down to 2. Storage billing catches up over a few hours, via the registry's garbage collection.\n\nIf you're evaluating serverless GPU, budget for registry storage. It's the line item nobody warns you about, and for a low-traffic workload it can quietly become your largest one.\n\n`--min-replicas 0`\n\n.Getting from zero to a working deployment took considerably longer than it should have. Pinning `torch`\n\nagainst `transformers`\n\n' requirements, a 504 caused by the order services start in, and — on Windows — `az acr build`\n\ncrashing the CLI on a character encoding issue mid-build. None of that is in the docs.\n\nI wrote all of it up, including the full deployment walkthrough, here:\n\n[Zero-Idle GPU: Running Inference on Azure Container Apps](https://leanpub.com/zero-idle-gpu)\n\nHappy to answer questions in the comments.", "url": "https://wpnews.pro/news/i-ran-a-gpu-inference-app-for-a-month-on-azure-serverless-gpu-here-s-the-actual", "canonical_source": "https://dev.to/aco_dog_32627df9d920ffba4/i-ran-a-gpu-inference-app-for-a-month-on-azure-serverless-gpu-heres-the-actual-bill-i35", "published_at": "2026-08-18 01:44:28+00:00", "updated_at": "2026-08-18 02:13:21.346603+00:00", "lang": "en", "topics": ["ai-infrastructure", "mlops", "developer-tools"], "entities": ["Azure Container Apps", "NVIDIA T4", "Azure Cost Management", "Container Registry", "Log Analytics", "FastAPI", "uvicorn", "CUDA"], "alternates": {"html": "https://wpnews.pro/news/i-ran-a-gpu-inference-app-for-a-month-on-azure-serverless-gpu-here-s-the-actual", "markdown": "https://wpnews.pro/news/i-ran-a-gpu-inference-app-for-a-month-on-azure-serverless-gpu-here-s-the-actual.md", "text": "https://wpnews.pro/news/i-ran-a-gpu-inference-app-for-a-month-on-azure-serverless-gpu-here-s-the-actual.txt", "jsonld": "https://wpnews.pro/news/i-ran-a-gpu-inference-app-for-a-month-on-azure-serverless-gpu-here-s-the-actual.jsonld"}}