{"slug": "should-you-run-an-llm-locally-or-in-the-cloud", "title": "Should you run an LLM locally or in the cloud?", "summary": "A developer at Nodus Compute laid out a rule for deciding whether to run large language models locally or in the cloud, based on memory: a 4-bit model needs roughly 0.56 GB per billion parameters, so a 70B model takes about 39.8 GB and exceeds the 32 GB of an RTX 5090, while 7-8B models fit in about 4.5 GB on any 16 GB machine. The post recommends filling no more than 90% of memory and budgeting about 0.33 MB of KV cache per token in flight for a 70B model, and it frames the hardware-versus-cloud choice as a break-even calculation dividing hardware price by the hourly rate for the same memory class. The conclusion is that most users should do both: tinker locally with small quantized models and send 70B serving, fine-tuning and overnight batches to cloud GPUs.", "body_md": "Run an LLM locally when the model fits comfortably in your machine's memory and your data should never leave it; move to the cloud when the model, the fine-tune or the batch job outgrows that memory. Nodus runs those bigger jobs on the GPU they actually need, caps each run at a dollar limit you set and shows exactly what it cost, so you pay for hours of work instead of a workstation. For most people the honest answer is both: tinker locally with small quantized models, and send the 70B serving, fine-tuning and overnight batches out.\n\nThe rest of this post turns that into numbers: a memory rule for picking where a model can run, and a break-even calculation for when buying hardware beats paying by the hour.\n\nMemory decides almost everything, so check it first. A model's weights take roughly their checkpoint size in memory, and you need room on top for the KV cache and the runtime.\n\nWe use the numbers from our post on [how much VRAM a 70B model needs](https://www.nodus-compute.ai/blog/vram-needed-for-70b-model/). Llama 3.3 70B takes 141.1 GB at 16-bit, 72.7 GB at FP8 and 39.8 GB at 4-bit. That 4-bit figure works out to about 0.56 GB per billion parameters (39.8 GB divided by 70.55 billion), which we use below to scale to other sizes. Plan to fill no more than 90% of memory, and budget about 0.33 MB of KV cache per token in flight for a 70B model.\n\n*A 70B model at 4-bit is already too big for the largest consumer graphics card.*\n\nThat gives a simple rule by model size, at 4-bit:\n\n| Model size | 4-bit weights, approx. | Where it runs well | \n|---|---|---|\n| 7 to 8B | about 4.5 GB | Any laptop or GPU with 16 GB | \n| 20 to 32B | 11 to 18 GB | A 24 to 32 GB GPU, or a Mac with 32 GB or more | \n| 70B | 39.8 GB | A Mac with 64 GB or more, or one 80 GB GPU in the cloud | \n| 70B at FP8 or 16-bit, 120B and up | 72.7 GB and up | Cloud: H200, B200 or several GPUs | \n\nThe 4-bit column is parameters times 0.56 GB, except the 70B row, which is the measured checkpoint. Real models vary by a few GB because some layers stay at 16-bit.\n\nThe NVIDIA GeForce RTX 5090 has [32 GB of GDDR7](https://www.nvidia.com/en-us/geforce/graphics-cards/50-series/rtx-5090/), so it handles everything up to about 32B at 4-bit. The [Mac Studio](https://www.apple.com/mac-studio/specs/) can be configured with 96 GB, 256 GB or 512 GB of unified memory on M5 Ultra. Unified memory is shared with the operating system and your apps, so leave headroom beyond the 90% rule.\n\nLocal wins whenever the model fits and the work is personal, private or constant. We would pick it in four cases.\n\nPrivate data you cannot send anywhere. [Ollama's FAQ](https://docs.ollama.com/faq) says that when you run locally, it does not see your prompts or data, and it documents a local-only mode that turns its cloud features off. No contract or data processing review beats data that never leaves the laptop.\n\nSmall models and quick experiments. The tools are excellent. [llama.cpp](https://github.com/ggml-org/llama.cpp) supports 1.5-bit to 8-bit integer quantization for faster inference and lower memory use. [LM Studio](https://lmstudio.ai/docs/app/system-requirements) recommends 16 GB of RAM or more. With Ollama, one command gets you a capable model:\n\n```\nollama run gpt-oss:20b\n```\n\nThe [Ollama library page](https://ollama.com/library/gpt-oss) says gpt-oss-20b runs on systems with as little as 16 GB of memory.\n\nOffline work and latency. A local model answers on a plane and has no network round trip.\n\nAlways-on use. If a GPU would be busy most hours of the day, owning it gets cheap fast, as the next section shows.\n\nDivide the hardware price by the hourly rate for the same memory class. That is the break-even in hours of actual use, and most people use far fewer hours than they think.\n\nWe priced two local setups against the Nodus rate for comparable memory:\n\n| Local setup | List price | Comparable cloud GPU | Nodus rate | Break-even | \n|---|---|---|---|---|\n| RTX 5090 card, 32 GB | $1,999 | RTX 5090, 32 GB | $0.48/hr | 4,165 hours | \n| Mac Studio M5 Ultra, 96 GB | $5,499 | A100, 80 GB | $1.09/hr | 5,045 hours | \n\nPrices checked October 26, 2026: the [RTX 5090 page](https://www.nvidia.com/en-us/geforce/graphics-cards/50-series/rtx-5090/) lists \"Starting at $1999\"; the [Mac Studio M5 Ultra with 96 GB and 1 TB](https://www.apple.com/shop/buy-mac/mac-studio/m5-ultra-chip-30-core-cpu-64-core-gpu-96gb-memory-1tb-storage) is $5,499 on Apple's store; both GPUs are listed at those hourly rates on the [Nodus pricing page](https://www.nodus-compute.ai/pricing/).\n\nThe arithmetic is $1,999 / $0.48 = 4,165 hours, and $5,499 / $1.09 = 5,045 hours. We paired the Mac with the A100 80 GB because it is the cheapest single GPU in our VRAM post that serves a 70B model at 4-bit.\n\nWhat that means in calendar time depends on how hard you use it:\n\n*Part-time use takes years to pay off a GPU; a machine that never sleeps pays off in about six months.*\n\nThe break-even flatters buying in one way and understates it in another. The card price leaves out the rest of the PC, electricity and your time spent on drivers, which all push break-even later. The cloud side leaves out storage and data transfer. Treat the result as a range, not a verdict.\n\nOur rule: below about 4 hours of real use a day, pay by the hour. At 8 hours a day or more, buying pays off in under two years, so buy the hardware if the model fits on it.\n\nThe moment it needs more memory than any single machine you would buy, or more hours in parallel than one box has. Four jobs hit that wall.\n\nServing a 70B model at higher precision or to many users. FP8 weights alone are 72.7 GB, and 32 requests of 8,192 tokens add 85.9 GB of KV cache. That is a data center GPU job.\n\nFine-tuning. Training keeps gradients and optimizer state, so memory jumps. For a 70B model our VRAM post works out about 50 GB for QLoRA, about 152 GB for LoRA and about 1,189 GB for full fine-tuning with AdamW across 8 GPUs.\n\n*Full fine-tuning a 70B model has no single-machine option at any price you would pay for a desk.*\n\nThose map to listed Nodus rates of $1.09/hr for one A100 80 GB, $1.35/hr for one RTX PRO 6000 96 GB, $5.50/hr for one B200 and $44.00/hr for 8 x B200 ($5.50 x 8). A full fine-tune that runs 10 hours on 8 x B200 costs $440 at that rate, against a machine with 1,440 GB of GPU memory you would otherwise have to own.\n\nBatch jobs. Embedding a million documents or scoring an eval set is the same work spread over many GPUs at once. Local hardware runs it one machine at a time.\n\nBursts. A demo, a launch or a weekly retrain needs big hardware for a few hours, then nothing. That is the worst case for buying and the best case for paying by the hour.\n\nTwo ways, depending on whether you need a hosted model or your own code on a GPU.\n\nFor hosted open models, Nodus [inference](https://www.nodus-compute.ai/docs/guides/inference/) is an OpenAI-compatible API. Install the CLI, sign in and send a request:\n\n```\npip install nodus-compute\nnodus login\nnodus inference chat --model nodus/gpt-oss-20b --max-tokens 64 \"What is the capital of France?\"\n```\n\nThe CLI prints the answer and reports the tokens and charge for that request. The same gpt-oss models you run with Ollama are in the catalog, so you can prototype locally and point the same OpenAI client at `https://inference.nodus-compute.ai/v1` when you need more throughput.\n\nFor your own weights, fine-tunes and batch scripts, use a [Job](https://www.nodus-compute.ai/docs/getting-started/). `nodus run` takes a `--gpu` type, a `--max-cost` dollar cap and a `--dry-run` flag that returns an estimate before anything is charged. Checkpointed Jobs save the files your code writes to `NODUS_CHECKPOINT_DIR` and restore them if a machine is lost, which matters most for long fine-tunes on spot capacity. Our post on [resuming training after spot preemption](https://www.nodus-compute.ai/blog/resume-training-after-spot-preemption/) shows the pattern.\n\nYes, if the model fits in about 90% of your memory at a precision whose quality you have checked, and your use is private, offline or constant. For anything 70B and up at FP8 or 16-bit, or any fine-tune beyond QLoRA, the cloud is cheaper than the hardware it would take.\n\nThe same open model at the same precision behaves the same wherever it runs. The quality gap comes from running a smaller or more heavily quantized model locally, so compare the exact checkpoint you would use.\n\nAt 4-bit, yes, with 64 GB of unified memory or more, since the weights are 39.8 GB plus cache. Fine-tuning that model beyond QLoRA is out of reach on a Mac.\n\nFor part-time and bursty use, usually, because idle hours cost nothing and a `--max-cost` cap on each Nodus Job bounds the worst case. For a GPU that would run all day, every day, owning it is cheaper. Compare current GPU rates in [our cloud GPU pricing guide](https://www.nodus-compute.ai/blog/cloud-gpu-pricing-october-2026/) and on the [Nodus pricing page](https://www.nodus-compute.ai/pricing/).\n\n*Originally published on the [Nodus blog](https://www.nodus-compute.ai/blog/run-llm-locally-vs-cloud/).*", "url": "https://wpnews.pro/news/should-you-run-an-llm-locally-or-in-the-cloud", "canonical_source": "https://dev.to/viswaretas_kotra/should-you-run-an-llm-locally-or-in-the-cloud-2mgi", "published_at": "2026-10-26 20:34:37+00:00", "updated_at": "2026-10-26 20:42:42.244137+00:00", "lang": "en", "topics": ["large-language-models", "ai-infrastructure", "ai-tools", "mlops"], "entities": ["Nodus Compute", "Llama 3.3 70B", "NVIDIA GeForce RTX 5090", "Mac Studio", "Ollama", "llama.cpp", "LM Studio", "gpt-oss-20b"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/should-you-run-an-llm-locally-or-in-the-cloud", "markdown": "https://wpnews.pro/news/should-you-run-an-llm-locally-or-in-the-cloud.md", "text": "https://wpnews.pro/news/should-you-run-an-llm-locally-or-in-the-cloud.txt", "jsonld": "https://wpnews.pro/news/should-you-run-an-llm-locally-or-in-the-cloud.jsonld"}}