{"slug": "how-to-run-a-big-model-on-cheap-hardware", "title": "How to Run a Big Model on Cheap Hardware?", "summary": "Running a large AI model on modest hardware requires reducing the memory it occupies, cutting the calculations it performs, or offloading work to slower hardware, according to a ByteByteGo article that outlines techniques including quantization, layer-wise offloading, mixture of experts, distillation, pruning, and speculative decoding. The article notes that an 8B model contains approximately eight billion parameters and that inference, unlike training, leaves the model's weights unchanged, which is why a model trained on expensive infrastructure can sometimes run on an ordinary computer.", "body_md": "## [Test Your Auth Flow Without Production (Sponsored)](https://go.bytebytego.com/WorkOS_092126)\n\nAuthentication is often the least-tested part of an app. Live environments need network access and real credentials, while mocks miss the failures that break production.\n\n@workos/emulate runs the WorkOS API locally for development and CI. Seed users, organizations, RBAC roles, and SSO connections, then test full AuthKit login flows, signed webhooks, token refresh, and error handling. Responses and event shapes come from the WorkOS OpenAPI spec, so tests exercise the same surface your app uses in production.\n\nImagine a scenario where a developer builds an AI-based coding assistant that runs on a desktop computer. The model is available to download, the application is quite straightforward, and the machine has plenty of storage. But when the program tries to load the model, it runs out of memory.\n\nThis is where local AI development becomes a hardware problem.\n\nMerely downloading a large AI model to our computer doesn’t mean we can actually run that model. In fact, even if we successfully load it, we can’t guarantee a useful response time.\n\nA large AI model can run on modest hardware only by reducing the memory it occupies, reducing the calculations it performs, or moving some work to slower hardware. Several techniques can help deal with these requirements. In this article, we’re going to look at these techniques. Here’s what we will cover:\n\n- What running a model actually means\n- What makes it difficult to fit a model on smaller hardware\n- Why make the effort to run locally\n- Quantization: Giving each weight a smaller representation\n- Layer-wise offloading: Move weights as they are needed\n- Mixture of experts: Use selected parts for each token\n- Distillation: Let a larger model teach a smaller model\n- Pruning: Remove work that contributes less\n- Speculative decoding: Propose several tokens before checking them\n\n## What Running a Model Actually Means\n\nAn AI model contains numerical values called parameters. These are also known as weights. They influence how input becomes output. During training, these values are adjusted so that the model becomes better at its task. An 8B model contains approximately eight billion parameters or weights.\n\nThese weights are organized into layers. Each layer performs calculations on incoming information and passes its results to the next layer.\n\nIn a language model, the input text is first divided into tokens. These tokens can represent words, parts of words, or even punctuation. The model processes them and produces probabilities for the next token. Once a token is selected and added to the sequence, the generation of text continues.\n\nProducing a complete answer is all about repeating this process.\n\nA couple of points to keep in mind here are as follows:\n\n- Using an already trained model is called inference. The model’s weights normally remain unchanged while it answers a question.\n- Training requires additional calculations to update those weights, making the process quite demanding. It requires expensive infrastructure.\n\nThis difference helps explain why a model trained on expensive infrastructure can sometimes run on an ordinary computer. The main thing we are discussing here is not the training aspects, but the inference part of a model.\n\n## [\\[Webinar\\] How to stop babysitting your agents (Sponsored)](https://getunblocked.com/events/how-to-stop-babysitting-your-agents-sep-23/?utm_source=bytebytego&utm_medium=email&utm_campaign=secondary_20260921https://go.bytebytego.com/Unblocked_092126)\n\nAgents can generate code. Getting it right for your system, team conventions, and past decisions is the hard part. You end up wasting time and tokens in the correction loops.\n\nMore MCPs, rules, and bigger context windows give agents access to information, but not understanding. The teams pulling ahead have a context layer to give agents exactly what they need for the task at hand.\n\n[Join us for a FREE webinar on Sep 23](https://go.bytebytego.com/Unblocked_092126) to see:\n\n- Where teams get stuck on the AI maturity curve and why common fixes fall short\n- How a context layer solves for quality, efficiency, and cost\n- Live demo: the same coding task with and without a context layer\n\nIf you want to maximize the value you get from AI agents, this one is worth your time.\n\n## What Makes it Difficult to Fit the Model on Smaller Hardware\n\nEvery parameter or weight occupies memory. A value stored using 16 bits requires two bytes. Following from this, 8 billion parameters require approximately 16 GB for the raw weights alone. Temporary calculations, software overhead, and information retained during generation require additional space.\n\nThe location of that memory plays a pivotal role. A computer’s RAM is its main working memory. A discrete graphics card has separate memory called VRAM. A desktop with 32 GB of RAM and 8 GB of VRAM doesn’t automatically provide one equally fast memory pool with 40 GB of space. Though computers with unified memory do share memory between processors, they still face capacity limits.\n\nA large SSD can hold the model file. However, executing the model requires its data to reach the processor. If we constantly have to retrieve weights from storage, it is much slower than keeping them in working memory.\n\nEither way, memory capacity is only one of the constraints. Neural networks also perform a large number of multiplications and additions. Though a CPU can execute these operations, a GPU is much more effective at performing many similar numerical operations in parallel. However, a GPU still needs sufficient memory and support for the numerical operations used by the model.\n\nLastly, even when a model fits in memory, the processor must read its weights and other information quickly enough.\n\nMemory bandwidth measures how much data can move between memory and the processor in a given time. For many workloads involving one user generating text, moving model data becomes a major limit.\n\nText generation and inference also has two stages:\n\n- During prefill, the model processes the supplied prompt, which allows considerable parallel work.\n- During decoding, it produces output tokens in sequence.\n\nThese stages can have different bottlenecks. A model can fit in memory and still respond too slowly to be useful.\n\n## Why Make the Effort to Run Locally?\n\nLocal execution can make existing hardware useful for experimentation and reduce dependence on rented computing resources. This can matter for private documents and proprietary source code.\n\nOffline availability matters for desktop assistants, remote installations, and products that cannot assume a reliable internet connection. Local execution also gives developers more control over model versions and application behavior.\n\nHowever, local execution is not automatically cheaper for every workload.\n\nHardware, electricity, maintenance, and response speed all matter. Occasional use of a hosted service can be economical, while frequent use may make local hardware more attractive. The comparison depends on the actual application.\n\nLet us now look at a few techniques that can help run a big model on cheap hardware.\n\n## Quantization: Give Each Weight a Smaller Representation\n\nQuantization reduces the precision used to represent numerical values. In a simplified scheme using steps of 0.1, the value 0.73 could be approximated as 0.7. The approximation introduces an error, but that error may be small enough for the application to tolerate.\n\nSee the diagram below that shows the process of quantization:\n\nReal methods map weights onto a smaller set of possible values. For example, a 4-bit code has sixteen possible values. Groups of weights can use separate scaling information to map these codes onto appropriate numerical ranges.\n\nThe potential savings are substantial. An 8B model whose weights require 16 GB at 16-bit precision has a raw weight size of approximately 4 GB at 4-bit precision. Reducing the theoretical weight size from 16 GB to 4 GB can leave room for other allocations. Its parameter count remains unchanged. But each parameter is stored more compactly.\n\nThe tradeoff is a possible degradation in answer quality. The extent of this degradation depends on the model, compression method, and the task at hand. A configuration that handles ordinary conversation well may perform less reliably on particular coding problems.\n\nLower precision also does not guarantee a proportional speed improvement. Some implementations store 4-bit weights while performing calculations at higher precision. Efficient execution depends on hardware support and the software handling those conversions.\n\nQuantization can be applied after training. This is known as post-training quantization. However, training can also account for the errors it introduces. Application developers often begin with an existing quantized version and evaluate whether its output remains suitable.\n\n## Layer-Wise Offloading: Move Weights as They Are Needed\n\nQuantization changes how much space weights occupy. Offloading changes where they reside or where their calculations execute.\n\nDuring a forward pass, information progresses through the model’s layers. The GPU doesn’t need every layer’s weights at precisely the same moment. A system can keep most weights in RAM, transfer one layer onto the GPU, execute it, and release that GPU copy before loading another layer. Other layers may remain permanently on the GPU.\n\nThis approach is known as layer-wise offloading. It allows execution without placing the entire model in GPU memory simultaneously. More aggressive arrangements can keep weights on disk and bring them to RAM before execution.\n\nThe trade-off with this approach is around repeated movement of data. For example, if a hypothetical setup transfers 10 GB of weights per generation step over a connection sustaining 10 GB per second. Those transfers alone require approximately one second per step. This shows why reducing GPU memory requirements can also make generation much slower.\n\nThese transfers may repeat as the model generates successive tokens. A model that produces an answer eventually may still be unsuitable for an interactive assistant. Offloading systems can improve efficiency by reusing transferred weights across batches of requests, which is quite useful when individual response latency is less important.\n\nAnother arrangement assigns some layers to the CPU and others to the GPU. Weights remain with their assigned processor, while intermediate results move between processors. Suitable software can also run a model entirely on the CPU if enough RAM is available.\n\nThese approaches make larger models accessible. But their response times must be measured.\n\n## Mixture of Experts: Use Selected Parts for Each Token\n\nMixture of Experts (MoE) changes the model’s architecture.\n\nIn a conventional dense transformer, each token passes through the same major computational blocks. However, in a sparse MoE model, certain blocks contain multiple alternative networks called experts. A small routing network selects which experts should process a particular token.\n\nFor example, a layer might contain 8 experts while using only two for each token. Another token can be sent to a different pair. The outputs of the selected experts are combined and passed onward.\n\nAn expert is usually just a component inside the model, rather than a complete chatbot. Its role is learned during training and need not correspond neatly to a specific subject such as mathematics or history.\n\nMoE introduces a distinction between total parameters and active parameters. Total parameters strongly influence weight storage. On the other hand, active parameters help indicate how much computation happens for each token.\n\nThe inactive experts still need to be stored somewhere because later tokens may use them. Keeping all experts resident supports fast execution. Offloading experts introduces transfer costs. A model with relatively few active parameters can therefore still require substantial memory. For example, if a hypothetical MoE contains 40 billion parameters, four-bit raw weight storage still comes to about 20 GB. A small active subset doesn’t change that storage calculation.\n\nMoE can provide substantial model capacity while doing less computation per token than a dense model with the same total parameter count. However, it cannot automatically make a large model suitable for a low-memory laptop.\n\n## Distillation: Let a Larger Model Teach a Smaller One\n\nKnowledge distillation uses a larger model to help train a smaller model.\n\nThe larger model is called the teacher, and the smaller model is called the student. The student learns from information produced by the teacher, such as its predictions. Once training is complete, the student can operate independently.\n\nFor example, a large teacher could help produce training examples for a smaller model that categorizes support requests and drafts short responses. The student may learn to perform that particular task well while requiring much less memory and computation.\n\nDistillation produces a different, smaller model. It doesn’t mean that the original large model has somehow been stored perfectly inside a smaller file. It is possible that the student model may preserve useful behavior while losing some breadth, reliability, or ability to handle unfamiliar problems. In other words, a strong result on a narrow task doesn’t establish equivalence across all tasks.\n\nCreating the student requires training work, but that cost can be paid once, and the resulting model deployed many times. An application developer can also choose an already distilled model.\n\n## Pruning: Remove Work That Contributes Less\n\nPruning attempts to remove weights or larger components whose removal causes an acceptable loss in quality.\n\nOne approach sets selected weights to zero. This produces sparsity, meaning that many entries in the model’s numerical structures are zero. Another approach removes larger structures, such as groups of computation units or complete layers.\n\nHowever, setting values to zero doesn’t automatically make execution cheaper. An ordinary array still occupies space when some entries are zero, and standard calculations may continue processing those entries.\n\nActual savings require an appropriate compressed representation, an execution engine that can skip the removed work, or structural changes that make the model smaller. Hardware support also matters.\n\nTherefore, pruning is useful when the modified model and its execution software work together. Removing too much can damage performance, and additional training may be needed to recover quality.\n\n## The Conversation Has Its Own Memory Requirements\n\nAfter shrinking the weights, another problem can occur. Short questions work, but a long document causes an out-of-memory error.\n\nDuring generation, a transformer stores certain intermediate results from earlier tokens in a key-value cache, usually called the KV cache. Reusing these results avoids repeating some previous calculations. The cache is working data for the sequence, separate from the learned weights.\n\nFor models that retain cached results for the entire sequence, the cache grows as the sequence becomes longer. Serving several distinct conversations generally requires additional cache memory too. The model file stays the same size while the workload’s memory requirements increase.\n\nOne solution is to reduce the amount of context supplied.\n\nA document assistant can retrieve relevant passages instead of inserting entire documents. Conversation history can be limited while preserving information needed for the current task.\n\nAnother option is to quantize the cache or move some of it into CPU memory. These choices can affect quality or speed.\n\n## Better Software Can Make the Same Hardware More Useful\n\nThe inference runtime is the software that executes the model. Its implementation can make a substantial difference even when the weights remain unchanged.\n\nOne example is FlashAttention.\n\nAttention is the mechanism that allows a token’s representation to incorporate relevant information from other tokens. A straightforward implementation can create large intermediate structures and move significant amounts of data through GPU memory.\n\nFlashAttention reorganizes this calculation into blocks and makes better use of the GPU’s fast internal memory. It computes exact attention while reducing memory traffic and intermediate storage requirements. It does not reduce the number of model weights.\n\nAnother example is PagedAttention, which was introduced with vLLM. It manages the KV cache in blocks, reducing memory wasted through inefficient allocation and duplication. This is especially helpful when serving many requests whose lengths change during generation.\n\nServing software can also group requests into batches, allowing multiple requests to share the work of reading and using model weights. This can improve total throughput. However, more simultaneous requests also require more working memory.\n\n## Speculative Decoding: Propose Several Tokens Before Checking Them\n\nSpeculative decoding addresses the sequential nature of text generation.\n\nIn this approach, a small but fast draft model proposes several tokens. The larger target model then evaluates those proposed tokens together. Since the candidate sequence is already available, the target can perform verification with more parallelism than ordinary token-by-token generation permits.\n\nAccepted tokens become part of the output. When a proposal is rejected, the algorithm uses the target model to produce an appropriate correction and continues. The speed improvement depends on how often the draft agrees with the target and the cost of verification.\n\nThe target model still needs to run, and a separate draft model can require additional memory. Speculative decoding therefore becomes useful after basic memory requirements have been addressed. It doesn’t by itself make an oversized target model fit.\n\n## Conclusion\n\nLet us return to our example of a system with 32 GB of RAM and 8 GB of VRAM. An 8B model’s theoretical weight requirement falls from 16 GB to 4 GB through four-bit quantization. A compatible runtime, a moderate context, and one active request provide a reasonable starting point for evaluation.\n\nIf memory remains insufficient, some layers or cache data can be offloaded. If transfers make generation too slow, a smaller model may deliver a better experience. Each adjustment should be checked against realistic tasks.\n\nAn evaluation process should measure answer quality, peak memory use, time to first token, and generation speed. For example, background document processing can tolerate delays that would frustrate a user waiting for code suggestions.\n\nQuantization, offloading, architectural choices, and execution improvements can work together, but their savings don’t simply multiply. They affect different parts of the workload. The goal is a configuration that meets the application’s quality and response-time requirements within its hardware budget.", "url": "https://wpnews.pro/news/how-to-run-a-big-model-on-cheap-hardware", "canonical_source": "https://blog.bytebytego.com/p/how-to-run-a-big-model-on-cheap-hardware", "published_at": "2026-09-21 15:32:11+00:00", "updated_at": "2026-09-21 15:55:47.251598+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "machine-learning", "ai-infrastructure"], "entities": ["ByteByteGo", "WorkOS", "AuthKit", "Unblocked"], "alternates": {"html": "https://wpnews.pro/news/how-to-run-a-big-model-on-cheap-hardware", "markdown": "https://wpnews.pro/news/how-to-run-a-big-model-on-cheap-hardware.md", "text": "https://wpnews.pro/news/how-to-run-a-big-model-on-cheap-hardware.txt", "jsonld": "https://wpnews.pro/news/how-to-run-a-big-model-on-cheap-hardware.jsonld"}}