Week 3, Day 1 of my AI engineering journey — a beginner-friendly walkthrough of quantization, the first technique that makes fine-tuning possible on ordinary hardware.
Catch up on the series so far: Week 1 — Understanding Large Language Models and
This is Day 1 of Week 3 in a self-study series where I’m learning AI engineering from the ground up, in public.
In Week 1, I learned the basics of how large language models actually work — how a model breaks text into small pieces it can understand (tokenization), turns those pieces into numbers (embeddings), and uses an architecture called a transformer to predict what comes next. In Week 2, I fine-tuned a small model for the first time using Supervised Fine-Tuning (SFT).
Then I did some quick research on what it would take to fine-tune a bigger, production-sized model — the kind companies actually deploy — using that same full fine-tuning approach from Week 2.
The answer stopped me: over 100GB of GPU memory, just to train it.
Most laptop GPUs — even in gaming laptops — max out around 8–16GB of VRAM (the GPU’s own memory, separate from your computer’s regular RAM). That’s not a small gap. It’s a wall.
And yet, people fine-tune huge models on ordinary laptops and free cloud accounts all the time. So this series became about answering one question: how?
This piece covers the first technique that makes it possible: quantization. No assumed background — every term is defined before it’s used. You’ll get the problem it solves, the exact system requirements, a short code example, and an analogy to anchor the concept.
Every technique in this guide depends on your hardware, so let’s check yours first — think of it like checking your car’s fuel type before a road trip. Get this wrong and nothing else will work, no matter how well you follow the steps.
Throughout this series, I’ll use a real 6GB laptop GPU (an RTX 4050) as my example. It’s small, which is actually helpful — it’ll show you exactly when each technique becomes necessary, not just nice to have.
Open a terminal and run:
nvidia-smi
Here’s what that output looks like:
+-----------------------------------------------------------------------------------------+| NVIDIA-SMI 610.74 KMD Version: 610.74 CUDA UMD Version: 13.3 |+-----------------------------------------+------------------------+----------------------+| GPU Name Driver-Model | Memory-Usage | GPU-Util Compute M. ||=========================================+========================+======================|| 0 NVIDIA GeForce RTX 4050 ... WDDM | 293MiB / 6141MiB | 0% Default |+-----------------------------------------+------------------------+----------------------+
Three things to look at here, since this can be a lot to take in the first time you see it:
So for this example, the number to write down is 6GB VRAM. Think of VRAM like a moving truck — the model has to fully fit inside it to train. A 6GB truck is small, so big models simply won’t fit as-is. That’s exactly the problem quantization exists to solve.
Write your own number down now — you’ll compare it against the requirements in everything that follows.
Here’s a trap that catches a lot of beginners: having an NVIDIA GPU isn’t enough on its own. PyTorch also needs to be installed in a GPU-enabled version — not the default, CPU-only one. It’s entirely possible to install PyTorch, assume it’s using your GPU, and never notice it’s been quietly running on your CPU the whole time (which can be 10–50x slower).
Install the correct version:
Then confirm it worked:
python -c "import torch; print(torch.cuda.is_available()); print(torch.cuda.get_device_name(0))"
You should see:
TrueNVIDIA GeForce RTX 4050
True means PyTorch found your GPU. If you see False instead, PyTorch installed the CPU-only build — reinstall using the command above before moving on.
This library is what actually performs the 4-bit and 8-bit compression used in quantization. It’s worth confirming it works now, rather than finding out mid-lesson:
python -c "import bitsandbytes as bnb; print('bitsandbytes version:', bnb.__version__)"
If this prints something like bitsandbytes version: 0.43.1, you're set. If it raises an error instead, reinstall it with pip install bitsandbytes — but only after confirming your GPU-enabled PyTorch check above passed, since bitsandbytes depends on it.
You can still follow along with most of this series — just slower for some parts. A free Google Colab account with a T4 GPU covers everything:
Every value stored inside an AI model — every weight — is just a number, and every number takes up memory depending on how precisely it’s stored. Quantization means storing those numbers less precisely, on purpose, in exchange for a big drop in memory use.
Model weights are typically stored using 32 bits per number, called FP32. Think of “bits” as boxes — the more boxes you use for one number, the more precisely you can write it down, but the more storage space it takes.
Here’s a single weight from a real model: 0.723647912. FP32 uses 32 boxes (bits) to store just this one number.
Now here’s the part that makes this a real problem: a model doesn’t have just one weight — it has billions of them. A “7-billion-parameter model” simply means it has 7 billion of these numbers, each one needing its own 32 boxes.
Add all of that up, and you get roughly 28GB just to store the weights — before training even starts. On a 6–8GB GPU, that model won’t even load. It’s simply too big to fit.
Fewer bits per weight means the same model takes up dramatically less space — small enough to fit on much smaller GPUs. Quantized down to INT4, that same weight might get rounded to something like 0.7 — close enough for the model to still work well, but using a fraction of the space.
Same trade-off as saving a photo at different compression levels. A RAW file is huge but perfectly detailed. A compressed JPEG is a fraction of the size, and to the eye, looks nearly identical. Quantization does this to a model’s numbers instead of a photo’s pixels.
from transformers import AutoModelForCausalLM, BitsAndBytesConfigimport torchbnb_config = BitsAndBytesConfig( load_in_4bit=True, bnb_4bit_quant_type="nf4", bnb_4bit_compute_dtype=torch.float16, bnb_4bit_use_double_quant=True,)model = AutoModelForCausalLM.from_pretrained( "Qwen/Qwen2.5-0.5B-Instruct", quantization_config=bnb_config, device_map="auto",)
Qwen2.5-0.5B-Instruct tells you exactly what you're getting, once you know how to read it. Qwen2.5 is the model family. The 0.5B is what matters most — it means 0.5 billion parameters, making this a genuinely small model. Instruct means it's already trained to follow instructions and chat, rather than just predict raw text. As a rule of thumb: look for the "B" number in any model's name (0.5B, 7B, 13B, 70B...) — that number alone tells you roughly how much memory you'll need, before you even load it.
BitsAndBytesConfig tells Hugging Face Transformers how to compress the model as it loads — here, to 4-bit precision using NF4, a scheme tuned for how AI weights are distributed. device_map="auto" places the model on your GPU automatically if one is available.
I ran all four precision levels (FP32, FP16, INT8, INT4) on a 6GB RTX 4050 GPU. But instead of using a full 7-billion-parameter model, I used a much smaller one — just 0.5 billion parameters, about 14 times smaller — so the demo runs quickly and fits easily, even on a small GPU.
That’s also why the numbers below won’t match the 28GB/14GB/7GB figures mentioned earlier — those were for the full 7B model. Scale that same FP32 memory use down by roughly 14x, and 28GB becomes under 2GB, which is exactly why it fits comfortably on a 6GB GPU here.
What does stay the same either way is the pattern: FP32 will still use the most memory of the four, and INT4 will still use the least — just at a smaller scale, because the model itself is smaller.
Mode Memory (MB) Load (s) Inference (s)=================================================fp32 1885.2 6.10 4.70fp16 958.1 2.01 4.21int8 611.4 4.70 13.89int4 445.4 3.11 3.58
inference time = how long the model took to read the prompt and generate its response (up to 40 words in this test). Shorter = faster answers for the end user.
Memory: FP32 → INT4 cut memory usage by 76% (1885.2MB → 445.4MB), closely matching the theoretical 8x reduction.
Speed — and here’s the surprise:
You’d expect fewer bits to always mean faster inference. INT8 breaks that expectation completely.
Your GPU is like a calculator that’s really good at doing math in certain “modes.” INT8 isn’t a mode it’s naturally fast at — so every time it needs to do math, it first converts the number into a mode it is fast at, does the calculation, then converts it back. That extra converting takes time.
INT4 doesn’t have this problem, because the tool we’re using (bitsandbytes) built a shortcut specifically for INT4 that skips most of that converting.
Both INT8 and INT4 run through bitsandbytes — neither is missing or unsupported. The difference is that bitsandbytes handles the two modes differently on the inside, like two different recipes for the same dish.
So INT8 isn’t slower because it’s broken or missing a shortcut — it’s slower because its recipe does one extra safety step that INT4’s recipe doesn’t.
In one line: same tool (bitsandbytes), two different internal methods — INT8’s method includes an extra accuracy-protecting step that costs time; INT4’s method doesn’t.
Bottom line: fewer bits usually saves memory, but doesn’t always save time — it depends on the specific method used for that precision level.
FP32: A neural network is a complex computational model inspired by the structure and function of the human brain, designed to learn and make predictions based on input data through an iterative process of training and testing.INT4: A neural network is a complex algorithm that mimics the way the human brain processes information to learn and make predictions through layers of interconnected nodes.
Same core idea, just worded differently — a good sign that quantization didn’t meaningfully change the output quality here.
This is quantization’s whole trade in one example: give up a small, barely-noticeable amount of precision, and get back a large amount of memory savings.
One last thing — the script doesn’t just print these results, it also saves them to experiment_log.json. That way you can revisit the numbers later — to chart them, compare against a different model, or just keep a record — without re-running the whole test.
Up next: quantization solves the problem — fitting a model into memory. But a model and training it are two different problems entirely. In the next post, I cover LoRA: how to train a model without touching almost any of its original weights.
Follow along with the full code and weekly progress here: [GitHub — NehaKhann/ai-engineering-journey]
How I Fit a Model That “Shouldn’t” Fit on a 6GB Laptop GPU was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.