{"slug": "how-int8-quantization-made-my-neural-network-60-smaller-a-tinyml-model", "title": "How INT8 Quantization Made My Neural Network 60% Smaller: A TinyML Model Compression Experiment", "summary": "An INT8 quantization experiment on a TinyML ECG arrhythmia detection model cut its size from 87.5 KB in FP32 to 34.6 KB, a roughly 60% reduction, while reported accuracy held at 93.8%, according to the Beat2Bit project writeup. The author converted the trained 1D CNN from 32-bit floating point to 8-bit integers to test how much compression the model could take before performance degraded. The result matters because it shows reduced numerical precision can shrink a heartbeat-reading model for constrained hardware without sacrificing its reported accuracy.", "body_md": "Last time, we looked at what happens behind the simple number on your smartwatch: **72 BPM**.\n\nBehind that number is a stream of physiological data, computation, and a model trying to make sense of it.\n\nBut there was one part of that story we hadn’t explored yet.\n\nWhat happens when the AI itself has to get smaller?\n\n**To find out, I started with the model itself.**\n\nThe first technique I tried was **INT8 quantization**, a way to reduce the numerical precision of a neural network so it can use less storage and computation.\n\nIn Part 1, I started with a simple question:\n\n**How much could I compress a TinyML model for ECG arrhythmia detection before its performance started to suffer?**\n\nIf you haven’t read it yet, you can start here: [**Part 1: What Happens When You Shrink an AI That Reads Heartbeats to Just 35 KB?**](https://medium.com/@padgaonkarsneha/what-happens-when-you-shrink-an-ai-that-reads-heartbeats-to-just-35-kb-b150f6a65000)\n\nTo answer that, I first needed a starting point.\n\nThe FP32 1D CNN had the following metrics on the reported MIT-BIH evaluation setup:\n\n**This was the model I was going to shrink.**\n\nThe first compression experiment was **INT8 quantization**.\n\nUnlike pruning, which removes or zeroes weights, quantization takes a different approach:\n\nWhat if I could represent the numbers inside the model using fewer bits?\n\nThat sounds like a small change.\n\nIt turned out to make a surprisingly big difference.\n\nNeural networks commonly represent their weights and intermediate values using numerical formats such as **32-bit floating point**, or FP32.\n\nSo, conceptually, the model starts as:\n\n**FP32 (32 bits per value)** → **INT8 (8 bits per value)**\n\nIn Beat2Bit, I converted the trained model to an **INT8 representation**, replacing the original 32-bit floating-point representation with 8-bit integers for the quantized model.\n\nThe potential benefit is fairly intuitive.\n\nIf each value can be represented using fewer bits, the model can require less storage.\n\nBut there is an obvious problem.\n\n**Less precision means less information.**\n\nConverting FP32 to INT8 was the easy part.\n\n**The real test was seeing whether the model would still behave the same way afterward.**\n\nMaking a model smaller is easy if you’re willing to sacrifice performance.\n\nThe challenge is finding out **how much you can remove before that trade-off becomes unacceptable.**\n\nThink about storing a measurement.\n\nIf I tell you that a room is: **23.472891°C**\n\nI’m giving you a lot of numerical precision.\n\nInstead if I tell you: **23.5°C**\n\nI’ve lost some precision.\n\nBut if the only question is whether the room is comfortable, that extra precision might not matter.\n\nQuantization makes a similar trade.\n\nThe goal isn’t necessarily to preserve every decimal place.\n\nIt is to preserve the **useful behaviour of the model**.\n\nNeural networks can sometimes tolerate reduced numerical precision because not every tiny difference in a parameter necessarily changes the final prediction.\n\nBut that’s only a hypothesis until you test the specific model.\n\nAnd that’s exactly what I wanted to find out with Beat2Bit.\n\nThis was an important distinction for the experiment.\n\nI wasn’t trying to win a competition for the smallest possible .tflite file.\n\nI wanted to find a useful balance between three things:\n\nA model that is tiny but performs poorly isn’t particularly useful.\n\nA highly accurate model that is too large for the intended hardware has a different problem.\n\nAnd a model that looks extremely fast on a benchmark isn’t enough if the benchmark doesn’t represent the actual deployment hardware.\n\nSo I needed to keep all three in view.\n\nThis was the first result that made me stop and look twice.\n\nThe reported FP32 model was: **87.5 KB**\n\nAfter INT8 quantization: **34.6 KB**\n\nThat’s a reduction of roughly **60% in model size**.\n\nBut the more interesting part wasn’t the size.\n\nIt was what happened to the performance.\n\nThe reported accuracy stayed at **93.8%**.\n\nSensitivity stayed at **75.0%**.\n\nPositive predictivity moved from **71.3% to 71.2%**.\n\nSo I had gone from: **87.5 KB → 34.6 KB**\n\nwithout a meaningful change in those reported headline metrics.\n\nThat was a much better result than I expected.\n\nBut I didn’t want to stop at the headline numbers.\n\nThere’s an important distinction here.\n\nQuantization doesn’t mean that I deleted 60% of the model’s learned knowledge.\n\nThe **60% reduction refers to the reported model artifact size**, going from 87.5 KB to 34.6 KB.\n\nThe underlying idea is different:\n\nInstead of representing numerical values with the original FP32 representation, the quantized model uses a lower-precision INT8 representation.\n\nSo:\n\n**Smaller representation ≠ less learned information**\n\nThat’s why the performance results matter.\n\nIf reducing numerical precision had caused a major degradation, the smaller model wouldn’t necessarily have been a useful trade.\n\nBut in this experiment, the reported INT8 model **retained the baseline’s accuracy and sensitivity**, with only a very small change in positive predictivity.\n\nThat made quantization look like a very promising first step.\n\nModel size was only one part of the problem.\n\nThe reported benchmark showed a substantial difference in inference time:\n\n**FP32: 18.0 ms**\n\n**INT8: 0.073 ms**\n\nThat’s roughly **247× faster** according to the project’s benchmark.\n\nThat number definitely caught my attention.\n\nBut there was an important detail behind it.\n\nThe **0.073 ms** measurement came from running the .tflite model on a **desktop CPU**, not on a physical microcontroller.\n\nSo while the benchmark showed a big improvement, I still couldn’t tell how the model would perform on the hardware I ultimately wanted to target.\n\n**That was something I would need to test for real.**\n\nFor now, the result was still useful. It showed that reducing the model’s numerical precision could have a much bigger effect than just cutting down its file size.\n\nAnd that made me wonder:\n\n**What else could I remove without breaking the model?**\n\nThis is where I moved from **quantization** to **pruning**.\n\nA neural network contains many weights, and not all of them contribute equally to the final prediction.\n\nPruning takes advantage of that idea.\n\nInstead of asking:\n\nHow can I represent all these weights more efficiently?\n\nit asks:\n\n**Do I need all these weights in the first place?**\n\nThe basic idea is to identify weights with relatively small magnitudes and set selected weights to zero\n\nConceptually:\n\nwhere τ is a threshold.\n\nThe result is a **sparse network**.\n\nFor example:\n\n**Before pruning:** [0.82, -0.03, 0.71, 0.01, -0.44]\n\n**After pruning:** [0.82, 0.00, 0.71, 0.00, -0.44]\n\nBeat2Bit tested **50% and 60% pruning levels**, followed by INT8 quantization.\n\nOn paper, this sounded like the obvious next step.\n\nIf INT8 had already made the model dramatically smaller, removing half the weights should make it smaller still.\n\nRight?\n\n**Not exactly.**\n\nThis was one of the most useful discoveries in the entire experiment.\n\nThe **50% pruned + INT8** model was reported at: **34.6 KB**\n\nThe **60% pruned + INT8** model was also: **34.6 KB**\n\nThe same reported size as the INT8 model without pruning.\n\nThat sounds strange at first.\n\nIf half the weights are zero, shouldn’t the file be half the size?\n\nNot necessarily.\n\nThere’s a difference between:\n\n**A neural network containing many zero weights**\n\nand\n\n**A file format that actually stores those zero weights more efficiently.**\n\nConsider:\n\n[0.82, 0.00, 0.71, 0.00, -0.44]\n\nIf the representation still stores every position, including the zeros, then those zeros haven’t disappeared from the file.\n\nPruning has created **sparsity**.\n\nIt hasn’t necessarily created **storage compression**.\n\nAnd that’s what happened here.\n\nThe serialized model remained **34.6 KB** after both 50% and 60% pruning.\n\n**50% pruning ≠ 50% smaller model file**\n\nand:\n\n**60% pruning ≠ 60% smaller model file**\n\nThat distinction completely changed how I thought about pruning.\n\nNow the comparison became much more interesting:\n\nAnd suddenly, the experiment wasn’t simply about compression anymore.\n\nIt became a question about **trade-offs**.\n\nINT8 gave me a substantial reported reduction in model size while preserving the baseline’s reported accuracy and sensitivity.\n\nBut pushing pruning further did something different.\n\nThe reported model size didn’t decrease further.\n\nMeanwhile, positive predictivity dropped from **71.2% for INT8 to 61.2% at 50% pruning and 54.9% at 60% pruning**.\n\nThat changes the question completely.\n\nThe goal shifted from compressing as much as possible to finding how much compression was actually useful.\n\nThis sounds obvious when you say it out loud.\n\nBut it is surprisingly easy to forget when you’re optimizing a machine learning model.\n\nIf you optimize only for model size, you might celebrate every reduction.\n\nIf you optimize only for latency, you might celebrate every faster benchmark.\n\nIf you optimize only for accuracy, you might ignore whether the model can actually run where you need it.\n\nReal deployment sits somewhere in the middle.\n\nBeat2Bit made that trade-off visible.\n\n**INT8 quantization** gave me the biggest win. The model became much smaller while its reported accuracy and sensitivity stayed the same.\n\nI then tried **pruning**, which removes weights the model seems to need less. But that came with a surprise: removing more weights didn’t make the model file any smaller, and its positive predictivity started to suffer.\n\nThat taught me something important:\n\nAn optimization is only useful if the thing you’re optimizing actually changes in the way you intended.\n\nMaking a network sparse is one thing.\n\nMaking the deployed artifact smaller is another.\n\nAnd improving a benchmark number is not automatically the same as improving a real deployment.\n\nBeat2Bit started with the idea of making a model smaller.\n\nBut the experiments showed me that **smaller doesn’t automatically mean better**.\n\nINT8 gave me a major reduction in model size while keeping the reported performance almost unchanged.\n\nPruning went further, but the extra sparsity didn’t make the model file smaller and started to hurt positive predictivity.\n\nThat changed how I think about optimization.\n\n**The best model isn’t necessarily the smallest one but the one that makes the right trade-offs for where it needs to run.**\n\nAnd that’s what makes edge AI interesting!\n\n🔗 **Want to explore the experiment yourself?** Beat2Bit is open source. Explore the implementation on [**GitHub**](https://github.com/snehapadgaonkar/beat2bit)**.**\n\n**What would you optimize first: size, speed, or performance?**\n\n[How INT8 Quantization Made My Neural Network 60% Smaller: A TinyML Model Compression Experiment](https://pub.towardsai.net/how-int8-quantization-made-my-neural-network-60-smaller-a-tinyml-model-compression-experiment-23791ba9b485) was originally published in [Towards AI](https://pub.towardsai.net) on Medium, where people are continuing the conversation by highlighting and responding to this story.", "url": "https://wpnews.pro/news/how-int8-quantization-made-my-neural-network-60-smaller-a-tinyml-model", "canonical_source": "https://pub.towardsai.net/how-int8-quantization-made-my-neural-network-60-smaller-a-tinyml-model-compression-experiment-23791ba9b485?source=rss----98111c9905da---4", "published_at": "2026-09-12 10:31:04+00:00", "updated_at": "2026-09-12 11:04:51.021396+00:00", "lang": "en", "topics": ["machine-learning", "ai-research", "ai-tools"], "entities": ["INT8 quantization", "Beat2Bit", "MIT-BIH", "FP32", "TinyML"], "alternates": {"html": "https://wpnews.pro/news/how-int8-quantization-made-my-neural-network-60-smaller-a-tinyml-model", "markdown": "https://wpnews.pro/news/how-int8-quantization-made-my-neural-network-60-smaller-a-tinyml-model.md", "text": "https://wpnews.pro/news/how-int8-quantization-made-my-neural-network-60-smaller-a-tinyml-model.txt", "jsonld": "https://wpnews.pro/news/how-int8-quantization-made-my-neural-network-60-smaller-a-tinyml-model.jsonld"}}