Last time, we looked at what happens behind the simple number on your smartwatch: 72 BPM.
Behind that number is a stream of physiological data, computation, and a model trying to make sense of it.
But there was one part of that story we hadn’t explored yet.
What happens when the AI itself has to get smaller?
To find out, I started with the model itself.
The first technique I tried was INT8 quantization, a way to reduce the numerical precision of a neural network so it can use less storage and computation.
In Part 1, I started with a simple question:
How much could I compress a TinyML model for ECG arrhythmia detection before its performance started to suffer?
If you haven’t read it yet, you can start here: Part 1: What Happens When You Shrink an AI That Reads Heartbeats to Just 35 KB? To answer that, I first needed a starting point.
The FP32 1D CNN had the following metrics on the reported MIT-BIH evaluation setup:
This was the model I was going to shrink.
The first compression experiment was INT8 quantization.
Unlike pruning, which removes or zeroes weights, quantization takes a different approach:
What if I could represent the numbers inside the model using fewer bits?
That sounds like a small change.
It turned out to make a surprisingly big difference.
Neural networks commonly represent their weights and intermediate values using numerical formats such as 32-bit floating point, or FP32.
So, conceptually, the model starts as:
FP32 (32 bits per value) → INT8 (8 bits per value) In Beat2Bit, I converted the trained model to an INT8 representation, replacing the original 32-bit floating-point representation with 8-bit integers for the quantized model.
The potential benefit is fairly intuitive.
If each value can be represented using fewer bits, the model can require less storage. But there is an obvious problem.
Less precision means less information.
Converting FP32 to INT8 was the easy part.
The real test was seeing whether the model would still behave the same way afterward.
Making a model smaller is easy if you’re willing to sacrifice performance.
The challenge is finding out how much you can remove before that trade-off becomes unacceptable.
Think about storing a measurement.
If I tell you that a room is: 23.472891°C I’m giving you a lot of numerical precision.
Instead if I tell you: 23.5°C
I’ve lost some precision.
But if the only question is whether the room is comfortable, that extra precision might not matter.
Quantization makes a similar trade.
The goal isn’t necessarily to preserve every decimal place.
It is to preserve the useful behaviour of the model.
Neural networks can sometimes tolerate reduced numerical precision because not every tiny difference in a parameter necessarily changes the final prediction.
But that’s only a hypothesis until you test the specific model.
And that’s exactly what I wanted to find out with Beat2Bit.
This was an important distinction for the experiment.
I wasn’t trying to win a competition for the smallest possible .tflite file.
I wanted to find a useful balance between three things:
A model that is tiny but performs poorly isn’t particularly useful.
A highly accurate model that is too large for the intended hardware has a different problem.
And a model that looks extremely fast on a benchmark isn’t enough if the benchmark doesn’t represent the actual deployment hardware.
So I needed to keep all three in view.
This was the first result that made me stop and look twice.
The reported FP32 model was: 87.5 KB
After INT8 quantization: 34.6 KB
That’s a reduction of roughly 60% in model size.
But the more interesting part wasn’t the size.
It was what happened to the performance.
The reported accuracy stayed at 93.8%.
Sensitivity stayed at 75.0%.
Positive predictivity moved from 71.3% to 71.2%.
So I had gone from: 87.5 KB → 34.6 KB
without a meaningful change in those reported headline metrics.
That was a much better result than I expected.
But I didn’t want to stop at the headline numbers.
There’s an important distinction here.
Quantization doesn’t mean that I deleted 60% of the model’s learned knowledge.
The 60% reduction refers to the reported model artifact size, going from 87.5 KB to 34.6 KB.
The underlying idea is different:
Instead of representing numerical values with the original FP32 representation, the quantized model uses a lower-precision INT8 representation.
So:
Smaller representation ≠ less learned information
That’s why the performance results matter.
If reducing numerical precision had caused a major degradation, the smaller model wouldn’t necessarily have been a useful trade. But in this experiment, the reported INT8 model retained the baseline’s accuracy and sensitivity, with only a very small change in positive predictivity.
That made quantization look like a very promising first step.
Model size was only one part of the problem.
The reported benchmark showed a substantial difference in inference time:
FP32: 18.0 ms
INT8: 0.073 ms
That’s roughly 247× faster according to the project’s benchmark.
That number definitely caught my attention.
But there was an important detail behind it.
The 0.073 ms measurement came from running the .tflite model on a desktop CPU, not on a physical microcontroller.
So while the benchmark showed a big improvement, I still couldn’t tell how the model would perform on the hardware I ultimately wanted to target.
That was something I would need to test for real.
For now, the result was still useful. It showed that reducing the model’s numerical precision could have a much bigger effect than just cutting down its file size. And that made me wonder:
What else could I remove without breaking the model?
This is where I moved from quantization to pruning.
A neural network contains many weights, and not all of them contribute equally to the final prediction.
Pruning takes advantage of that idea.
Instead of asking:
How can I represent all these weights more efficiently?
it asks:
Do I need all these weights in the first place?
The basic idea is to identify weights with relatively small magnitudes and set selected weights to zero
Conceptually:
where τ is a threshold.
The result is a sparse network.
For example:
**Before pruning:** [0.82, -0.03, 0.71, 0.01, -0.44]
**After pruning:** [0.82, 0.00, 0.71, 0.00, -0.44]
Beat2Bit tested 50% and 60% pruning levels, followed by INT8 quantization.
On paper, this sounded like the obvious next step.
If INT8 had already made the model dramatically smaller, removing half the weights should make it smaller still. Right?
Not exactly.
This was one of the most useful discoveries in the entire experiment.
The 50% pruned + INT8 model was reported at: 34.6 KB
The 60% pruned + INT8 model was also: 34.6 KB
The same reported size as the INT8 model without pruning.
That sounds strange at first.
If half the weights are zero, shouldn’t the file be half the size? Not necessarily.
There’s a difference between:
A neural network containing many zero weights
and
A file format that actually stores those zero weights more efficiently.
Consider:
[0.82, 0.00, 0.71, 0.00, -0.44]
If the representation still stores every position, including the zeros, then those zeros haven’t disappeared from the file.
Pruning has created sparsity.
It hasn’t necessarily created storage compression.
And that’s what happened here.
The serialized model remained 34.6 KB after both 50% and 60% pruning.
50% pruning ≠ 50% smaller model file
and:
60% pruning ≠ 60% smaller model file
That distinction completely changed how I thought about pruning.
Now the comparison became much more interesting:
And suddenly, the experiment wasn’t simply about compression anymore.
It became a question about trade-offs.
INT8 gave me a substantial reported reduction in model size while preserving the baseline’s reported accuracy and sensitivity.
But pushing pruning further did something different.
The reported model size didn’t decrease further.
Meanwhile, positive predictivity dropped from 71.2% for INT8 to 61.2% at 50% pruning and 54.9% at 60% pruning.
That changes the question completely.
The goal shifted from compressing as much as possible to finding how much compression was actually useful.
This sounds obvious when you say it out loud.
But it is surprisingly easy to forget when you’re optimizing a machine learning model.
If you optimize only for model size, you might celebrate every reduction.
If you optimize only for latency, you might celebrate every faster benchmark.
If you optimize only for accuracy, you might ignore whether the model can actually run where you need it.
Real deployment sits somewhere in the middle.
Beat2Bit made that trade-off visible.
INT8 quantization gave me the biggest win. The model became much smaller while its reported accuracy and sensitivity stayed the same.
I then tried pruning, which removes weights the model seems to need less. But that came with a surprise: removing more weights didn’t make the model file any smaller, and its positive predictivity started to suffer.
That taught me something important:
An optimization is only useful if the thing you’re optimizing actually changes in the way you intended.
Making a network sparse is one thing.
Making the deployed artifact smaller is another.
And improving a benchmark number is not automatically the same as improving a real deployment.
Beat2Bit started with the idea of making a model smaller.
But the experiments showed me that smaller doesn’t automatically mean better.
INT8 gave me a major reduction in model size while keeping the reported performance almost unchanged.
Pruning went further, but the extra sparsity didn’t make the model file smaller and started to hurt positive predictivity.
That changed how I think about optimization.
The best model isn’t necessarily the smallest one but the one that makes the right trade-offs for where it needs to run.
And that’s what makes edge AI interesting!
🔗 Want to explore the experiment yourself? Beat2Bit is open source. Explore the implementation on GitHub.
What would you optimize first: size, speed, or performance?
How INT8 Quantization Made My Neural Network 60% Smaller: A TinyML Model Compression Experiment was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.