Hi everyone,
I’m an independent technology enthusiast with a strong interest in computer architecture, hardware optimization and artificial intelligence.
While thinking about memory bandwidth and CPU cache limitations during LLM inference, I came up with an idea that I believe could be worth exploring.
I’m not an AI researcher, but I’d like to share this concept with the community and hear your thoughts.
I’ve attached a short document describing the idea, its potential benefits and some practical examples.
I’d be very interested in feedback from researchers and developers who might be willing to evaluate or experiment with this approach.
I would like to propose an approach worth investigating for the next generation of Large Language Models (LLMs): clustering similar neural network weights and replacing groups of sufficiently similar values with shared representative values.
The motivation comes from a potential bottleneck in local AI inference: memory bandwidth.
During autoregressive token generation, a model repeatedly performs matrix-vector operations involving enormous weight matrices. Depending on the model architecture, quantization, hardware and cache hierarchy, a substantial portion of the model’s weights may need to be fetched from system memory repeatedly.
This can make inference memory-bandwidth-bound, particularly on CPUs and systems with limited memory bandwidth.
My hypothesis is that the way weights are currently stored may leave room for a different kind of optimization.
Consider a hypothetical set of neural network weights:
0.8121
0.8118
0.8124
0.8130
0.8115
0.2742
0.2739
0.2745
0.2751
0.2740
-0.6210
-0.6207
-0.6213
Instead of storing every value independently, we could cluster sufficiently similar weights and represent each cluster with a shared centroid:
Cluster A: 0.8122
Cluster B: 0.2743
Cluster C: -0.6210
The original weights would then be represented by references to their corresponding cluster.
For example:
Original:
[0.8121, 0.8118, 0.8124, 0.8130, 0.8115,
0.2742, 0.2739, 0.2745, 0.2751, 0.2740]
Compressed representation:
[0, 0, 0, 0, 0, 1, 1, 1, 1, 1]
Shared values:
[0.8122, 0.2743]
In this simplified example, ten floating-point values become two representative values plus ten small indices.
The important part is that the original values are approximated, not necessarily reproduced exactly.
There are potentially two benefits.
If a large number of weights can be represented by a relatively small number of centroids and compact indices, the model could require significantly less memory.
This could allow:
Larger models to fit into RAM.
More model data to fit into CPU caches.
Reduced memory traffic.
Lower memory consumption on devices with limited RAM.
This is the more interesting aspect of the proposal.
If many weights share the same representative values, the number of distinct weight values that must be stored is reduced.
In principle, this could improve the probability that frequently accessed representative values remain in CPU cache.
However, there is an important implementation detail: simply replacing weights with indices does not automatically improve cache performance. If every matrix operation must perform an additional random lookup into a codebook, the lookup overhead could offset the benefits.
A successful implementation would therefore need to optimize the representation and computation together.
For example, the codebook could be small enough to remain in cache, while packed indices and specialized matrix multiplication kernels could minimize lookup overhead.
Imagine a matrix containing 100 million FP16 weights.
Original storage:
100,000,000 × 2 bytes = 200 MB
Suppose we cluster the weights into 256 representative values.
The codebook would require:
256 × 2 bytes = 512 bytes
Each weight could then be represented by an 8-bit index:
100,000,000 × 1 byte = 100 MB
The total would be approximately 100 MB, plus negligible codebook overhead.
This would reduce the storage requirement by roughly 50%.
If 16-bit indices were used instead, the compression benefit would disappear relative to the original FP16 representation. Therefore, the number of clusters and index precision are critical design parameters.
More aggressive compression could use fewer clusters, variable-length indices, or structured encoding.
Not every layer necessarily has the same weight distribution.
Different layers may have different statistical properties, sensitivity to quantization, and effects on the final model output.
Instead of applying one global codebook, we could investigate:
A separate codebook for each layer.
Multiple codebooks for different blocks within a layer.
Adaptive cluster sizes based on weight distributions.
Different precision levels for sensitive and less-sensitive layers.
The clustering process could also be performed during model conversion or fine-tuning, rather than during inference.
I believe this approach deserves experimental investigation.
Some important questions would be:
How much redundancy or similarity exists among the weights of modern LLMs?
What is the relationship between the number of clusters and model quality?
Can clustering be performed while preserving perplexity and benchmark performance?
Can specialized kernels perform matrix multiplication directly using codebook values and compact indices?
Does this representation actually reduce DRAM bandwidth during autoregressive generation?
Can the codebook be kept in CPU cache, and does that produce measurable performance gains?
How does the approach compare with existing quantization methods such as INT8, INT4, GPTQ and AWQ?
A straightforward experiment could be:
Select an existing open-source LLM.
Analyze the distribution of weights in each layer.
Cluster the weights using k-means or another vector quantization method.
Generate compressed versions using different codebook sizes.
Measure model quality, perplexity and inference speed.
Monitor actual memory-controller traffic during token generation.
Compare CPU cache misses, RAM bandwidth and tokens per second against the original model and conventional quantized versions.
It would be particularly interesting to compare models with the same effective precision and similar memory footprint, so that the benefits of the representation itself can be isolated.
The main idea is not simply to reduce the number of bits used to represent neural network weights.
It is to investigate whether representing many similar weights through a small shared codebook can reduce the amount of unique data that needs to be fetched from memory, improve cache residency, and ultimately increase inference throughput.
Even if the first implementation does not outperform existing quantization techniques, the investigation could reveal useful information about weight distributions, memory access patterns and the relationship between model compression and hardware cache behavior.
I would be very interested in feedback from researchers working on model compression, quantization, CPU inference engines and memory-efficient neural network architectures.
Could this be a useful direction for the next generation of LLM inference optimization?
Thanks for your time!