The "memory wall" is the single biggest bottleneck for edge AI; fitting a 27B parameter model into a phone's RAM is useless if the SoC bus can't move that data fast enough to generate tokens without overheating the device or killing the battery. Even with extreme quantization like PrismML's ternary weights, the physical act of transferring gigabytes of data across the bus for every single token is inefficient.
This is a necessary evolution for on-device AI. If we want a truly functional "Semantic Router" or a sophisticated agent running locally on a smartphone, we have to stop treating memory as a passive storage bin and start treating it as a compute element. This is a practical step toward actual AI-capable silicon that doesn't melt the chassis.
CaSA solves this by implementing Processing-In-Memory (PIM) for ternary LLM inference. Instead of moving data to the compute unit, it performs the inference natively inside COTS DRAM using charge-sharing. This effectively bypasses the memory bus entirely, addressing the thermal throttling and power drain issues that plague current NPU implementations.
For those looking for a deep dive into the architecture, the project is available here:
https://github.com/pcdeni/CaSA
Technical Breakdown #
The Problem: Standard LPDDR bandwidth limitations. Even quantized models require massive data movement, leading to high energy consumption per token.The Solution: CaSA (Charge-Sharing Architecture) leverages the physical properties of DRAM to execute operations where the data lives.Hardware Impact: By eliminating the need to shuttle weights across the SoC bus, it reduces the power overhead of the "data movement" phase of inference.Synergy: This works best when paired with ternary weights (1.58-bit), as the simplified weight values make in-memory computation physically viable.
This is a necessary evolution for on-device AI. If we want a truly functional "Semantic Router" or a sophisticated agent running locally on a smartphone, we have to stop treating memory as a passive storage bin and start treating it as a compute element. This is a practical step toward actual AI-capable silicon that doesn't melt the chassis.
Next Real-Time AI Tracker: A Practical Tutorial →
All Replies （4） #
Z
Finally. My local Llama setup crawls because of memory bandwidth, not the GPU.
0
A
Wait, is that why my tokens-per-second are so low? I didn't realize bandwidth was the main bottleneck.
0
J
Tried something similar with a custom kernel; latency was atrocious. Just another overhyped whitepaper until there's actual hardware.
0
L
Hope they've considered the power draw, as RAM-heavy tasks usually kill mobile battery life quickly.
0