# Tether solves the memory challenge for AI builders who don’t want to rely on the cloud

> Source: <https://www.cio.com/article/4250405/tether-solves-the-memory-challenge-for-ai-builders-who-dont-want-to-rely-on-the-cloud.html>
> Published: 2026-11-18 20:54:28+00:00

**Key takeaway:** TurboQuant KV-cache quantization in QVAC SDK 0.12.0 cuts a 262,000-token context for a 4B-parameter model from about 8 GB at 16-bit to 1.6 GB, enabling long-context local inference on consumer GPUs.

Thanks to the rise of a new generation of highly [capable](https://qvac.tether.io/products/models) [small](https://qvac.tether.io/products/models) [AI](https://qvac.tether.io/products/models) [models](https://qvac.tether.io/products/models), on-device or local inference has become increasingly viable for a greater range of applications.   

However, limited memory capacity often prevents local AI from reaching its full potential. A model that fits comfortably on a laptop or phone can hit a wall the moment it has to hold a long conversation or search a large database.

To advance on-device AI capabilities, Tether’s [QVAC](https://qvac.tether.io/products/sdk) [SDK](https://qvac.tether.io/products/sdk) tackles the memory challenge from two directions.

By [integrating](https://www.networkworld.com/article/4186147/tether-is-shipping-turboquant-kv-cache-quantization-with-vulkan-support-into-its-qvac-sdk.html) [TurboQuant](https://www.networkworld.com/article/4186147/tether-is-shipping-turboquant-kv-cache-quantization-with-vulkan-support-into-its-qvac-sdk.html) and TurboVec, the software development kit shrinks two of the biggest memory costs in a local AI pipeline: the model’s working memory during inference, and the index it searches when answering questions from your own documents.  

**Compressing what AI remembers**  

Modern language models generate text through a process called [self-attention](https://jalammar.github.io/illustrated-transformer/), in which each token (the basic units of text) gets encoded into three vectors: a query, a key, and a value. The query is what the current token is “looking for”; the key is what each earlier token offers up to be matched against that query; and the value is the actual content pulled forward once a match is found.   

AI models don’t reprocess the whole conversation for every new word. They keep a running store of key and value projections for every token already seen: the KV cache.

Originally developed by Google researchers, [TurboQuant](https://research.google/blog/turboquant-redefining-ai-efficiency-with-extreme-compression/) is an open-source compression algorithm for this KV cache that transforms 16- or 32-bit vectors into smaller ones, without losing too much accuracy. The result is up to a five times reduction in the amount of memory used to store context during inference.  

Reducing the size of a model’s KV cache enables AI running on consumer hardware to process longer context prompts that would otherwise exceed local memory capacity.

Consider a four billion parameter model like Qwen3.5-4B running at 262,000 tokens of context.

While the model weights alone are small enough to fit on a smartphone’s VRAM, with that context, a standard 16-bit KV cache would balloon to around 8 GB.

Accounting for the model itself and the necessary overhead, that’s more than what the average consumer GPU can handle.

In contrast, by applying TurboQuant to the same model, a 262,000 token KV cache is reduced to a much more manageable 1.6 GB.

This orientation toward everyday hardware is exactly what Tether is targeting with QVAC. For instance, TurboQuant was [integrated](https://qvac.tether.io/blog/turboquant-in-qvac-sdk-0-12-0-kv-cache-quantization-for-production-local-ai) [into](https://qvac.tether.io/blog/turboquant-in-qvac-sdk-0-12-0-kv-cache-quantization-for-production-local-ai) [QVAC](https://qvac.tether.io/blog/turboquant-in-qvac-sdk-0-12-0-kv-cache-quantization-for-production-local-ai) [SDK](https://qvac.tether.io/blog/turboquant-in-qvac-sdk-0-12-0-kv-cache-quantization-for-production-local-ai) v0.12.0 with Windows and Linux support for machines equipped with NVIDIA, AMD, and Intel GPUs.   

**Compressing what AI searches**  

While applying TurboQuant to a model’s KV cache streamlines AI inference, another tool in the QVAC SDK, [TurboVec](https://qvac.tether.io/blog/turbovec-local-vector-search), applies the same quantization algorithm to a different bottleneck: vector indexing.  

To respond to user queries, modern AI tools aren’t limited to knowledge contained within their training data; they can also search for relevant information on the internet or specific databases using a technique known as [retrieval-augmented](https://arxiv.org/abs/2005.11401) [generation](https://arxiv.org/abs/2005.11401) (RAG).   

To enable this search, the system turns text into vectors, then orders them so vectors with similar meaning sit close together. Each search then compares your question’s vector against the stored ones and returns passages of text with the closest values.

After applying TurboVec, ten million typical embedding vectors shrink from roughly 61GB to about 8GB. In terms of the necessary RAM, that’s the difference between an expensive custom setup and a standard desktop computer.

In its [original](https://github.com/RyanCodrai/turbovec) [form](https://github.com/RyanCodrai/turbovec), TurboVec was an open-source Rust project developed by Ryan Codrai, but QVAC doesn’t depend on that Rust code. Instead, the Tether team implemented the same quantizer natively in their own stack, equipping SDK users with another powerful compression tool to boost speed and reduce memory requirements.   

*One algorithm, two efficiency boosts*  

For QVAC, TurboQuant and TurboVec apply the same compression math to different ends of the AI pipeline. One shrinks the memory a model uses while it thinks. The other shrinks the memory it searches to find what to think about. For anyone trying to run AI on a desktop or smartphone, the respective VRAM and RAM savings mean greater capabilities for memory-constrained devices.

By bringing RAG on to everyday edge devices, apps can plug into sensitive databases without users having to worry about potential leaks. AI can recall its own usage data and agent memory, or private documents such as legal or medical records without uploading anything to the cloud.

Reducing cloud dependency can also save money. When every gigabyte TurboQuant and TurboVec claw back is a gigabyte that doesn’t have to be rented from a cloud provider, the long-term cost savings are significant.

Interested in learning more? [Discover](https://qvac.tether.io/products/sdk) how the QVAC SDK could help reduce your app’s memory footprint and slash your cloud bill.
