AI/ML
Lightbits says its Inferra software accelerates KV cache operations, breaking down the memory wall, and improving GPU utilization.
High-performance block storage software supplier Lightbits has developed its Inferra software to virtualize a GPU’s high-bandwidth memory (HBM). As we wrote here, a key-value cache (KV cache) is a mechanism used to store past Gen AI large language model (LLM) layers’ activations (keys and values) during inferencing. It allows LLMs to bypass recomputation of these activations, improving performance. The cache serves as a repository to “remember” previous information, the pre-computed key and value pairs, reducing the need to reprocess entire sequences repeatedly.
The KV cache is held in data blocks in a GPU’s high-bandwidth memory (HBM). When that is full and new KV pairs are generated they go into HBM and existing content is evicted. If it is needed again then it has to be recomputed, taking time. A KV cache scheme extends the cache to include, first, the DRAM in the GPU server’s embedded x86 server, then local NVMe SSDs in that server (tier zero), and then external, network-linked NVMe SSDs. Each tier in this KV cache scheme takes longer to access than the preceding tier, but, if enough tokens have to have their vectors recomputed, then the recomputation time can be longer than the access time to that tier.
Inferra software is designed to mitigate the memory bottlenecks created by steadily increasing KV cache sizes with long context lengths, and is intended to enable Neoclouds and enterprises to run AI inference faster, scale more, and not spend so much cash on adding GPUs and their built-in HBM to extend capacity before tiered KV caching comes into play.
Avigdor Willenz, Co-Founder and Chairman of Lightbits Labs, said: ”The AI inference market will top $117 billion this year, forcing legacy vendors designed for model training and ingest to scramble and re-architect. Inference is a completely different paradigm - retrofitting legacy training and storage systems to solve the KV cache bottleneck simply doesn’t work. It was built for a different era.”
He added: “We engineered Inferra from the ground up specifically to solve the GPU efficiency problem, and it has already proven successful in customer beta programs. Disrupting the status quo with tech that raises the bar for data infrastructure performance and efficiency is in our DNA. Applying those same principles to Inference that we applied when inventing NVMe over TCP makes Inferra uniquely positioned to deliver significant value for this market.”
Inferra works by prefetching KV Cache data, using cache algorithms specific to the KV Cache workload, before it is needed by the GPUs.
Inferra provides:
- Up to 16x more concurrent inference sessions on existing GPU infrastructure, enabling customers to run multi-session clusters with strict SLA guarantees, and without hardware upgrades,
- 100x Latency Improvement & 10M-Token Context Windows which “drastically lowers Time-to-First-Token (TTFT) and Time Per Output Token (TPOT) by proactively and predictively pre-fetching attention states from storage rather than recomputing them,”
- Secure tenant isolation and intelligent KV cache management across shared inference infrastructure.
Lightbits says Inferra fundamentally changes the economics of inference by virtualizing GPU memory across memory and storage tiers, and transforming the KV cache into an intelligent, persistent data layer.
We wrote in March, “It runs in a GPU server’s embedded x86 system and monitors KV Cache data block activity. With this data it runs a Sub-Linear Sparse Attention Prefetch (SLSAP) engine to identify KV blocks most likely to be needed next. The engine uses techniques like locality-sensitive hashing (LSH) combined with statistical reuse patterns (observing historical access locality in attention computations) to score and rank KV blocks, and then identify and select the ones most likely to be needed next by the GPUs. … A second algorithm is based on understanding that recent tokens, semantically similar tokens, and certain structural patterns (e.g., in RAG or multi-turn chat) tend to be reused frequently.”
Inferra fetches these token blocks from the x86 server’s DRAM or, if not there, from external SSDs, and pre-loads them into the GPU’s HBM over RDMA links.
Yaniv Fdida, Chief Product & Technology Officer at OVH, confirmed this, saying: “With Inferra’s intelligent KV cache tiering, we were able to demonstrate substantial GPU utilization gains, paving the way to providing our customers a more scalable and cost-effective foundation for their AI agent and RAG workloads.”
Public time to first token (TTFT) benchmarks include these numbers:
- Qwen 2.5-7B @410K - 72.6 seconds shortened to 711 msec with Inferra; a 102x speed up,
- DeepSeek-R1-70B @ 141k - 70.8 seconds reduced to 465 msec; a 152x improvement,
- Llama-4-Scout @ 10M - 1.5 hours lowered to13 sec; a 380x speed up.
Inferra works on any storage and GPU environment. Lightbits says it’s the only player with algorithmic sub-linear TTFT scaling vs. context length — every named competitor scales linearly or streams reactively, meaning DDN, VAST Data, WEKA and others.
A 10M-token context has been demonstrated on commodity L40S GPUs; no other vendor has published an equivalent benchmark.
Inferra is HW agnostic with deployment supported on any open-source software framework. It has dynamo, vLLM and LMCache plugins.
Ramesh Chettuvetty, SVP of AI Product and Business, Lightbits Labs, said: ”Inferra delivers instant payback and generates net positive savings from day one. It eliminates the stalls that traditionally limit long-context performance, so Neoclouds and enterprises can serve larger models and longer conversations at lower infrastructure cost.”
Lightbits has a Pod Efficiency Analyzer which Neoclouds and other interested parties can access to see what difference Inferra could make to their inferencing costs.
Bootnote
In March Lightbits Labs and ScaleFlux produced a 100x to 280x speed up of KV cache workloads using Inferra KV cache SW reading data off ScaleFlux computational storage SSDs.