cd /news/artificial-intelligence/hibana-fast-agentic-inference-on-uni… Β· home β€Ί topics β€Ί artificial-intelligence β€Ί article
[ARTICLE Β· art-170383] src=sharar-project.github.io β†— pub= topic=artificial-intelligence verified=true sentiment=↑ positive

Hibana: Fast Agentic Inference on Unified-Memory Devices

A Cornell University-led research team published Hibana on November 4, 2026, a co-designed model, memory and decoding system that runs many local agents on a single DGX Spark unified-memory device. Hibana uses 3-bit weights with an 8-entry FP8 codebook (3.25 bits per weight including an 8-bit scale per 32 weights), INT8 recurrent state, a 2-bit draft head that shortlists 16 of over 100,000 tokens and agrees with the FP8 head on 99.76% of tokens, and SSD spilling of agent state; on Qwen3.8-Flash-Next with 192 conversations the SSD tier lifts cache hits to 80–98% and cuts time to first token by up to 42Γ— against vLLM, llama.cpp and TensorFold baselines. The work targets the 128 GB shared memory pool and limited bandwidth of unified-memory devices, where every generated token streams the weights.

read4 min views3 publishedNov 16, 2026

Paper Β· November 4, 2026

On a unified-memory device, the model and every agent's history share one memory. Hibana co-designs the model, its memory, and its decoding so many agents run fast on a single DGX Spark.

<sup>1</sup>Cornell University<sup>2</sup>University of Illinois Urbana-Champaign<sup>3</sup>Independent researcher<sup>4</sup>National Yang Ming Chiao Tung University<sup>*</sup>Equal contribution

Qwen3.8-Flash-Next on one DGX Spark, against the fastest of vLLM, llama.cpp, and TensorFold.

Local agents extend long histories, for tools, and run side by side. On a unified-memory device such as the DGX Spark, the weights, the active requests, and every saved history share one 128 GB pool, and every generated token streams the weights through limited bandwidth. Hibana attacks both limits at once.

Model surgery. Edit the trained model to fit the hardware: 3-bit weights, INT8 recurrent state, and a 2-bit draft head, so each token reads fewer bytes.

Unified-memory management. Keep every agent's state across turns, spilling it to the SSD when GPU memory runs out, so a returning agent resumes instead of recomputing.

Adaptive speculation. Share experts across drafted tokens and draft only as far as acceptance pays for, so each weight read yields more tokens.

Decoding one request is bound by memory bandwidth, so the fastest token is the one that reads the fewest bytes.

Each group of 32 weights snaps to an 8-entry FP8 codebook, so each weight is stored as a 3-bit index. The kernel looks up the codes and feeds FP8 tensor cores directly.

Eight values per group, three bits per weight. Codes take 3 bits and each group of 32 adds one 8-bit scale: 3 + 8/32 = 3.25 bits per weight. The codebook here is fitted to the group for illustration.

The drafter's output head scores every token in the vocabulary and is the largest matrix the drafter reads. A 2-bit copy is cheap but noisy, so Hibana uses it only to shortlist candidates, then rescores the shortlist in FP8.

Shortlist in 2 bits, decide in FP8. Context: for i in range(. The served head shortlists 16 of over 100,000 tokens and agrees with the FP8 head on 99.76% of tokens.

Agents take turns. By the time an agent returns, other agents have usually pushed its state out of GPU memory.

Discard an evicted agent's state and its next turn recomputes the whole context. Hibana spills the KV cache, recurrent state, and draft state to the SSD instead, and reads them back when the agent returns.

Spill, don't discard. Illustration with 8 agents and room for 4 in GPU memory. On Qwen3.8-Flash-Next with 192 conversations, the SSD tier lifts cache hits to 80–98% and cuts time to first token by up to 42Γ— against the baselines.

A drafter proposes several tokens and the target checks them in one pass. Hibana makes each of these rounds cheaper and sizes it to what will be accepted.

In a mixture-of-experts layer, verifying drafted tokens loads every expert any of them picked. Hibana gives the round a shared expert budget, so it reads fewer weights.

Fewer experts per round. Toy example: 6 tokens, 12 experts, top-2 routing. Shaded columns are the experts read from memory; β†Ί marks a second choice replaced by one inside the budget.

A longer draft can yield more tokens per round, but each extra draft token costs time to draft and verify. Hibana picks the draft length that maximizes expected tokens per unit time:

Draft only as far as acceptance pays for. Idealized model. Bars show E[N<sub>d</sub>] / T(d) for each draft length; the red bar is d*. High acceptance earns long drafts.

Four models on one DGX Spark against vLLM, Flare, TensorFold, and llama.cpp.

Faster for one user, and for many. Left: SPEED-Bench with 16k prompts, one point per concurrency level; up and to the right is better. Right: AgentPerf-Local agent replay. Hover a point for its numbers.

GPQA LCB IFBench AIME25 AIME26
Exact 90.7 87.7 85.3 100.0 100.0
+ 3-bit weights 90.4 βˆ’0.3 85.4 βˆ’2.3 83.3 βˆ’2.0 99.2 βˆ’0.8 99.2 βˆ’0.8
+ Expert sharing 90.4 βˆ’0.3 85.4 βˆ’2.3 83.3 βˆ’2.0 99.2 βˆ’0.8 99.2 βˆ’0.8
+ Relaxed verif. (fast) 89.1 βˆ’1.6 84.6 βˆ’3.1 81.7 βˆ’3.6 99.2 βˆ’0.8 99.2 βˆ’0.8

Within 3.6 points of the exact model. Task accuracy (%) on Qwen3.8-Flash-Next as each lossy optimization is added; small numbers are the drop from the exact model.

Install uv, then:

git clone https://github.com/sharar-project/hibana && cd hibana
uv sync
uv run hibana serve 27b          # Qwen3.8-27B on http://127.0.0.1:30001
uv run hibana serve flashnext    # Qwen3.8-Flash-Next (125B MoE)
uv run hibana launch claude      # point a coding agent at it

OpenAI- and Anthropic-compatible APIs, 262K context, one Spark or two over ConnectX. Add --exact for the model's own outputs.

@misc{emara2026hibana,
  title  = {Hibana: Fast Agentic Inference on Unified-Memory Devices},
  author = {Emara, Yahya and Chang, Chi-Chih and Fei, Anthony and
            AbouElhamayed, Ahmed F. and Chen, Chong-Yan and Chiang, Hung-Yueh and
            Wu, Kai-Chiang and Abdelfattah, Mohamed S.},
  year   = {2026},
  note   = {Preprint}
}
── more in #artificial-intelligence 4 stories Β· sorted by recency
── more on @hibana 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain β€” perfect for shipping the agent you just read about.

$git push zahid main
β†’ Live at https://your-agent.zahid.host βœ“
Get free account β†’ Pricing
from €0/mo Β· no card required
LIVE [news/hibana-fast-agentic-…] indexed:0 read:4min 2026-11-16 Β· β€”