Cognitora v0.9.1: gossip discovery, live KV prefix, and a native engine

Cognitora released v0.9.1 in September 2026, adding per-model inflight caps on the gateway, fleet tokens_per_watt gauges, and an energy benchmark harness to its open-source LLM inference orchestration layer. The release follows six minor versions since v0.3.0 in May, which introduced gossip discovery (state_backend = "gossip"), etcd-optional operation, and a native cgn-infer engine built on Candle with mmap GGUF support. Cognitora positions itself as a bare-metal-first, Rust-only, engine-agnostic alternative to NVIDIA Dynamo, coordinating vLLM, SGLang, TensorRT-LLM, llama.cpp, and MLX into a KV-aware disaggregated cluster.

Cognitora v0.9.1: gossip discovery, live KV prefix, and a native engine Cognitora https://github.com/antonellof/cognitora-inference is an open-source LLM inference orchestration layer. Statically linked Rust binaries sit above vLLM, SGLang, TensorRT-LLM, llama.cpp, and MLX. Those engines stay the token factories. Cognitora coordinates them into a KV-aware, disaggregated cluster. The router scores replicas by prefix overlap with sequence-chained BLAKE3 digests. Prefill and decode run on different GPUs with NIXL between them when you ask for disaggregation. Tiered KV spill lives in cgn-kvcached . No Python control plane. No hard Kubernetes dependency. The same artifacts run as systemd units on a rack, as recipes on one host, or via Helm. One node with vLLM saturates well. None of the token factories are a fleet. Hot prefixes land on the wrong replica. Prefill wants different hardware than decode. You end up writing routing, cache, and deployment layers yourself. NVIDIA Dynamo https://github.com/ai-dynamo/dynamo covers part of the space. Cognitora lands bare-metal-first, Rust-only, and engine-agnostic. The original post https://www.fratepietro.com/2026/cognitora-inference-llm-orchestration/ covers the full architecture and the Dynamo comparison. The May write-up described v0.3.0 https://github.com/antonellof/cognitora-inference/releases/tag/v0.3.0 : real /v1/embeddings , a Kubernetes CPU quickstart on GKE, cgn-ctl install --apply , and Llama 3 8B/70B recipes. Six months and six minor releases later, Cognitora sits at v0.9.1 https://github.com/antonellof/cognitora-inference/releases/tag/v0.9.1 September 2026 . The thesis stayed put: one Rust runtime, KV reuse as the routing signal, bare-metal-first. Most of what I called “designed but not fully wired” in May is wired now. A few bets landed after the May write-up. Pin the current release curl -fsSL https://inference.cognitora.dev/install | CGN VERSION=v0.9.1 sh Fastest sanity check: 8B aggregated on one GPU bash recipes/llama3-8b/vllm/agg/up.sh Cognitora-native KV offload into cgn-kvcached preview bash recipes/llama3-8b/vllm/agg-cgn/up.sh Disagg with CognitoraConnector + NIXL multi-connector preview, 2 GPUs bash recipes/llama3-8b/vllm/disagg-cgn/up.sh Native GGUF engine, no external vLLM required preview bash recipes/llama3-8b/cgn-infer/single-node/up.sh Release map: v0.3 → v0.9.1 | Version | Theme | Headline | |---|---|---| | 0.4.0 | Make the KV layer live | Prefix index populated after dispatch. cgn-kvcached eviction loop RAM watermark spill + SSD TTL . Real hit/miss/spill stats | | 0.5.0 | Native inference engine | cgn-infer : seventh binary, Candle + mmap GGUF, continuous batching, distributed layer-pipeline gRPC preview | | 0.6.0 | Feed the routing score | Engine Prometheus scraper feeds queue/cache into the score. Closed autoscaler cordon loop. Streaming cascade. TensorRT-LLM spawn driver | | 0.7.0 | OpenAI parity + live KV prefix | Tool calling, structured output, multimodal image passthrough. Post-generation KV prefixes written to etcd. SLA planner in operator. Turnkey Helm. Multi-node e2e in CI | | 0.8.0 | Etcd-optional | Gossip discovery state backend = "gossip" . Cross-cluster federation wired. Soft watt limit . ROCm. Capability-aware routing. Fleet dashboard | | 0.9.0 | Carbon + live KV prefix reconciliation | Carbon-aware admission for deferrable workloads. KV epoch + eviction-burst prefix reconciliation | | 0.9.1 | Admission + energy metrics | Per-model inflight caps on the gateway. Fleet tokens per watt gauges. Energy benchmark harness | | Unreleased | CognitoraConnector | kv offload = "cgn" spills vLLM blocks into cgn-kvcached via python/cgn-kv-connector . Router prefix digest passthrough to the connector | The sections below group by operator impact, not by semver. The routing score now uses real signals In May I described KV-aware routing as sequence-chained BLAKE3 digests plus longest-prefix overlap, with load, power, and capacity terms in the score. In v0.3 those secondary terms were mostly inert. The architecture supported them. The data path did not fill them. v0.4 turned on optimistic prefix recording after every successful dispatch. Prefill and decode nodes in a disaggregated pair both get indexed. v0.6 closed the loop on load and capacity. cgn-agent scrapes the engine Prometheus /metrics vLLM queue depth and GPU cache usage, SGLang equivalents and publishes numbers in the heartbeat and Agent.Health RPC. Nodes with unknown capacity report total blocks == 0 instead of inventing a value. v0.7 replaced pure optimism with live KV prefix tracking. After a generation completes, the agent writes lease-bound etcd keys under /cognitora/kv/<node /<digest . The router watcher mirrors PUT/DELETE into the prefix map. Entries expire with the node’s heartbeat lease. The agent prunes under KV pressure under 5% free blocks and past a 4096-key cap. The overlap score tracks what engines cached, not what the router hoped would land. v0.9.0 added KV epoch reconciliation. Agents publish a monotonic kv epoch in heartbeats bumped on engine restart, block-pool resize, or large free-block jumps . The router prunes stale prefix entries on epoch changes, eviction bursts, and pressure. Cognitora treats this as a workable alternative to engine block-hash KV events, which do not map cleanly onto the BLAKE3 digest chain. If you evaluate Cognitora on prefix-heavy traffic, this progression is the gap between a neat design and a router stopping you from hitting a node whose system prompt left HBM ten minutes ago. Etcd is optional. Federation forwards. The original post treated etcd as the cluster state backend and called cross-cluster federation operationally heavy. Both need a September update. Gossip discovery state backend = "gossip" , v0.8 wraps chitchat https://github.com/quickwit-oss/chitchat scuttlebutt over UDP. Agents republish the same JSON node record they would write to etcd. The router joins as a record-less member and reconciles live members every 2.5s. Multi-node clusters from a laptop to a rack no longer require standing up etcd. Live KV prefix tracking, cordon flags, routing policy hot-reload, and autoscaler hints stay etcd-only by design. See docs/architecture/gossip.md https://github.com/antonellof/cognitora-inference/blob/main/docs/architecture/gossip.md . Cross-cluster federation also v0.8 is wired in the gateway. When local routing finds no eligible node and router.federation is enabled, the router forwards to the lowest-connect-latency reachable peer. Forwards are counted in cgn router federation forwards total{model,peer} . Single-hop by construction. For a small fleet with zero external services, gossip is the headline. For multi-region without a Kubernetes-of-Kubernetes layer, federation matches the docs. Seven binaries, not six cgn-infer v0.5, preview is Cognitora’s first-party inference engine: mmap GGUF via Candle, OpenAI HTTP surface, BLAKE3 block-hash prefix cache aligned with cgn-kvcached , continuous batching, and a distributed layer-pipeline mode over a new activation-streaming gRPC service. The binary ships in the same release tarballs and Docker image as the original six. Recipe: recipes/llama3-8b/cgn-infer/single-node/ . Only kv offload = "none" is valid. Disaggregation and external connector offload do not apply to the native engine yet. I would not replace vLLM with cgn-infer for production LLM serving today. I would use the binary to see a fully self-contained Cognitora stack on a Mac or a CPU-only dev loop, and to watch how the layer-pipeline story evolves. MLX on Apple Silicon landed in the v0.3.x line engine.kind = "mlx" , mlx-lm’s mlx lm.server . The README lists MLX alongside vLLM and llama.cpp in the engine matrix. Same constraint: kv offload = "none" . OpenAI surface: tools, images, streaming cascade v0.7 brought the gateway closer to what production clients send: - tools , tool choice , response format pass through to vLLM/SGLang via a new extensions json proto field. - Streaming delta.tool calls fragments aggregate into complete tool calls on buffered responses. - Multimodal content arrays type: image url , … round-trip to the engine. Prefix hashing uses text parts only so images do not pollute KV-overlap scoring. - Cascade is bypassed for tool requests tool formats are not portable across cascade models . v0.6 made the SLM→LLM cascade work on streaming traffic. Early cascade steps run buffered for confidence gating. If every cheap model escalates, the final model streams live token-by-token. Gateway dispatch retry/failover up to 3 attempts, strictly before the first token landed in the same release. KV strategy: one more knob cgn The original post listed engine.kv offload as none | nixl | lmcache | hicache | kvbm . The recipe matrix now walks every backend: | Recipe | Backend | GPUs | |---|---|---| | llama3-8b/vllm/agg-lmcache | LMCache | 1 | | llama3-8b/vllm/agg-kvbm | NVIDIA KVBM Dynamo parity | 1 | | llama3-8b/sglang/agg-hicache | SGLang HiCache | 1 | | llama3-8b/vllm/agg-cgn | CognitoraConnector → cgn-kvcached | 1 | | llama3-8b/vllm/disagg-cgn | CognitoraConnector + NIXL multi-connector | 2 | The new kv offload = "cgn" path unreleased / preview installs python/cgn-kv-connector , which implements vLLM’s CognitoraConnector and spills blocks into the colocated cgn-kvcached daemon over gRPC PutBlock . The router forwards sequence-chained prefix digests as kv transfer params.cgn prefix digests so the connector probes the cache. When kv offload = "cgn" , agents also BatchLookup local kvcached and pass cgn resident digests for scheduler-side hits. Layer-4 routing intelligence and layer-2 engine offload share the same cgn-kvcached index without forcing LMCache or KVBM as intermediaries. The path is preview. Full layer-wise GPU↔host transfer is still being hardened. Energy, carbon, and admission are enforced The May post called energy-aware scheduling a strong tiebreaker with Redfish/IPMI/DCGM telemetry. The v0.8 to v0.9 line turned several of those ideas into working knobs: - Soft power cap agent .watt limit : rides the heartbeat, mirrored to cgn cluster node watt limit . The selector prefers under-cap nodes. - Cluster energy gauges v0.9.1 : cgn cluster power watts total and cgn cluster tokens per watt , refreshed every 5s. - Energy benchmark harness scripts/bench/energy/ : samples router /metrics around bench client.py runs and writes J/token to summary.md . - Carbon-aware admission v0.9.0 : the router polls a grid-intensity provider static , Electricity Maps, or WattTime and rejects low-priority requests X-CGN-Priority: low while gCO₂/kWh exceeds threshold. Fails open until the first successful poll. - Router admission control v0.9.1 : per- model, role inflight caps from router.admission .max queue enforced on chat, embeddings, and gRPC Generate. Deadline admission rejects when estimated TTFT exceeds the request deadline. ROCm support v0.8 extends the power story to AMD. cgn-power reads rocm-smi . Agents publish GPU identity from NVML or ROCm. Engine spawn pins both CUDA VISIBLE DEVICES and HIP VISIBLE DEVICES . Observability without Grafana v0.8 shipped a standalone fleet dashboard dashboard/ : zero dependencies, polls any CORS-enabled /metrics endpoint, in-browser ring buffers and canvas charts for req/s, tokens/s, latency p50/p95, TTFT p95, queue depth, power vs cap, KV used %, J/token, plus a live node table with GPU identity. Deep-linkable via ?endpoint= and ?interval= . A stdlib-only mock fleet generator simulates a 16-node mixed GPU fleet for demos. The router mirrors its node registry into cgn cluster node gauges every 5s and observes cgn router chat ttft seconds on the first streamed token. /metrics answers CORS preflight so browser apps scrape any Cognitora listener directly. If you read the original post and cared about sub-500 µs routing decisions, you also need to see whether those decisions help on your traffic. The dashboard fills the gap without standing up Prometheus + Grafana on day one. Kubernetes and CI maturity Several items from the original honest-limits list moved: - Turnkey Helm v0.7 : engine sidecar block, mTLS off by default for first contact, hostNetwork opt-in, chart README. Still no published OCI chart. Install from a git checkout or use the CPU quickstart manifest. - Multi-node e2e in default CI v0.7 : tests/e2e/multi node kv.sh with a stub OpenAI engine, including a real 2-agent prefix-affinity assertion. - GPU disaggregation bench harness scripts/bench/disagg/ + workflow dispatch : reproducible prefill/decode split benchmarks. --mode disagg-cgn for the CognitoraConnector topology. - Soft perf gate since v0.3 : cargo bench on routing/prefix paths posts a sticky PR comment. Hard S3 baselines remain planned. The CPU quickstart manifest from v0.3 deploy/kubernetes/quickstart/cognitora-cpu.yaml remains the fastest path to a public URL without GPU quota. Heterogeneous fleets Capability-aware routing v0.8 : agents publish gpu name , gpu vendor , vram total mb . Per-model min vram mb and require gpu filter candidates. Nodes reporting no GPU identity are never filtered, so older agents keep working. The recipe catalog expanded beyond Llama 3: Qwen-3 7B, DeepSeek-V4-Flash B200 and GB200 variants for both vLLM and SGLang , and the full 8B KV-backend walkthrough. Llama-3.3 70B disagg-single-node 8× H100/H200, TP=4 prefill + TP=4 decode remains the datacenter-shaped target. See the original post for hardware context. What I would revise in the original comparison Reading the May Dynamo comparison today, a few rows need nuance: | May write-up | September reality | |---|---| | Cross-cluster federation “off by default, operationally heavy” | Wired in v0.8. Still off by default. No longer aspirational | | Energy telemetry “strong tiebreaker” | Soft caps, fleet J/token gauges, carbon admission, and energy bench harness shipped | | “Performance numbers are targets” | Still true for your traffic. Routing score terms, live KV prefix tracking, and engine telemetry are no longer placeholders | | “No multimodal/video” | Text+image passthrough for chat models shipped in v0.7. Video diffusion serving still out of scope | | “Gang scheduling is basic” | Still basic node selectors, capability constraints . NVL72-aware co-scheduling not landed | | “Helm chart maturity” | Turnkey in v0.7. OCI publish still on the roadmap | Dynamo remains ahead on multimodal/video pipelines and NVL72-shaped gang scheduling. Cognitora’s differentiation llama.cpp + MLX + OpenAI-compat + native cgn-infer , gossip discovery, cross-cluster federation, carbon admission, positionally correct digests with live KV prefix tracking is stronger than in May. The gap on “is the routing score real?” has largely closed. Honest limits September 2026 - Still pre-1.0. Pin CGN VERSION=v0.9.1 and read the changelog https://github.com/antonellof/cognitora-inference/blob/main/CHANGELOG.md . Expect Proto and TOML shifts in minor releases. - cgn-infer and CognitoraConnector are preview. Do not bet production on them until the changelog drops the preview tag. - No published OCI Helm chart. Local chart path or CPU quickstart manifest. - Gossip vs etcd is a real tradeoff. Gossip gives you discovery. Live KV prefix tracking and policy hot-reload still want etcd. - Cross-cluster federation and carbon admission are easy to misconfigure. Defaults stay conservative for a reason. - Benchmark numbers on your hardware still require running the harness. CI gates routing microbenchmarks. Disagg and energy numbers need GPU hosts. Try the new paths Install v0.9.1 curl -fsSL https://inference.cognitora.dev/install | CGN VERSION=v0.9.1 sh Gossip-backed two-node lab no etcd . See docs/architecture/gossip.md state backend = "gossip" in cognitora.toml Preview: vLLM blocks into cgn-kvcached pip install ./python/cgn-kv-connector from repo checkout bash recipes/llama3-8b/vllm/agg-cgn/up.sh Preview: native GGUF, no vLLM bash recipes/llama3-8b/cgn-infer/single-node/up.sh Fleet dashboard against a running router python3 dashboard/mock metrics.py & optional mock fleet open "dashboard/index.html?endpoint=http://127.0.0.1:8080/metrics" Energy benchmark needs a live cluster + bench client bash scripts/bench/energy/run.sh Links: