{"slug": "show-hn-reflex-a-gguf-cuda-inference-engine-tuned-for-cold-start-latency", "title": "Show HN: Reflex – a GGUF/CUDA inference engine tuned for cold-start latency", "summary": "Lateos AI released Reflex, an open-source GGUF-native Rust and CUDA inference engine that compiles every CUDA kernel ahead-of-time with nvcc at build time to eliminate the multi-second JIT tax on first use. On an RTX A6000, Reflex's system1 subcommand scored three candidates for the prompt \"The capital of France is\" in 8524.253 ms from process start to result, assigning \" Paris\" a probability of 0.997350 versus 0.002239 for \" London\" and 0.000411 for \" Berlin\". The project targets cold-start workloads such as serverless/FaaS, single-shot CLI calls, batch jobs and edge devices, and its system1 path currently supports dense and MoE Qwen3 only, while generate supports Qwen3, the Qwen3.5 hybrid mixer and DeepSeek-V2/V3 MLA.", "body_md": "A high-performance, GGUF-native Rust & CUDA inference engine optimized for cold-start latency and real-time \"System 1\" agent decision loops — process launch to first token, not sustained server throughput.\n\nEvery CUDA kernel is compiled ahead-of-time by `nvcc` at *build* time and shipped\ninside the binary — never compiled at runtime via NVRTC — so there's no multi-second\nJIT tax on first use, the way there is with a runtime-compilation design. That's the\nwhole bet: be the fastest way to turn a cold process into one output token, then get\nout of the way.\n\nRequires the CUDA toolkit (`nvcc` on `PATH`, or `CUDA_PATH`/` CUDA_HOME` set) and an\nNVIDIA GPU. `REFLEX_SKIP_CUDA=1 cargo build` skips kernel compilation for\nediting/type-checking on a machine without CUDA (no subcommand will actually run\nkernels in that mode).\n\n```\ngit clone https://github.com/lateos-ai/reflex.git\ncd reflex\ncargo build --release\n\n# Prove the AOT pipeline works end to end on your GPU:\ncargo run --release --bin reflex -- smoke\n\n# Run a real forward pass against a GGUF file:\ncargo run --release --bin reflex -- generate <path-to-gguf> \"Once upon a time\"\n```\n\n`reflex` is a single binary with subcommands: `generate` (load a GGUF and generate\ntokens), `system1` (single-pass, non-autoregressive candidate scoring — the \"System 1\"\ndecision-loop path), `smoke` (the AOT-pipeline check above), `bench` (warm-latency\nmicrobenchmark), `check` (byte-exact-vs-reference correctness check, CI-scriptable),\nand `stdio`/` uds` (local JSON-line IPC, both need `--features ipc`). Run `reflex <subcommand>` with no further arguments to see that subcommand's own usage.\n\n`system1` takes a local GGUF path, so download the file first with the\n[`hf` CLI](https://huggingface.co/docs/huggingface_hub/guides/cli) (`pip install -U huggingface_hub`), then point `system1` at it — this scores each `--candidate`\nagainst the prompt in a single pass, with no autoregressive decode loop:\n\n```\nhf download Qwen/Qwen3-0.6B-GGUF Qwen3-0.6B-Q8_0.gguf --local-dir .\n\ncargo run --release --bin reflex -- system1 Qwen3-0.6B-Q8_0.gguf \\\n  \"The capital of France is\" \\\n  --candidate \" Paris\" --candidate \" London\" --candidate \" Berlin\"\n```\n\nReal output from this exact command (RTX A6000):\n\n```\nREFLEX_SYSTEM1_CANDIDATE_OK idx=0 text=\" Paris\" token_ids=[12095] score=17.407064 probability=0.997350\nREFLEX_SYSTEM1_CANDIDATE_OK idx=1 text=\" London\" token_ids=[7148] score=11.308186 probability=0.002239\nREFLEX_SYSTEM1_CANDIDATE_OK idx=2 text=\" Berlin\" token_ids=[19846] score=9.612655 probability=0.000411\nREFLEX_SYSTEM1_OK process_start_to_result_ms=8524.253 num_candidates=3 best_idx=0 best_text=\" Paris\" entropy=0.028154\n```\n\n`probability` is relative to this candidate set only, not a vocab-wide probability —\nsee `Model::system1_evaluate`'s doc comment in `src/model.rs`. `system1` currently\nsupports dense/MoE Qwen3 only; the Qwen3.5 hybrid mixer and DeepSeek-V2/V3 (MLA) are\nrejected with a clear error (`generate` supports all four architectures — swap in a\nDeepSeek GGUF the same way for a `generate` run instead).\n\n`generate` can also pull a GGUF straight from the Hub itself, via this project's own\nRust `hf-hub` integration — `--model <org/repo:file.gguf>` or `--quickstart`, both\nrequiring `cargo build --features download`:\n\n```\ncargo run --release --features download --bin reflex -- generate --quickstart \"Once upon a time\"\n```\n\nClosing the steady-state-throughput gap with llama.cpp/vLLM is a kernel-optimization\nrace against projects with a multi-year head start — Rust as a language doesn't change\nwho wins it. What none of llama.cpp, vLLM, or `candle` are built for or measured\nagainst is a **cold** invocation — serverless/FaaS, single-shot CLI/dev-tool calls,\nbatch/cron jobs, edge devices that wake on demand. A naive runtime-JIT design pays a\nreal, measured multi-second tax on first kernel use; vLLM took ~235s to become ready\n(CUDA graph capture) before serving one request; llama.cpp avoids both because its\nkernels are compiled by `nvcc` at *build* time, not at process start.\n\n**Target metric**: energy-to-first-token from cold start (joules, process launch to\nfirst generated token) — a real, underexplored gap. Existing energy benchmarks measure\nwarm/steady-state joules-per-token, not full-lifecycle cold-start cost.\n\n**Target models**: Qwen and DeepSeek families.\n\nAll comparisons are cold-start (process launch to first token/result), same\nThunderCompute A6000, `n=3`, external wall-clock (`/usr/bin/time -v` — process launch\nto exit, not just Reflex's own internal timer). Full methodology, disclosed caveats,\nand per-run numbers for every comparison below are in `DECISIONS.md` and `HISTORY.md`.\n\n| vs. | Result | Caveat | \n|---|---|---|\n| **llama.cpp** | **~1.3–1.4x faster** (4.71–5.05s vs. 6.45–6.56s) | Both AOT-compiled — doesn't exercise the JIT-tax claim below | \n| **vLLM** | **~24–52x faster** (4.71–5.05s vs. 121–244s, depending on`torch.compile` cache state) | Installed vLLM has no GGUF support; ran against an HF safetensors checkpoint instead, disclosed | \n| **Ollama** | Directly competitive when it doesn't stall (~6–7s), but its bundled `llama-server` intermittently hits an internal GPU-discovery-watchdog timeout (~55–62s) | Wraps llama.cpp's own runtime — tests packaging/daemon overhead, not the AOT-vs-JIT bet | \n| **TypeSafe Jev** , cold-start-to-decision | Reflex loses, **~10–60x slower** | Different deployment model: Jev is an always-warm managed API; this measures a genuine cold local process launch | \n| **TypeSafe Jev** , warm/compute-only | **Competitive, within ~1.3–2x** (19.4ms vs. Jev's cited 10–15ms) | Jev's figures are self-reported/published, not independently reproduced here | \n\nThe llama.cpp/Ollama/Jev \"loses\" results above are reported as-is, not smoothed over —\nsee `DECISIONS.md`'s benchmark-methodology entries for why each comparison is framed\nthe way it is.\n\nEvery CUDA kernel is compiled **ahead of time** (`build.rs` invokes `nvcc`, see\n`build.rs` and `src/kernels_cuda/`), never at runtime via NVRTC. `src/aot.rs` loads the\nprecompiled PTX/cubin at process start via the CUDA driver API. Default mode emits\nportable PTX (small driver-side JIT-to-SASS cost at load); set `REFLEX_CUDA_ARCH=sm_XX`\nto compile straight to a `cubin` for one target architecture (true zero-JIT, at the cost\nof needing a matching cubin per deployment target). Which one actually wins on real\nhardware is unverified — that's the first thing to measure, not assume.\n\nRun `cargo run --bin reflex -- smoke` on a real GPU instance as the very first\nreal-hardware step: it proves the AOT pipeline works end to end and reports actual\nprocess-start-to-first-result wall clock on the simplest possible kernel, before any\nmodel-architecture work begins.\n\n**Linux build prerequisite for `--features download`/` ipc`/` python`** (`--all-features`\nincluded): these pull in `hf-hub`, whose `ureq` HTTP client needs `libssl-dev` +\n`pkg-config` on the build host, or `cargo build` fails with `openssl-sys` unable to find\nan OpenSSL installation. Not needed for the default feature-less build. On Ubuntu/Debian:\n\n```\nsudo apt-get install -y libssl-dev pkg-config\n```\n\n(Discovered on a fresh ThunderCompute instance during the MVP-release adoption round —\nnot needed on the Windows dev machine that round otherwise developed on, since\n`native-tls` uses a different TLS backend there.)\n\nAll four steps below are **done** and real-hardware-verified (see `STATUS.md` for\ncurrent state, `HISTORY.md` for the full verification write-up of each):\n\n1. **Dense Qwen3** — the best-understood, most well-documented architecture to build\nagainst first; proves the AOT-compilation + cold-start-benchmark harness works at all.\n2. **Qwen3-MoE**\n3. **Qwen3.5 hybrid Gated DeltaNet mixer**\n4. **DeepSeek-V2/V3 MLA** — deliberately last; a genuinely different (compressed\nlatent-KV) caching strategy, not an incremental GQA extension.\n\nThese are permanent constraints on this engine, not just current-MVP scope — the whole\nreason Reflex exists is to win a narrower bet (cold-start energy/latency) than\nsustained-server throughput. A broad serving feature set re-inherits the exact\nthroughput/serving race that's unwinnable against llama.cpp/vLLM/SGLang's head start.\nMulti-tenancy and persistent state belong in the *host\norchestrator*, not in this engine:\n\n- **`batch_size` is always 1.** No request queue, no continuous batching, no\nPagedAttention-style dynamic allocation, no context preemption. Horizontal scaling\n(many concurrent jobs) is the orchestrator's job — spin up N`Reflex` processes across GPU slices/time-slices — not this engine's, ever.\n- **No internal multi-tenant LoRA router/scheduler.**\n- **No internal NVMe/S3 KV-cache manager or cache-hit logic.**\n- **No concurrent HTTP/gRPC server** , no request auth/rate-limiting, no autoscaling\ndecision-making. If a warm-context mode ever exists (see Phase 4 below), it accepts\none job at a time, strictly sequentially — never a thread pool.\n\n**No in-core HTTP/gRPC server, ever, not deferred.** This is not a separate exception\nto the rule above — a concurrent HTTP listener is the exact same violation\n(\"`batch_size` always 1... never a thread pool\") under a different name, and an\nadoption/UX ask asking for one doesn't get to reopen it. If HTTP access to this engine\nis ever genuinely needed, the pattern is a **separate, optional sidecar binary** (e.g.\n`system1-openai-adapter`) that talks to this core engine over local IPC only — the core\nengine itself never grows a network socket. Building that sidecar is out of scope for\nnow; this paragraph only records the escape-hatch pattern so a future HTTP ask gets\nrouted there instead of back into this engine.\n\nFor local, non-network ergonomics, this engine may instead expose: a **stdio JSON-line\nmode** (`reflex stdio`, one JSON request per stdin line, fully processed before the next\nline is read) and a **Unix Domain Socket mode** (`reflex uds <path>`, Unix-only, one\nconnection fully processed before the next is accepted) — both strictly sequential,\nnever a thread pool, mirroring the same request/response protocol. A shared-memory\nring-buffer transport was considered and deliberately deferred — crash-safety and\nsynchronization design is disproportionate complexity for the ergonomics it would buy\n— recorded here as a future-work idea only, not designed.\n\nOnce the model-architecture MVP proves the engine handles the target model\nfamilies at all, the next axis is making the *cold-start path itself* faster and\nadoptable — without ever crossing into building a serving platform. The framing: let\nvLLM win the warm-throughput race; Reflex wins by being the fastest way to turn\ncold compute into one output token, then getting out of the way. All four phases below\nare **done** — see `HISTORY.md` for the full per-round write-up of each:\n\n- **Phase 1** — Single-shot CLI: process launch -> one forward pass -> exit.\n- **Phase 2 — Fast IO** : weights upload to the GPU once (not re-uploaded per kernel\ncall), device-resident activations through a whole layer, and on-GPU dequant kernels\nfor the block types this project's fixtures use for the bulk of weight bytes. This is\nthe work behind the llama.cpp benchmark result above.\n- **Phase 3 — State I/O** :`--export-kv <file>` /`--import-kv <file>` for raw K/V-cache\ndump/load/resume, across all four architectures. Reflex stays ignorant of*where* that file lives (NVMe, an S3-backed FUSE mount, tmpfs) — that's the orchestrator's\njob, not this engine's.\n- **Phase 4 — Embeddability** :`--lora <path>` (load-time adapter application, no\nruntime hot-swap multiplexer) and a Rust C-FFI surface (`src/ffi.rs` ,`include/reflex_engine.h` ) so an external orchestrator can embed Reflex directly\ninstead of`exec` -ing a binary.\n\n- `src/gguf.rs` — GGUF metadata/tensor-directory parsing (mmap-based).\n- `src/dequant.rs` ,`src/dequant_iq.rs` ,`src/dequant_iq_tables.rs` — standard and\ni-quant dequantization, verified byte-exact against`gguf-py` .\n- `src/tokenizer.rs` — verified against real sentencepiece/BPE references.\n\nNone of these care how kernels get compiled — they're pure host-side GGUF/tokenizer logic. There is deliberately no NVRTC runtime-compile-and-load path anywhere in this codebase — that's the thing this project's AOT design replaces, not reuses.\n\nA multi-stage `Dockerfile` is included: the builder stage has the full CUDA devel\ntoolkit (`nvcc`) to compile the AOT kernels; the runtime stage only needs the CUDA\n*runtime* libraries, since every kernel byte is embedded directly into the compiled\nbinary at build time — the runtime image never runs `nvcc` and never needs the devel\ntoolkit.\n\n```\n# Defaults to sm_86 (RTX A6000/3090-class). Pass --build-arg REFLEX_CUDA_ARCH=sm_XX\n# for a different target compute capability, or --build-arg REFLEX_CUDA_ARCH= (empty)\n# for a portable PTX build that JITs to whatever GPU the container actually runs on.\ndocker build --build-arg REFLEX_CUDA_ARCH=sm_86 -t reflex .\n\n# Needs nvidia-container-toolkit on the host. The default entrypoint is\n# `reflex generate`, so pass just the GGUF path and prompt:\ndocker run --rm --gpus all -v /path/to/models:/models \\\n  reflex /models/Qwen3-0.6B-Q4_K_M.gguf \"Once upon a time\"\n```\n\nFor any other subcommand (`system1`/` smoke`/` bench`/` check`/` stdio`/` uds`), override\nthe entrypoint:\n\n```\ndocker run --rm --gpus all -v /path/to/models:/models \\\n  --entrypoint /usr/local/bin/reflex reflex \\\n  system1 /models/Qwen3-0.6B-Q4_K_M.gguf \"Q: ...? A:\" --candidate \" Yes\" --candidate \" No\"\n```\n\nThe CUDA major/minor version in both Docker stages must stay consistent with\n`Cargo.toml`'s pinned `cudarc` feature (`\"cuda-12000\"`, i.e. CUDA 12.x) — a mismatch is\na build-time/runtime library version mismatch this Dockerfile can't catch for you.\n\nReflex is a single-shot CLI, not a server (see Non-goals above) — the natural\nKubernetes primitive is a **Job**, one cold-start invocation per Pod, never a\n`Deployment`/` Service`. A minimal example running `generate` against a GGUF baked into\na volume, requesting one GPU via the standard NVIDIA device plugin:\n\n```\napiVersion: batch/v1\nkind: Job\nmetadata:\n  name: reflex-generate\nspec:\n  backoffLimit: 0\n  template:\n    spec:\n      restartPolicy: Never\n      containers:\n        - name: reflex\n          image: reflex:latest\n          args: [\"/models/Qwen3-0.6B-Q4_K_M.gguf\", \"Once upon a time\"]\n          resources:\n            limits:\n              nvidia.com/gpu: 1\n          volumeMounts:\n            - name: models\n              mountPath: /models\n              readOnly: true\n      volumes:\n        - name: models\n          persistentVolumeClaim:\n            claimName: reflex-models\n```\n\nFor `system1`/` bench`/` check`/other subcommands, set `command: [\"/usr/local/bin/reflex\"]`\nand put the subcommand as the first entry in `args`, same as the Docker override above.\nThis is exactly the \"orchestrator's job\" this engine intentionally stays out of —\nReflex itself never grows a scheduler, a request queue, or a `batch_size > 1`; Kubernetes\n(or cron, or a FaaS platform) is where that concurrency/scheduling belongs.", "url": "https://wpnews.pro/news/show-hn-reflex-a-gguf-cuda-inference-engine-tuned-for-cold-start-latency", "canonical_source": "https://github.com/lateos-ai/reflex", "published_at": "2026-09-23 17:51:45+00:00", "updated_at": "2026-09-23 18:00:51.094198+00:00", "lang": "en", "topics": ["ai-infrastructure", "large-language-models", "ai-tools", "mlops", "ai-agents"], "entities": ["Reflex", "Lateos AI", "CUDA", "NVIDIA", "RTX A6000", "Qwen3", "DeepSeek-V2", "llama.cpp"], "alternates": {"html": "https://wpnews.pro/news/show-hn-reflex-a-gguf-cuda-inference-engine-tuned-for-cold-start-latency", "markdown": "https://wpnews.pro/news/show-hn-reflex-a-gguf-cuda-inference-engine-tuned-for-cold-start-latency.md", "text": "https://wpnews.pro/news/show-hn-reflex-a-gguf-cuda-inference-engine-tuned-for-cold-start-latency.txt", "jsonld": "https://wpnews.pro/news/show-hn-reflex-a-gguf-cuda-inference-engine-tuned-for-cold-start-latency.jsonld"}}