Show HN: Low-latency local LLM runner via OpenJDK Panama FFM (Java 22)

David released libargus, a low-latency local LLM runner that uses OpenJDK Panama FFM to interface directly with llama.cpp and whisper.cpp from the JVM, achieving zero-allocation on hot paths. The project bundles pre-compiled native binaries and aims to replace RAGs with a spatio-temporal memory layer called L-TABB.

I wanted to run AI from inside the JVM. I started out with the standard REST sidecar, ripped that out to use Project Panama Foreign Function & Memory API in the new JDK versions to interface directly with llama.cpp. I still wasn't happy with how that functioned, so I built libargus.cc to get a clean ABI to expose a structured API up in the JVM landscape. It still uses Project Panama to interface directly with llama.cpp, whisper.cpp, and ggml compute graphs. I have zero-allocation on the hot paths, memory segments for prompts and tokens are allocated once inside confined Arenas. Raw pointers pass straight through down to the low C level. This avoids primitive array cloning and heap churn. I mapped out the native structures from llama.cpp and whisper.cpp while matching the compiler's padding to maintain safe memory access. I bundle pre-compiled native binaries in the jar for easy deployment. This execution engine provides the foundation I need for work I'm doing on a spatio-temporal memory layer L-TABB to replace RAGs. I'd love to get technical feedback to polish any issues while I continue working on the next layer. Deep-dives from anyone hacking on Project Panama or low-latency systems in modern JDK would be very appreciated I'm much better with code than prose, so I'll let the code do most of the talking. Happy Hacking /David Code: https://libargus.cc https://libargus.cc Project Landing Page: https://projectargus.cc https://projectargus.cc Comments URL: https://news.ycombinator.com/item?id=48907681 https://news.ycombinator.com/item?id=48907681 Points: 2 Comments: 0