{"slug": "show-hn-low-latency-local-llm-runner-via-openjdk-panama-ffm-java-22", "title": "Show HN: Low-latency local LLM runner via OpenJDK Panama FFM (Java 22)", "summary": "David released libargus, a low-latency local LLM runner that uses OpenJDK Panama FFM to interface directly with llama.cpp and whisper.cpp from the JVM, achieving zero-allocation on hot paths. The project bundles pre-compiled native binaries and aims to replace RAGs with a spatio-temporal memory layer called L-TABB.", "body_md": "I wanted to run AI from inside the JVM. I started out with the standard REST sidecar, ripped that out to use Project Panama (Foreign Function & Memory API) in the new JDK versions to interface directly with llama.cpp. I still wasn't happy with how that functioned, so I built libargus.cc to get a clean ABI to expose a structured API up in the JVM landscape. It still uses Project Panama to interface directly with llama.cpp, whisper.cpp, and ggml compute graphs.\n\nI have zero-allocation on the hot paths, memory segments for prompts and tokens are allocated once inside confined Arenas. Raw pointers pass straight through down to the low C level. This avoids primitive array cloning and heap churn.\n\nI mapped out the native structures from llama.cpp and whisper.cpp while matching the compiler's padding to maintain safe memory access.\n\nI bundle pre-compiled native binaries in the jar for easy deployment.\n\nThis execution engine provides the foundation I need for work I'm doing on a spatio-temporal memory layer (L-TABB) to replace RAGs. I'd love to get technical feedback to polish any issues while I continue working on the next layer. Deep-dives from anyone hacking on Project Panama or low-latency systems in modern JDK would be very appreciated!\n\nI'm much better with code than prose, so I'll let the code do most of the talking.\n\nHappy Hacking! /David\n\nCode: [https://libargus.cc](https://libargus.cc)\nProject Landing Page: [https://projectargus.cc](https://projectargus.cc)\n\nComments URL: [https://news.ycombinator.com/item?id=48907681](https://news.ycombinator.com/item?id=48907681)\n\nPoints: 2\n\n# Comments: 0", "url": "https://wpnews.pro/news/show-hn-low-latency-local-llm-runner-via-openjdk-panama-ffm-java-22", "canonical_source": "https://github.com/projectargus-cc/libargus.cc", "published_at": "2026-07-14 14:40:46+00:00", "updated_at": "2026-07-14 14:47:53.254574+00:00", "lang": "en", "topics": ["large-language-models", "ai-infrastructure", "developer-tools"], "entities": ["OpenJDK", "llama.cpp", "whisper.cpp", "Project Panama", "libargus", "David"], "alternates": {"html": "https://wpnews.pro/news/show-hn-low-latency-local-llm-runner-via-openjdk-panama-ffm-java-22", "markdown": "https://wpnews.pro/news/show-hn-low-latency-local-llm-runner-via-openjdk-panama-ffm-java-22.md", "text": "https://wpnews.pro/news/show-hn-low-latency-local-llm-runner-via-openjdk-panama-ffm-java-22.txt", "jsonld": "https://wpnews.pro/news/show-hn-low-latency-local-llm-runner-via-openjdk-panama-ffm-java-22.jsonld"}}