Show HN: Gall – Inkling 2 Mini on 24GB M2, frees RAM between messages A developer released Gall, a custom inference engine that runs the Q3-quantized Inkling 2 Mini model on a 24GB M2 MacBook Air, below Thinking Machines Lab's stated 32GB minimum RAM requirement for the model on Mac. Gall loads strictly one model at a time with queued requests, saves the KV cache and unloads the model after every response or a configurable delay, keeping RAM at 50MB between calls while keeping reload fast; a menu bar app shows status and changes its icon during prefilling and generating. The developer reports near-instant responses in new sessions with a roughly 2,800-token system prompt and tool list used through the hax agent tool, and positions Gall as a local agent setup for infrequent messages such as a Telegram bridge. Thinking Machines Lab gives 32GB as the minimum RAM requirement for Inkling 2 Mini on Mac, but with the Q3 quant I felt it must be possible to get somewhere reasonable on my 24GB M2 MacBook Air. Llama.cpp wasn't cutting it even with patches, so I spent this week's Fable allowance with a sprinkling of Astra on this custom inference engine, and took the opportunity to add some optimisations and ergonomics for running on a memory-constrained laptop: - strictly one model loaded at a time, requests queued; - KV cache gets saved and model unloaded after every response or configurable delay , so RAM stays at 50MB between calls, but reload is fast; - menu bar app shows status and icon changes when prefilling/generating. So far I've been using it with hax 1 with my normal system prompt + tool list, which runs to about 2,800 tokens, and getting near-instant responses in new sessions due to the cache-and-reload. Works great as a local agent setup for infrequent messages too, e.g. via a Telegram bridge. I'm sure this vibecoded engine and the model itself will be superseded in time, but for now it's by far the best local model setup I've ever managed on this machine, so wanted to share for others in the same boat. Please do hack away at it. 1. https://github.com/OleksandrChekhovskyi/hax https://github.com/OleksandrChekhovskyi/hax Comments URL: https://news.ycombinator.com/item?id=50604195 https://news.ycombinator.com/item?id=50604195 Points: 2 Comments: 0