{"slug": "show-hn-gall-inkling-2-mini-on-24gb-m2-frees-ram-between-messages", "title": "Show HN: Gall – Inkling 2 Mini on 24GB M2, frees RAM between messages", "summary": "A developer released Gall, a custom inference engine that runs the Q3-quantized Inkling 2 Mini model on a 24GB M2 MacBook Air, below Thinking Machines Lab's stated 32GB minimum RAM requirement for the model on Mac. Gall loads strictly one model at a time with queued requests, saves the KV cache and unloads the model after every response or a configurable delay, keeping RAM at 50MB between calls while keeping reload fast; a menu bar app shows status and changes its icon during prefilling and generating. The developer reports near-instant responses in new sessions with a roughly 2,800-token system prompt and tool list used through the hax agent tool, and positions Gall as a local agent setup for infrequent messages such as a Telegram bridge.", "body_md": "Thinking Machines Lab gives 32GB as the minimum RAM requirement for Inkling 2 Mini on Mac, but with the Q3 quant I felt it must be possible to get somewhere reasonable on my 24GB M2 MacBook Air.\n\nLlama.cpp wasn't cutting it even with patches, so I spent this week's Fable allowance (with a sprinkling of Astra) on this custom inference engine, and took the opportunity to add some optimisations and ergonomics for running on a memory-constrained laptop:\n\n- strictly one model loaded at a time, requests queued; - KV cache gets saved and model unloaded after every response (or configurable delay), so RAM stays at 50MB between calls, but reload is fast; - menu bar app shows status and icon changes when prefilling/generating.\n\nSo far I've been using it with hax [1] with my normal system prompt + tool list, which runs to about 2,800 tokens, and getting near-instant responses in new sessions due to the cache-and-reload. Works great as a local agent setup for infrequent messages too, e.g. via a Telegram bridge.\n\nI'm sure this vibecoded engine and the model itself will be superseded in time, but for now it's by far the best local model setup I've ever managed on this machine, so wanted to share for others in the same boat. Please do hack away at it.\n\n1. [https://github.com/OleksandrChekhovskyi/hax](https://github.com/OleksandrChekhovskyi/hax)\n\nComments URL: [https://news.ycombinator.com/item?id=50604195](https://news.ycombinator.com/item?id=50604195)\n\nPoints: 2\n\n# Comments: 0", "url": "https://wpnews.pro/news/show-hn-gall-inkling-2-mini-on-24gb-m2-frees-ram-between-messages", "canonical_source": "https://github.com/ltrgoddard/gall", "published_at": "2026-11-22 21:40:38+00:00", "updated_at": "2026-11-22 22:14:40.470181+00:00", "lang": "en", "topics": ["large-language-models", "ai-tools", "ai-agents", "ai-infrastructure"], "entities": ["Gall", "Inkling 2 Mini", "Thinking Machines Lab", "Apple M2 MacBook Air", "llama.cpp", "hax", "OleksandrChekhovskyi"], "also_reported_by": [], "takeaway": "Saving the KV cache and unloading the model after each response keeps RAM at 50MB between calls on a 24GB M2 while reload stays fast enough for near-instant responses with a ~2,800-token prompt.", "recommendation": null, "alternates": {"html": "https://wpnews.pro/news/show-hn-gall-inkling-2-mini-on-24gb-m2-frees-ram-between-messages", "markdown": "https://wpnews.pro/news/show-hn-gall-inkling-2-mini-on-24gb-m2-frees-ram-between-messages.md", "text": "https://wpnews.pro/news/show-hn-gall-inkling-2-mini-on-24gb-m2-frees-ram-between-messages.txt", "jsonld": "https://wpnews.pro/news/show-hn-gall-inkling-2-mini-on-24gb-m2-frees-ram-between-messages.jsonld"}}