cd /news/large-language-models/show-hn-gall-inkling-2-mini-on-24gb-… · home › topics › large-language-models › article
[ARTICLE · art-174352] src=github.com ↗ pub= topic=large-language-models verified=true sentiment=↑ positive

Show HN: Gall – Inkling 2 Mini on 24GB M2, frees RAM between messages

A developer released Gall, a custom inference engine that runs the Q3-quantized Inkling 2 Mini model on a 24GB M2 MacBook Air, below Thinking Machines Lab's stated 32GB minimum RAM requirement for the model on Mac. Gall loads strictly one model at a time with queued requests, saves the KV cache and unloads the model after every response or a configurable delay, keeping RAM at 50MB between calls while keeping reload fast; a menu bar app shows status and changes its icon during prefilling and generating. The developer reports near-instant responses in new sessions with a roughly 2,800-token system prompt and tool list used through the hax agent tool, and positions Gall as a local agent setup for infrequent messages such as a Telegram bridge.

key takeaway

Saving the KV cache and unloading the model after each response keeps RAM at 50MB between calls on a 24GB M2 while reload stays fast enough for near-instant responses with a ~2,800-token prompt.

read1 min views3 publishedNov 22, 2026

Thinking Machines Lab gives 32GB as the minimum RAM requirement for Inkling 2 Mini on Mac, but with the Q3 quant I felt it must be possible to get somewhere reasonable on my 24GB M2 MacBook Air.

Llama.cpp wasn't cutting it even with patches, so I spent this week's Fable allowance (with a sprinkling of Astra) on this custom inference engine, and took the opportunity to add some optimisations and ergonomics for running on a memory-constrained laptop:

  • strictly one model loaded at a time, requests queued; - KV cache gets saved and model unloaded after every response (or configurable delay), so RAM stays at 50MB between calls, but reload is fast; - menu bar app shows status and icon changes when prefilling/generating.

So far I've been using it with hax [1] with my normal system prompt + tool list, which runs to about 2,800 tokens, and getting near-instant responses in new sessions due to the cache-and-reload. Works great as a local agent setup for infrequent messages too, e.g. via a Telegram bridge.

I'm sure this vibecoded engine and the model itself will be superseded in time, but for now it's by far the best local model setup I've ever managed on this machine, so wanted to share for others in the same boat. Please do hack away at it.

1. [https://github.com/OleksandrChekhovskyi/hax](https://github.com/OleksandrChekhovskyi/hax)

Comments URL: [https://news.ycombinator.com/item?id=50604195](https://news.ycombinator.com/item?id=50604195)

Points: 2

── more in #large-language-models 4 stories · sorted by recency
── more on @gall 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/show-hn-gall-inkling…] indexed:0 read:1min 2026-11-22 · —