Setup for local ai models on apple silicon A community benchmark for oMLX 0.6.3rc2 on an Apple M5 Pro with 48 GB of memory reports peak memory of 23.2 GB at 32K context, 28.5 GB at 64K and 39.5 GB at 128K when running Qwen3.8-27B with 4-bit weights and 8-bit KV cache, though its published recipe lists max_context_window=71680 despite the 128K benchmark input. The write-up recommends starting with a maintained 4-bit weight build and a verified 32K–64K served context on Ollama before comparing oMLX with DFlash2 on the same coding task, and warns that weight quantization and KV-cache precision are separate settings. The author states the setup was not tested on a Mac and the earlier memory spike was not reproduced, so the cause remains open. Hmm… I’m not very familiar with Macs, so I did some digging: For Qwen3.8-27B with Pi Harness on an M5 Pro/48 GB , my first choice would be reliable 4-bit weights and a verified 32K–64K working context , using your existing Ollama setup as the reference. Then I would compare oMLX + DFlash2 on the same coding task. If 125K/128K is genuinely useful, keep it as a second target rather than abandoning it: the question is whether it fits with usable latency, reasonable memory pressure, and correct tool calls. - Quant: Start with a maintained 4-bit weight build. Do not confuse weight quantization with KV-cache precision : 4-bit weights can still use full-precision KV. An 8-bit KV setting may save context memory if the specific backend actually supports it; GGUF and MLX quantization labels are not automatically equivalent. - Context: Begin with 32K/64K actually served , then try 96K/125K/128K if useful. Model-card context, configured maximum, an accepted API request and effective retained agent history are different things. - Server: Ollama first least disruption , oMLX + DFlash2 next strong decoding candidate , with llama.cpp/GGUF and LM Studio/Splash as alternative packages. Compare full Pi tasks, not just decode tok/s. A close community oMLX result for M5 Pro/48 GB https://omlx.ai/benchmarks/performance/vd6i0f5f reports peaks of 23.2 GB at 32K , 28.5 GB at 64K and 39.5 GB at 128K , with 4-bit weights and 8-bit KV. It used oMLX 0.6.3rc2 , and its published recipe lists max context window=71680 despite the 128K benchmark input. This shows experimental feasibility in a related stack, not that a normal Pi API request can safely use 128K on your Ollama installation. I have not tested this on a Mac or reproduced your memory spike . The observations below come from published documentation, source code, issue discussions and other people’s measurements, so I would keep the exact cause of your earlier spike open. 1. With the model loaded, run ollama --version and ollama ps ; note the model tag, PROCESSOR , and effective CONTEXT Ollama context guide https://docs.ollama.com/context-length . Don’t infer these from an assumed default. 2. Was 125K the actual prompt/history , or only the configured ceiling? Compare Pi’s declared model window with the server’s effective window. 3. Did memory rise on the first long prefill , after repeated Pi/tool turns , or only with overlapping requests ? These point to different mechanisms. On macOS, watch Memory Pressure and Swap Used , not just a peak process number Apple’s Activity Monitor guide https://support.apple.com/en-mide/guide/activity-monitor/actmntr1004/mac . A brief allocation, a retained cache that plateaus and sustained system-wide swapping are different outcomes. My decision map would be roughly: Was ~125K the configured ceiling, rather than the actual prompt size? Yes: align Pi's model context declaration with the server's real limit. Check accepted request size before diagnosing a 125K prefill. No: did memory jump on the first long request? Yes: investigate active KV, prefill peak, and headroom/swap. No: did it grow after many Pi turns or simultaneous requests? Yes: investigate retained prefixes, agent history, compaction, cache lifecycle, and concurrency. No: check exact version, model build, and reproducibility. After that, compare one alternative server with the same actual task. 1. Choosing weight quantization, KV precision, and useful context The word “quant” gets used for more than one object. Weight quantization changes the representation of the model’s parameters: a 4-bit model generally costs much less to keep loaded than unquantized bf16 weights. KV-cache quantization changes the storage format of attention keys and values created while processing prompts and generating text. A 4-bit weight build can still have a bf16 KV cache; equally, a backend may support 8-bit or 4-bit KV on top of some weight format. A draft model used for speculative decoding can be a third memory consumer. For 48 GB unified memory , a maintained 4-bit conversion is the sensible first trial. Quantization quality is particularly important for a coding agent: tool-call schemas, correct file edits, and recovery after mistakes count more than plausible text alone. More aggressive 2/3-bit weights may be worth exploring later, but saving memory at the expense of reliable tool execution can lose time overall. I would not assume that “Q4” in a GGUF name, “4bit” on an MLX repository, and a particular Ollama tag mean mathematically identical quantizers. They can differ in group size, outlier handling, tensor exemptions, conversion version and the underlying runner. Two builds labeled 4-bit can have different disk sizes, runtime allocations and quality. For a fair comparison, keep the exact model artifact or at least record the precise tag/revision. If a comparison switches model format and inference engine at the same time, the result belongs to the combination , not to one magic property of MLX or GGUF. Ollama’s upstream configuration reference https://github.com/ollama/ollama/blob/main/docs/faq.mdx documents OLLAMA KV CACHE TYPE f16 , q8 0 , q4 0 with Flash Attention. However, the availability and effect of those generic settings depend on the version and actual runner; historical MLX paths did not necessarily honor every documented cache knob. I would not prescribe q8 0 as an assured fix without checking that your Qwen backend uses it. oMLX https://github.com/jundot/omlx provides its own cache formats, including affine/TurboQuant choices on supported paths. Do not translate an Ollama q8 0 flag directly into an oMLX setting. Test memory and quality under the specific release, since low-precision cache, kernels and cache reuse can interact. For an agent, the practical quality checks are retrieval of a fact near the beginning of a long history, consistent tool arguments, and a successful multi-step edit. Smaller cache formats can save memory without preserving every long-context task equally well. 32K/64K are starting points, not architectural ceilings. At 32K there is more headroom for prefill, tools and macOS. At 64K, many coding sessions can preserve substantially more context; if memory pressure and Swap remain comfortable, increase only when the task really benefits. A claimed 128K ceiling does not by itself mean 128K of useful input fits: leave room for system/tool messages and output tokens. 1. Model capability Qwen’s published maximum . 2. Server admission/allocation CONTEXT , model settings . 3. Actual prompt tokens sent on a particular request. 4. Useful Pi history remaining after tool output and compaction. The second and third need checking on the exact Pi-to-server route; the fourth is about task quality, not merely token count. Decision rule: If a 64K real task is stable and satisfactory, keep it. If it fails because facts or tool outputs are regularly dropped too early, extend the window cautiously. If the 125K attempt causes persistent swap or much worse latency, return to the smaller window while checking whether cache precision or server behavior can improve the larger run. None of these choices requires assuming your original spike was a bug. 2. Which server would I compare, and why? Because Ollama is what you have, the first question is not “Which alternative should I install?” but “Can the existing server be made predictable enough to use as a baseline?” If it can, you can compare the result with any new runtime on the same Pi task rather than comparing someone else’s benchmark with your own anecdote. If it cannot, an alternative engine becomes a sensible way to distinguish a server-specific effect from a model-size or workflow limit. Ollama’s own docs illustrate why checking the live value matters: the upstream configuration reference https://github.com/ollama/ollama/blob/main/docs/faq.mdx describes a 4096-token default, while the dedicated context page https://docs.ollama.com/context-length lists memory-dependent defaults and shows a CONTEXT column in ollama ps . Rather than choose a default from the docs, inspect the loaded model and actual request route. Your proposed oMLX https://github.com/jundot/omlx + DFlash2 https://github.com/z-lab/dflash is a serious alternative. The Qwen3.8 DFlash2 draft https://huggingface.co/z-lab/Qwen3.8-27B-DFlash2 is designed to propose token blocks that the target verifies. When acceptance is good this can accelerate decoding ; the draft also adds state and has its own compatibility requirements. Evaluate three separate outcomes : a cold-prefill memory peak/pressure, b generation speed, and c a complete Pi tool cycle. DFlash principally targets b , not automatically a . A memory improvement may instead come from KV precision, quantized weights, a cache policy or a specific runtime fix; a decoding speedup can be offset by prefill or failed tool calls. oMLX releases https://github.com/jundot/omlx/releases move quickly. 0.8.2 was current in this November 21 check ; its release page https://github.com/jundot/omlx/releases/tag/v0.8.2 identifies a hotfix inheriting changes from 0.8.1. Don’t import pre-integration DFlash cache flags or old recipes uncritically: the model settings code https://github.com/jundot/omlx/blob/main/omlx/model settings.py has distinct settings for context admission, caches and speculative decoding. The published 128K benchmark’s max context window=71680 illustrates a key distinction: a direct benchmark input and a normal API chat admission limit need not be identical. Before comparing Pi performance, check that the server actually accepts the requested context through the endpoint Pi uses; do not extrapolate a benchmark rung into an API guarantee. A suitable llama.cpp https://github.com/ggml-org/llama.cpp build is worth keeping in the comparison set, especially if you want a GGUF-based server, explicit slot/context settings, or a different cache implementation. Different runner implementations can make a large difference for a hybrid architecture. That does not automatically mean the generic llama.cpp build is faster or more economical than oMLX on your chip; the exact model revision, quantization, build and flags matter. LM Studio https://lmstudio.ai/ is a reasonable option if you prefer a GUI-driven local-model workflow and compatible API serving. Splash https://github.com/incoai/splash is another relevant inference implementation for this model family, and one published same-hardware comparison reports attractive throughput and memory figures in its specific configuration. Its README documents a splash pi helper, so it may be relatively easy to try with Pi after verifying the installed release. It is worth including in a comparison. “LM Studio” does not name one invariant backend; versions and model routes matter. A reported MLX context-setting issue https://github.com/lmstudio-ai/lmstudio-bug-tracker/issues/2318 makes served-context verification worthwhile, but does not prove current LM Studio releases have the same problem. Similarly, a Splash result on one version is not a general ranking. For Pi, the practical questions are decisive: does it accept the model you want, report usage/context correctly, preserve tool-call schemas and tool results, permit the window you configured, and stay responsive across a long sequence? A runtime that is much faster on a stand-alone completion but makes tool calling unreliable is not necessarily the better agent server . A runtime that is slightly slower but stable and easy to operate may be a better everyday choice. That judgment is yours, not something a benchmark can decide by itself. | Package | Main reason to try | Main condition to check | | Current Ollama | Minimal migration; baseline | Effective context, Mac pressure, Pi tools | | oMLX + DFlash2 | Apple Silicon options; speculative decode | Cold vs warm latency, KV settings, accepted window | | llama.cpp/GGUF | Different cache implementation; portability | Exact model build and context/slot settings | | LM Studio/Splash | GUI or another inference profile | Bundled runtime, context, Pi API/tool calls | 3. Why the memory numbers are unusual for Qwen3.8-27B Qwen’s official model configuration https://huggingface.co/Qwen/Qwen3.8-27B/blob/main/config.json has 64 layers: 16 full-attention and 48 Gated DeltaNet/linear-attention layers . Those are different state representations. Treating all 64 as ordinary token-indexed KV layers would overstate the conventional attention-cache calculation; treating the model as having no other long-session state would understate the runtime problem. Here is a simple calculation , rather than a measurement. The official text config has four KV heads per full-attention layer, a 256-dimensional head and bf16 as a declared dtype. If all sixteen full-attention layers held conventional bf16 K and V for each token, the idealized storage would be: 16 full-attention layers 2 K and V 4 KV heads 256 values/head 2 bytes/value = 65,536 bytes per token 64 KiB per token 32,768 tokens: 2.00 GiB of this attention K/V 65,536 tokens: 4.00 GiB 125,000 tokens: about 7.63 GiB 131,072 tokens: 8.00 GiB This is illustrative arithmetic, not measured memory . It assumes conventional bf16 KV only in the 16 full-attention layers and excludes 4-bit model weights, recurrent state, preserved prefixes, cache metadata, temporary prefill allocations, any DFlash draft, MLX allocator overhead and macOS. Quantized KV, padding, batching and other implementations change real allocations; the figure is not a whole-process bound. - Weights/draft/vision: the loaded baseline. - Live context: K/V for full-attention layers plus recurrent/linear state. - Prefill workspace: potentially large but transient buffers during a cold long request. - Retained prefixes/snapshots: saved work for reuse; may occupy memory or disk after the reply. - Allocator and macOS: unified memory also serves the OS and tools; Memory Pressure/Swap expose the practical cost. A pinned Ollama MLX prefix-cache snapshot https://github.com/ollama/ollama/blob/2bd4d6aabdf4caa39e7095e201dcda4cc1a3a406/mlxrunner/prefix cache.go has an 8 GiB budget for paged-out snapshots and eviction logic. That is not a cap on total server RAM or live KV . The separate recurrent-cache code https://github.com/ollama/ollama/blob/2bd4d6aabdf4caa39e7095e201dcda4cc1a3a406/mlxrunner/cache/recurrent.go shows why cache accounting cannot be reduced to conventional KV alone. It is a versioned code observation, not proof that your installation used this exact path. A plateau can still be painful. Suppose the server is retaining caches by design, but the Mac starts compressing and swapping aggressively. The memory use may be “expected” from the server’s point of view and still produce a bad experience for the person using the machine. Conversely, a high but stable allocation with little memory pressure may be less urgent than a lower peak that repeatedly forces disk swap. The practical goal is to keep Pi usable. If the problem occurs as the first long prompt arrives , I would prioritize prefill peak, active K/V and model/draft headroom. Lower KV precision or a smaller input may help; prefill kernels and runtime versions may also change the temporary allocation. If the problem occurs only after many Pi turns , retained prefixes, recurrent snapshots, compaction behavior and request lifetimes move higher in the ranking. If it occurs when two activities overlap , look at batching, concurrency and cache copying. These are related mechanisms, not mutually exclusive diagnoses. 4. Pi context metadata and automatic compaction are another boundary There is an additional complication because your model is being used through Pi Harness rather than only through a one-shot server benchmark. The agent has its own model metadata, token accounting and history management. The server has its own context allocation and request-admission rules. The model architecture has its native capability. And there is the prompt that actually reaches inference. Those four limits can disagree without any single component being “broken.” The inspected public Pi models documentation https://github.com/earendil-works/pi/blob/main/packages/coding-agent/docs/models.md shows how a compatible Ollama/LM Studio-style endpoint can be configured in models.json . The compaction documentation https://github.com/earendil-works/pi/blob/main/packages/coding-agent/docs/compaction.md describes automatic history summarization. In the inspected source, the relevant basic threshold is: start considering compaction when: contextTokens contextWindow - reserveTokens ordinary documented reserveTokens default: 16,384 ordinary documented keepRecentTokens default: 20,000 In the inspected public Pi source these are defaults reserveTokens=16,384 , keepRecentTokens=20,000 , not observations of your local Pi. The first value is a reserve for the compaction trigger , not a statement that the agent always runs on 16K context. contextWindow is Pi’s declared window, which can differ from the server’s actual allocation. For example, Pi at 262,144 but Ollama at 64K would let Pi plan around a larger budget than the server can handle. Pi at 32K but a server at 128K may cause earlier-than-needed summarization. These are hypothetical mismatches, not a diagnosis or proof of a memory spike: the actual symptom might be compaction timing, rejection, or an interrupted tool loop. The simplest check is the selected model’s Pi context declaration versus the loaded model’s Ollama CONTEXT . If they differ materially, try aligning them to a comfortable served window such as 64K before troubleshooting deeper. That is preferable to editing lots of hidden knobs or manually forcing memory to a very high limit. If they already agree, that eliminates one common integration question and strengthens the case for looking at cache behavior or the exact prompt size. A coding agent may place substantial output from read , grep , a shell command or a test log into the next model request. The user sees “the same conversation,” but from the server’s perspective the prompt can change considerably between turns. The sequence may alternate between very large inputs, short requests that share earlier prefixes, and summaries after a compaction. A single long-context benchmark doesn’t reproduce all of that. It measures an important piece of performance, but not the entire agent workflow. Compaction introduces its own trade-off. Shorter retained history may reduce context cost and improve responsiveness, but a summary can omit details the model will later need for a correct edit. Keeping more verbatim history can improve continuity at the cost of memory and time. This is why I would not advise turning compaction off as a first response to a memory spike, or turning it up so aggressively that Pi loses the project state you need. A stable middle ground is often to configure a served window that is actually comfortable, align the agent’s declaration with it, and let the ordinary summarization mechanism do its intended job. The same integration check applies if you move to oMLX, llama.cpp or LM Studio. A new server may advertise a much larger nominal limit, but Pi still needs correct model metadata and a compatible API. A successful curl request to a bare completion endpoint is encouraging; a successful Pi tool cycle including the expected schema and retained history is stronger evidence that the migration will help you day to day. 5. Nearby measurements, issues, and version-specific fixes The oMLX community result https://omlx.ai/benchmarks/performance/vd6i0f5f is unusually close to your hardware. The page reports Qwen3.8-27B-4bit on an M5 Pro with 48 GB , under oMLX 0.6.3rc2 and a DFlash/TurboQuant-KV-8-bit configuration. The recorded ladder is: | Benchmark input | Reported peak memory | Reported prefill speed | Reported generation speed | | 32K tokens | 23.2 GB | 389.0 tok/s | 41.9 tok/s | | 64K tokens | 28.5 GB | 324.9 tok/s | 26.2 tok/s | | 128K tokens | 39.5 GB | 229.4 tok/s | 20.8 tok/s | The figures are community benchmark peaks , not macOS Swap measurements. In the same published settings max context window=71680 appears alongside the 128K test. Thus the run supports feasibility on that benchmark path , not automatic 128K acceptance by a normal Pi chat endpoint. Do not copy experimental memory-cap overrides from another M5 Pro recipe https://github.com/vikesh-c/Qwen-3.8-27B-oMLX without understanding the machine-specific guard and risk. The Garden Desk discussion https://github.com/Private-Garden-Labs/garden-desk/issues/176 includes an M5 Pro / 48-GB Qwen3.8-27B comparison of oMLX, oMLX + DFlash2, Splash and llama.cpp variants. It reports both very short prompts and prompts around 31K tokens . For example, an early 31K comparison gave oMLX at 27.3 GiB reported peak with bf16 KV , oMLX + DFlash2 at 25.4 GiB with bf16 KV , and Splash at 18.6 GiB with INT8 KV . The speed figures also differed substantially. In Ollama issue 18131 https://github.com/ollama/ollama/issues/18131 , an Apple Silicon user reported high swap during agent workloads and focused on the MLX prefix-cache budget. The issue is open at the time of this check. It involves a different Mac and agent setup, so the right takeaway is that cache retention can matter materially for whole-machine memory pressure , not that Pi on your M5 Pro necessarily hits the same code path or version-dependent behavior. Ollama issue 17924 https://github.com/ollama/ollama/issues/17924 is a useful counterexample: a rising memory trace was initially called a leak, but later observations showed a prefix-cache plateau , and the diagnosis was corrected. That does not prove every cache increase is harmless; watch Swap and whether growth levels off, and keep the exact version/runner distinct. PR 5253 https://github.com/jundot/omlx/pull/5253 addresses quantized KV rollback during batched speculative verification , and PR 5321 https://github.com/jundot/omlx/pull/5321 reduces extra KV copies when requests join or restore a batch . Both merged November 20, 2026. The relevant reports include long overlapping Qwen3.8 requests. They show how a steady-state estimate can miss temporary spikes; they do not diagnose a single Ollama prefill. These are oMLX-specific concurrency/cache paths . Some comparisons came from a 64-GB machine with a 48-GB software memory guard, which is not equivalent to physically having 48 GB. If using a recent release, check whether those fixes are present; don’t claim that installing 0.8.2 alone fixes the poster’s earlier Ollama memory event. The current release notes and source are also moving fast. The 0.8.2 release page https://github.com/jundot/omlx/releases/tag/v0.8.2 identifies a small hotfix and carries forward 0.8.1 features; it should not be described as a release that uniquely invented every preceding November cache improvement. Verify the actual build you test, especially if following an August or September recipe, because cache setting names and DFlash integration have changed over time. For context, there are reports involving MLX-LM memory during long-lived serving https://github.com/ml-explore/mlx-lm/issues/1807 , LM Studio’s MLX context setting https://github.com/lmstudio-ai/lmstudio-bug-tracker/issues/2318 , and hybrid-state cache behavior in llama.cpp https://github.com/ggml-org/llama.cpp/issues/29324 . These are useful pointers if you decide to reproduce an issue on another implementation. They are not independent confirmation of one common bug in Ollama, oMLX, Pi and llama.cpp. Similar symptoms can arise from different layers: actual context allocation, cache retention, recurrent-state snapshots, temporary batch copying, or incorrect model metadata. 6. A small comparison that answers the practical question If you already have Ollama running, these commands are enough for an initial snapshot: ollama --version ollama ps When the model is loaded, note the exact tag, effective CONTEXT , and whether it is being handled as expected under PROCESSOR . The Ollama guide https://docs.ollama.com/context-length shows an example of those columns. In Pi, note the selected model and its declared context if that information is readily visible. You do not need to dump private code, upload full conversation history, or run an expensive external benchmark just to answer the two basic questions. For a useful comparison, hold the model checkpoint/weight precision, task and effective context as close as practical. Mark whether each request was cold no reusable prefix or warm reuse possible . Otherwise an engine with a hot prefix may look much faster than one doing the same input from scratch. Change one main variable at a time : server first for a package comparison; KV precision for memory; speculation for decode speed; or Pi’s declared context for an integration check. Pi tool-call correctness, retries and time to completion matter more than a single token-rate headline. A compact note could look like this, with no need to fill every field: | Setup | Actual input | Warm/cold | Memory Pressure / Swap Used | Result | | Current Ollama, 4-bit, effective 64K | Representative Pi task | Cold | Observe, not estimated | Tool task completed? | | Same setup | Comparable later tool turn | Warm/reused | Observe | Still responsive? | | oMLX + DFlash2, comparable quant | Same task if convenient | Cold and then warm | Observe | Better total latency/quality? | | Optional 125K/128K | Only if you need it | Stated clearly | Stop if pressure becomes severe | Actual request accepted and useful? | Judge time to a correct result : cold prefill, first-token time, decode, tool schemas, retries, and whether the Mac stays responsive. The best raw decode benchmark need not win a coding-agent session. If you can only supply two things, the highest information value seems to be: 1 whether 125K was an actual input/history or only the configured limit , and 2 whether the memory spike happened on the first long prefill or after repeated Pi/tool calls . Those two facts distinguish several branches before any version-specific detective work. Memory Pressure and Swap observations would help but are not prerequisites. If you later have an exact reproducer, an issue report becomes much easier to act on with the software version, exact model tag and runner, effective context, number of overlapping requests, input token count if known, and whether the symptom disappears when the model is unloaded. You need not share private prompts to do that; a synthetic public prompt with roughly the same length can be sufficient when it reproduces the same behavior. If a synthetic prompt does not reproduce the issue, that is also a useful clue about cache reuse or the actual agent sequence. I would avoid raising Metal/wired-memory limits with sudo , disabling safety guards or blindly copying somebody else’s 128K recipe. Reserving OS headroom is safer than maximizing one GPU allocation at the cost of system-wide swap. Also avoid treating a setting from a closed PR, another backend, or an older DFlash engine as active in the version you installed. Check the release’s docs and the actual runtime path. An ignored knob can make comparisons misleading. A successful setup should load the selected 4-bit Qwen3.8 model , let Pi operate at a verified effective 32K or 64K context , complete the tool workflows you actually use, and avoid persistent memory pressure or debilitating swap. From there, it becomes an optimization problem rather than a troubleshooting emergency: increase real context if needed, test a supported KV-compression mode, and compare oMLX + DFlash2 against the reference on a representative workload. That preserves your original goal and leaves room to change your mind about the server. Bottom line: I would not assume you need to give up Qwen3.8-27B, Pi, or the possibility of 125K. I would start with a known 4-bit build and a verified 32K/64K serving window on Ollama, then compare oMLX + DFlash2 under the same useful task. The nearby M5 Pro/48-GB results make a longer window plausible, but they also show how little headroom a 128K run may leave. If you remember only whether the 125K figure was an actual request and whether the spike was cold or after many Pi turns, that would already narrow the explanation a great deal; the options above do not depend on having those answers first.