# Setup for local ai models on apple silicon

> Source: <https://discuss.huggingface.co/t/setup-for-local-ai-models-on-apple-silicon/192253#post_3>
> Published: 2026-11-21 21:47:07+00:00

**Recommendation:** Start with 4-bit weights and a verified 32K–64K served context on Ollama, then compare oMLX with DFlash2 on a full coding task, watching Memory Pressure and Swap Used rather than peak process memory.
More in the playbook "Inference and self-hosting": <https://wpnews.pro/playbooks/inference-serving>

Hmm… I’m not very familiar with Macs, so I did some digging:

## 

For **Qwen3.8-27B with Pi Harness on an M5 Pro/48 GB**, my first choice would be **reliable 4-bit weights and a verified 32K–64K working context**, using your existing Ollama setup as the reference. Then I would compare **oMLX + DFlash2** on the same coding task. If 125K/128K is genuinely useful, keep it as a second target rather than abandoning it: the question is whether it fits with usable latency, reasonable memory pressure, and correct tool calls.

- **Quant:** Start with a maintained**4-bit weight** build. Do not confuse weight quantization with**KV-cache precision** : 4-bit weights can still use full-precision KV. An 8-bit KV setting may save context memory if the specific backend actually supports it; GGUF and MLX quantization labels are not automatically equivalent.
- **Context:** Begin with**32K/64K actually served** , then try 96K/125K/128K if useful. Model-card context, configured maximum, an accepted API request and effective retained agent history are different things.
- **Server:****Ollama first** (least disruption),**oMLX + DFlash2 next** (strong decoding candidate), with**llama.cpp/GGUF** and**LM Studio/Splash** as alternative packages. Compare full Pi tasks, not just decode tok/s.

A close [community oMLX result for M5 Pro/48 GB](https://omlx.ai/benchmarks/performance/vd6i0f5f) reports peaks of **23.2 GB at 32K**, **28.5 GB at 64K** and **39.5 GB at 128K**, with 4-bit weights and 8-bit KV. It used **oMLX 0.6.3rc2**, and its published recipe lists `max_context_window=71680` despite the 128K benchmark input. This shows experimental feasibility in a related stack, **not** that a normal Pi API request can safely use 128K on your Ollama installation.

I have **not tested this on a Mac or reproduced your memory spike**. The observations below come from published documentation, source code, issue discussions and other people’s measurements, so I would keep the exact cause of your earlier spike open.

### 

1. With the model loaded, run `ollama --version` and`ollama ps` ; note the model tag,`PROCESSOR` , and effective`CONTEXT` ([Ollama context guide](https://docs.ollama.com/context-length) ). Don’t infer these from an assumed default.
2. Was **125K the actual prompt/history** , or only the configured ceiling? Compare Pi’s declared model window with the server’s effective window.
3. Did memory rise on the **first long prefill** ,**after repeated Pi/tool turns** , or**only with overlapping requests** ? These point to different mechanisms.

On macOS, watch **Memory Pressure and Swap Used**, not just a peak process number ([Apple’s Activity Monitor guide](https://support.apple.com/en-mide/guide/activity-monitor/actmntr1004/mac)). A brief allocation, a retained cache that plateaus and sustained system-wide swapping are different outcomes.

My decision map would be roughly:

```
Was ~125K the configured ceiling, rather than the actual prompt size?
  Yes: align Pi's model context declaration with the server's real limit.
       Check accepted request size before diagnosing a 125K prefill.
  No:  did memory jump on the first long request?
         Yes: investigate active KV, prefill peak, and headroom/swap.
         No:  did it grow after many Pi turns or simultaneous requests?
                Yes: investigate retained prefixes, agent history,
                     compaction, cache lifecycle, and concurrency.
                No:  check exact version, model build, and reproducibility.
After that, compare one alternative server with the same actual task.
```

## 
1. Choosing weight quantization, KV precision, and useful context

### 

The word “quant” gets used for more than one object. **Weight quantization** changes the representation of the model’s parameters: a 4-bit model generally costs much less to keep loaded than unquantized bf16 weights. **KV-cache quantization** changes the storage format of attention keys and values created while processing prompts and generating text. A 4-bit *weight* build can still have a bf16 KV cache; equally, a backend may support 8-bit or 4-bit KV on top of some weight format. A draft model used for speculative decoding can be a third memory consumer.

For **48 GB unified memory**, a maintained 4-bit conversion is the sensible first trial. Quantization quality is particularly important for a coding agent: tool-call schemas, correct file edits, and recovery after mistakes count more than plausible text alone. More aggressive 2/3-bit weights may be worth exploring later, but saving memory at the expense of reliable tool execution can lose time overall.

I would **not** assume that “Q4” in a GGUF name, “4bit” on an MLX repository, and a particular Ollama tag mean mathematically identical quantizers. They can differ in group size, outlier handling, tensor exemptions, conversion version and the underlying runner. Two builds labeled 4-bit can have different disk sizes, runtime allocations and quality. For a fair comparison, keep the exact model artifact or at least record the precise tag/revision. If a comparison switches model format and inference engine at the same time, the result belongs to the **combination**, not to one magic property of MLX or GGUF.

### 

Ollama’s [upstream configuration reference](https://github.com/ollama/ollama/blob/main/docs/faq.mdx) documents `OLLAMA_KV_CACHE_TYPE` (`f16`, `q8_0`, `q4_0`) with Flash Attention. However, the availability and *effect* of those generic settings depend on the version and actual runner; historical MLX paths did not necessarily honor every documented cache knob. I would not prescribe `q8_0` as an assured fix without checking that your Qwen backend uses it.

[oMLX](https://github.com/jundot/omlx) provides its **own** cache formats, including affine/TurboQuant choices on supported paths. Do not translate an Ollama `q8_0` flag directly into an oMLX setting. Test memory **and** quality under the specific release, since low-precision cache, kernels and cache reuse can interact.

For an agent, the practical quality checks are retrieval of a fact near the beginning of a long history, consistent tool arguments, and a successful multi-step edit. Smaller cache formats can save memory without preserving every long-context task equally well.

### 

**32K/64K are starting points, not architectural ceilings.** At 32K there is more headroom for prefill, tools and macOS. At 64K, many coding sessions can preserve substantially more context; if memory pressure and Swap remain comfortable, increase only when the task really benefits. A claimed 128K ceiling does not by itself mean 128K of useful input fits: leave room for system/tool messages and output tokens.

### 

1. **Model capability** (Qwen’s published maximum).
2. **Server admission/allocation** (`CONTEXT` , model settings).
3. **Actual prompt tokens** sent on a particular request.
4. **Useful Pi history** remaining after tool output and compaction.
The second and third need checking on the exact Pi-to-server route; the fourth is about task quality, not merely token count.

**Decision rule:** If a 64K real task is stable and satisfactory, keep it. If it fails because facts or tool outputs are regularly dropped too early, extend the window cautiously. If the 125K attempt causes persistent swap or much worse latency, return to the smaller window while checking whether cache precision or server behavior can improve the larger run. None of these choices requires assuming your original spike was a bug.

 
## 
2. Which server would I compare, and why?

### 

Because Ollama is what you have, the first question is not “Which alternative should I install?” but “Can the existing server be made predictable enough to use as a baseline?” If it can, you can compare the result with any new runtime on the **same Pi task** rather than comparing someone else’s benchmark with your own anecdote. If it cannot, an alternative engine becomes a sensible way to distinguish a server-specific effect from a model-size or workflow limit.

Ollama’s own docs illustrate why checking the live value matters: the [upstream configuration reference](https://github.com/ollama/ollama/blob/main/docs/faq.mdx) describes a 4096-token default, while the [dedicated context page](https://docs.ollama.com/context-length) lists memory-dependent defaults and shows a `CONTEXT` column in `ollama ps`. Rather than choose a default from the docs, inspect the loaded model and actual request route.

### 

Your proposed [oMLX](https://github.com/jundot/omlx) + [DFlash2](https://github.com/z-lab/dflash) is a serious alternative. The [Qwen3.8 DFlash2 draft](https://huggingface.co/z-lab/Qwen3.8-27B-DFlash2) is designed to propose token blocks that the target verifies. When acceptance is good this can accelerate **decoding**; the draft also adds state and has its own compatibility requirements.

Evaluate **three separate outcomes**: (a) cold-prefill memory peak/pressure, (b) generation speed, and (c) a complete Pi tool cycle. DFlash principally targets (b), **not automatically (a)**. A memory improvement may instead come from KV precision, quantized weights, a cache policy or a specific runtime fix; a decoding speedup can be offset by prefill or failed tool calls.

[oMLX releases](https://github.com/jundot/omlx/releases) move quickly. **0.8.2 was current in this November 21 check**; its [release page](https://github.com/jundot/omlx/releases/tag/v0.8.2) identifies a hotfix inheriting changes from 0.8.1. Don’t import pre-integration DFlash cache flags or old recipes uncritically: the [model settings code](https://github.com/jundot/omlx/blob/main/omlx/model_settings.py) has distinct settings for context admission, caches and speculative decoding.

The published 128K benchmark’s `max_context_window=71680` illustrates a key distinction: a **direct benchmark input** and a **normal API chat admission limit** need not be identical. Before comparing Pi performance, check that the server actually accepts the requested context through the endpoint Pi uses; do not extrapolate a benchmark rung into an API guarantee.

### 

A suitable [llama.cpp](https://github.com/ggml-org/llama.cpp) build is worth keeping in the comparison set, especially if you want a GGUF-based server, explicit slot/context settings, or a different cache implementation. Different runner implementations can make a large difference for a hybrid architecture. That does **not** automatically mean the generic llama.cpp build is faster or more economical than oMLX on your chip; the exact model revision, quantization, build and flags matter.

### 

[LM Studio](https://lmstudio.ai/) is a reasonable option if you prefer a GUI-driven local-model workflow and compatible API serving. [Splash](https://github.com/incoai/splash) is another relevant inference implementation for this model family, and one published same-hardware comparison reports attractive throughput and memory figures in its specific configuration. Its README documents a `splash pi` helper, so it may be relatively easy to try with Pi after verifying the installed release. It is worth including in a comparison.

“LM Studio” does not name one invariant backend; versions and model routes matter. A [reported MLX context-setting issue](https://github.com/lmstudio-ai/lmstudio-bug-tracker/issues/2318) makes served-context verification worthwhile, but does not prove current LM Studio releases have the same problem. Similarly, a Splash result on one version is not a general ranking.

For Pi, the practical questions are decisive: does it accept the model you want, report usage/context correctly, preserve tool-call schemas and tool results, permit the window you configured, and stay responsive across a long sequence? A runtime that is much faster on a stand-alone completion but makes tool calling unreliable is not necessarily the better **agent server**. A runtime that is slightly slower but stable and easy to operate may be a better everyday choice. That judgment is yours, not something a benchmark can decide by itself.

### 

| Package | Main reason to try | Main condition to check | 
| Current Ollama | Minimal migration; baseline | Effective context, Mac pressure, Pi tools | 
| oMLX + DFlash2 | Apple Silicon options; speculative decode | Cold vs warm latency, KV settings, accepted window | 
| llama.cpp/GGUF | Different cache implementation; portability | Exact model build and context/slot settings | 
| LM Studio/Splash | GUI or another inference profile | Bundled runtime, context, Pi API/tool calls | 

  
## 
3. Why the memory numbers are unusual for Qwen3.8-27B

Qwen’s [official model configuration](https://huggingface.co/Qwen/Qwen3.8-27B/blob/main/config.json) has **64 layers: 16 full-attention and 48 Gated DeltaNet/linear-attention layers**. Those are different state representations. Treating all 64 as ordinary token-indexed KV layers would overstate the conventional attention-cache calculation; treating the model as having no other long-session state would understate the runtime problem.

Here is a simple *calculation*, rather than a measurement. The official text config has four KV heads per full-attention layer, a 256-dimensional head and bf16 as a declared dtype. If all sixteen full-attention layers held conventional bf16 K and V for each token, the idealized storage would be:

```
16 full-attention layers * 2 (K and V) * 4 KV heads
    * 256 values/head * 2 bytes/value
= 65,536 bytes per token (64 KiB per token)

32,768 tokens:    2.00 GiB of this attention K/V
65,536 tokens:    4.00 GiB
125,000 tokens:   about 7.63 GiB
131,072 tokens:   8.00 GiB
```

This is **illustrative arithmetic, not measured memory**. It assumes conventional bf16 KV only in the 16 full-attention layers and excludes 4-bit model weights, recurrent state, preserved prefixes, cache metadata, temporary prefill allocations, any DFlash draft, MLX allocator overhead and macOS. Quantized KV, padding, batching and other implementations change real allocations; the figure is not a whole-process bound.

### 

- **Weights/draft/vision:** the loaded baseline.
- **Live context:** K/V for full-attention layers plus recurrent/linear state.
- **Prefill workspace:** potentially large but transient buffers during a cold long request.
- **Retained prefixes/snapshots:** saved work for reuse; may occupy memory or disk after the reply.
- **Allocator and macOS:** unified memory also serves the OS and tools; Memory Pressure/Swap expose the practical cost.

### 

A pinned [Ollama MLX prefix-cache snapshot](https://github.com/ollama/ollama/blob/2bd4d6aabdf4caa39e7095e201dcda4cc1a3a406/mlxrunner/prefix_cache.go) has an **8 GiB budget for paged-out snapshots** and eviction logic. That is **not a cap on total server RAM or live KV**. The separate [recurrent-cache code](https://github.com/ollama/ollama/blob/2bd4d6aabdf4caa39e7095e201dcda4cc1a3a406/mlxrunner/cache/recurrent.go) shows why cache accounting cannot be reduced to conventional KV alone. It is a versioned code observation, not proof that your installation used this exact path.

A plateau can still be painful. Suppose the server is retaining caches by design, but the Mac starts compressing and swapping aggressively. The memory use may be “expected” from the server’s point of view and still produce a bad experience for the person using the machine. Conversely, a high but stable allocation with little memory pressure may be less urgent than a lower peak that repeatedly forces disk swap. The practical goal is to keep Pi usable.

### 

If the problem occurs **as the first long prompt arrives**, I would prioritize prefill peak, active K/V and model/draft headroom. Lower KV precision or a smaller input may help; prefill kernels and runtime versions may also change the temporary allocation. If the problem occurs **only after many Pi turns**, retained prefixes, recurrent snapshots, compaction behavior and request lifetimes move higher in the ranking. If it occurs **when two activities overlap**, look at batching, concurrency and cache copying. These are related mechanisms, not mutually exclusive diagnoses.

 
## 
4. Pi context metadata and automatic compaction are another boundary

There is an additional complication because your model is being used through **Pi Harness** rather than only through a one-shot server benchmark. The agent has its own model metadata, token accounting and history management. The server has its own context allocation and request-admission rules. The model architecture has its native capability. And there is the prompt that actually reaches inference. Those four limits can disagree without any single component being “broken.”

The [inspected public Pi models documentation](https://github.com/earendil-works/pi/blob/main/packages/coding-agent/docs/models.md) shows how a compatible Ollama/LM Studio-style endpoint can be configured in `models.json`. The [compaction documentation](https://github.com/earendil-works/pi/blob/main/packages/coding-agent/docs/compaction.md) describes automatic history summarization. In the inspected source, the relevant basic threshold is:

```
start considering compaction when:
    contextTokens > contextWindow - reserveTokens

ordinary documented reserveTokens default: 16,384
ordinary documented keepRecentTokens default: 20,000
```

In the inspected public Pi source these are **defaults** (`reserveTokens=16,384`, `keepRecentTokens=20,000`), not observations of your local Pi. The first value is a *reserve for the compaction trigger*, not a statement that the agent always runs on 16K context. `contextWindow` is Pi’s declared window, which can differ from the server’s actual allocation.

For example, **Pi at 262,144 but Ollama at 64K** would let Pi plan around a larger budget than the server can handle. **Pi at 32K but a server at 128K** may cause earlier-than-needed summarization. These are hypothetical mismatches, not a diagnosis or proof of a memory spike: the actual symptom might be compaction timing, rejection, or an interrupted tool loop.

The simplest check is the selected model’s **Pi context declaration** versus the loaded model’s Ollama `CONTEXT`. If they differ materially, try aligning them to a comfortable served window such as 64K before troubleshooting deeper. That is preferable to editing lots of hidden knobs or manually forcing memory to a very high limit. If they already agree, that eliminates one common integration question and strengthens the case for looking at cache behavior or the exact prompt size.

### 

A coding agent may place substantial output from `read`, `grep`, a shell command or a test log into the next model request. The user sees “the same conversation,” but from the server’s perspective the prompt can change considerably between turns. The sequence may alternate between very large inputs, short requests that share earlier prefixes, and summaries after a compaction. A single long-context benchmark doesn’t reproduce all of that. It measures an important piece of performance, but not the entire agent workflow.

Compaction introduces its own trade-off. Shorter retained history may reduce context cost and improve responsiveness, but a summary can omit details the model will later need for a correct edit. Keeping more verbatim history can improve continuity at the cost of memory and time. This is why I would not advise turning compaction off as a first response to a memory spike, or turning it up so aggressively that Pi loses the project state you need. A stable middle ground is often to configure a served window that is actually comfortable, align the agent’s declaration with it, and let the ordinary summarization mechanism do its intended job.

The same integration check applies if you move to oMLX, llama.cpp or LM Studio. A new server may advertise a much larger nominal limit, but Pi still needs correct model metadata and a compatible API. A successful `curl` request to a bare completion endpoint is encouraging; a successful **Pi tool cycle** including the expected schema and retained history is stronger evidence that the migration will help you day to day.

 
## 
5. Nearby measurements, issues, and version-specific fixes

### 

The [oMLX community result](https://omlx.ai/benchmarks/performance/vd6i0f5f) is unusually close to your hardware. The page reports Qwen3.8-27B-4bit on an M5 Pro with **48 GB**, under **oMLX 0.6.3rc2** and a DFlash/TurboQuant-KV-8-bit configuration. The recorded ladder is:

| Benchmark input | Reported peak memory | Reported prefill speed | Reported generation speed | 
| 32K tokens | 23.2 GB | 389.0 tok/s | 41.9 tok/s | 
| 64K tokens | 28.5 GB | 324.9 tok/s | 26.2 tok/s | 
| 128K tokens | 39.5 GB | 229.4 tok/s | 20.8 tok/s | 

 The figures are **community benchmark peaks**, not macOS Swap measurements. In the same published settings `max_context_window=71680` appears alongside the 128K test. Thus the run supports feasibility **on that benchmark path**, not automatic 128K acceptance by a normal Pi chat endpoint. Do not copy experimental memory-cap overrides from [another M5 Pro recipe](https://github.com/vikesh-c/Qwen-3.8-27B-oMLX) without understanding the machine-specific guard and risk.

### 

The [Garden Desk discussion](https://github.com/Private-Garden-Labs/garden-desk/issues/176) includes an M5 Pro / 48-GB Qwen3.8-27B comparison of oMLX, oMLX + DFlash2, Splash and llama.cpp variants. It reports both very short prompts and prompts around **31K tokens**. For example, an early 31K comparison gave oMLX at **27.3 GiB reported peak with bf16 KV**, oMLX + DFlash2 at **25.4 GiB with bf16 KV**, and Splash at **18.6 GiB with INT8 KV**. The speed figures also differed substantially.

### 

In [Ollama issue #18131](https://github.com/ollama/ollama/issues/18131), an Apple Silicon user reported high swap during agent workloads and focused on the MLX prefix-cache budget. The issue is open at the time of this check. It involves a different Mac and agent setup, so the right takeaway is **that cache retention can matter materially for whole-machine memory pressure**, not that Pi on your M5 Pro necessarily hits the same code path or version-dependent behavior.

[Ollama issue #17924](https://github.com/ollama/ollama/issues/17924) is a useful counterexample: a rising memory trace was initially called a leak, but later observations showed a **prefix-cache plateau**, and the diagnosis was corrected. That does **not** prove every cache increase is harmless; watch Swap and whether growth levels off, and keep the exact version/runner distinct.

### 

[PR #5253](https://github.com/jundot/omlx/pull/5253) addresses quantized KV rollback during **batched speculative verification**, and [PR #5321](https://github.com/jundot/omlx/pull/5321) reduces extra KV copies when requests **join or restore a batch**. Both merged November 20, 2026. The relevant reports include long overlapping Qwen3.8 requests. They show how a steady-state estimate can miss temporary spikes; they do **not** diagnose a single Ollama prefill.

These are **oMLX-specific concurrency/cache paths**. Some comparisons came from a 64-GB machine with a 48-GB software memory guard, which is not equivalent to physically having 48 GB. If using a recent release, check whether those fixes are present; don’t claim that installing 0.8.2 alone fixes the poster’s earlier Ollama memory event.

The current release notes and source are also moving fast. The [0.8.2 release page](https://github.com/jundot/omlx/releases/tag/v0.8.2) identifies a small hotfix and carries forward 0.8.1 features; it should not be described as a release that uniquely invented every preceding November cache improvement. Verify the actual build you test, especially if following an August or September recipe, because cache setting names and DFlash integration have changed over time.

### 

For context, there are reports involving [MLX-LM memory during long-lived serving](https://github.com/ml-explore/mlx-lm/issues/1807), [LM Studio’s MLX context setting](https://github.com/lmstudio-ai/lmstudio-bug-tracker/issues/2318), and [hybrid-state cache behavior in llama.cpp](https://github.com/ggml-org/llama.cpp/issues/29324). These are useful pointers if you decide to reproduce an issue on another implementation. They are **not** independent confirmation of one common bug in Ollama, oMLX, Pi and llama.cpp. Similar symptoms can arise from different layers: actual context allocation, cache retention, recurrent-state snapshots, temporary batch copying, or incorrect model metadata.

 
## 
6. A small comparison that answers the practical question

### 

If you already have Ollama running, these commands are enough for an initial snapshot:

```
ollama --version
ollama ps
```

When the model is loaded, note the exact tag, effective `CONTEXT`, and whether it is being handled as expected under `PROCESSOR`. The [Ollama guide](https://docs.ollama.com/context-length) shows an example of those columns. In Pi, note the selected model and its declared context if that information is readily visible. You do not need to dump private code, upload full conversation history, or run an expensive external benchmark just to answer the two basic questions.

### 

For a useful comparison, hold the model checkpoint/weight precision, task and effective context as close as practical. Mark whether each request was **cold** (no reusable prefix) or **warm** (reuse possible). Otherwise an engine with a hot prefix may look much faster than one doing the same input from scratch.

Change **one main variable at a time**: server first for a package comparison; KV precision for memory; speculation for decode speed; or Pi’s declared context for an integration check. Pi tool-call correctness, retries and time to completion matter more than a single token-rate headline.

A compact note could look like this, with no need to fill every field:

| Setup | Actual input | Warm/cold | Memory Pressure / Swap Used | Result | 
| Current Ollama, 4-bit, effective 64K | Representative Pi task | Cold | Observe, not estimated | Tool task completed? | 
| Same setup | Comparable later tool turn | Warm/reused | Observe | Still responsive? | 
| oMLX + DFlash2, comparable quant | Same task if convenient | Cold and then warm | Observe | Better total latency/quality? | 
| Optional 125K/128K | Only if you need it | Stated clearly | Stop if pressure becomes severe | Actual request accepted and useful? | 

 Judge **time to a correct result**: cold prefill, first-token time, decode, tool schemas, retries, and whether the Mac stays responsive. The best raw decode benchmark need not win a coding-agent session.

### 

If you can only supply two things, the highest information value seems to be: **(1) whether 125K was an actual input/history or only the configured limit**, and **(2) whether the memory spike happened on the first long prefill or after repeated Pi/tool calls**. Those two facts distinguish several branches before any version-specific detective work. Memory Pressure and Swap observations would help but are not prerequisites.

If you later have an exact reproducer, an issue report becomes much easier to act on with the software version, exact model tag and runner, effective context, number of overlapping requests, input token count if known, and whether the symptom disappears when the model is unloaded. You need not share private prompts to do that; a synthetic public prompt with roughly the same length can be sufficient when it reproduces the same behavior. If a synthetic prompt does not reproduce the issue, that is also a useful clue about cache reuse or the actual agent sequence.

### 

I would avoid raising Metal/wired-memory limits with `sudo`, disabling safety guards or blindly copying somebody else’s 128K recipe. Reserving OS headroom is safer than maximizing one GPU allocation at the cost of system-wide swap.

Also avoid treating a setting from a closed PR, another backend, or an older DFlash engine as active in the version you installed. Check the release’s docs and the actual runtime path. An ignored knob can make comparisons misleading.

### 

A successful setup should **load the selected 4-bit Qwen3.8 model**, let Pi operate at a **verified effective 32K or 64K context**, complete the tool workflows you actually use, and avoid persistent memory pressure or debilitating swap. From there, it becomes an optimization problem rather than a troubleshooting emergency: increase real context if needed, test a supported KV-compression mode, and compare oMLX + DFlash2 against the reference on a representative workload. That preserves your original goal and leaves room to change your mind about the server.

 
**Bottom line:** I would not assume you need to give up Qwen3.8-27B, Pi, or the possibility of 125K. I would start with a known 4-bit build and a *verified* 32K/64K serving window on Ollama, then compare oMLX + DFlash2 under the same useful task. The nearby M5 Pro/48-GB results make a longer window plausible, but they also show how little headroom a 128K run may leave. If you remember only whether the 125K figure was an actual request and whether the spike was cold or after many Pi turns, that would already narrow the explanation a great deal; the options above do not depend on having those answers first.
