# How to Use unsloth/Qwen3.8–27B-GGUF in Claude Code via Ollama Without Dying in the Process? (2/2) > Source: > Published: 2026-09-01 23:01:01+00:00 In [Part 1](https://pub.towardsai.net/how-to-use-unsloth-qwen3-8-27b-gguf-in-claude-code-via-ollama-without-dying-in-the-process-1-2-31acd5105227?source=friends_link&sk=043480bd957da697f72701d36d55f92f) we patched the template and walked away with a GGUF that no longer blows up with System message must be at the beginning. That was the easy part. Now for the other thing: fitting 27 billion parameters, a context window that’s actually useful, and an overthinking agent into 24 GB of VRAM. Spoiler: there is no ideal configuration. There’s a negotiation, and you’re going to have it with your own hardware. The minimal Modelfile, text inference only: ``` FROM ./Qwen3.8-27B-UD-Q4_K_M-cc.ggufPARAMETER num_ctx 172032PARAMETER num_batch 768PARAMETER temperature 1.0PARAMETER top_p 0.95PARAMETER top_k 20PARAMETER min_p 0.0PARAMETER presence_penalty 0.0PARAMETER repeat_penalty 1.0 ollama create unsloth-qwen3.8-27b-q4_k_m-cc-ctx168k -f Modelfile ``` One thing worth saying out loud before anything else: **Ollama’s default context is 4096 tokens.** It does not inherit the model’s native window. Qwen3.8–27B ships with 262,144 tokens of native context and Ollama will still hand you 4K unless you say otherwise. PARAMETER num_ctx in the Modelfile is what overrides it (or OLLAMA_CONTEXT_LENGTH on the server, which sets the default for every model). If you skip this step, everything else in this article is pointless. The sampling parameters are the ones Qwen recommends for this family. Two are worth understanding instead of copying: repeat_penalty 1.0 means **disabled**. That's not an oversight. In code, legitimate repetition is constant (indentation, self. on every line, block closers, imports) and penalizing it produces exactly the class of output that looks correct until it doesn't compile. Just in case, bugs aren't legitimate repetitions. min_p 0.0 combined with top_k 20 and top_p 0.95 leaves the filtering to the latter two. If you're coming from tuning min_p on chat models, leave it alone here. If your project needs the agent to see images (UI screenshots, diagrams, mockups) you add a second FROM with the projector: ``` FROM ./Qwen3.8-27B-UD-Q4_K_M-cc.ggufFROM ./mmproj-F16.ggufPARAMETER num_ctx 163840... ``` The mmproj file **is not patched**. It goes in exactly as you downloaded it from [here](https://huggingface.co/unsloth/Qwen3.8-27B-GGUF/tree/main). All the work in [Part 1](https://pub.towardsai.net/how-to-use-unsloth-qwen3-8-27b-gguf-in-claude-code-via-ollama-without-dying-in-the-process-1-2-31acd5105227?source=friends_link&sk=043480bd957da697f72701d36d55f92f) was on the main model's chat template; the projector has no template to break. And this isn’t optional if you need it: **if you don’t include the projector and Claude Code tries to send an image, the request fails.** It doesn’t degrade gracefully, it doesn’t ignore the image. The request dies :’(! ``` OLLAMA_HOST=0.0.0.0 \OLLAMA_ORIGINS='*' \OLLAMA_FLASH_ATTENTION=1 \OLLAMA_KV_CACHE_TYPE=q8_0 \OLLAMA_NUM_PARALLEL=1 \OLLAMA_KEEP_ALIVE=-1 \ollama serve ``` Let’s be explicit with two of these variables: **OLLAMA_FLASH_ATTENTION=1 is not a prerequisite you have to remember anymore.** Ollama enables flash attention automatically wherever the selected backend and devices support it. The variable is a three-state override: unset means auto, 1 forces it on, 0 forces it off. I set it to 1 because I'd rather force a known state than inherit an auto-detection, but if you leave it out on supported hardware, nothing breaks. **OLLAMA_NUM_PARALLEL=1 is already the documented default.** Setting it explicitly is belt and braces, not a fix. The reason it matters is the scaling rule: required memory grows as OLLAMA_NUM_PARALLEL × OLLAMA_CONTEXT_LENGTH, so at 160K a second slot is not a rounding error, it's a second 160K KV cache. Pin it so a config change somewhere else can't quietly double your footprint. What the other three do: VariableWithout itOLLAMA_KV_CACHE_TYPE=q8_0the KV cache runs at f16 and weighs twice as much; your context gets cut in halfOLLAMA_KEEP_ALIVE=-1the model unloads after 5 minutes and you pay a cold load every time you come back from coffeeOLLAMA_HOST=0.0.0.0you can't reach it from outside the host **The q8_0 KV cache is the entry requirement, not an optimization.** Without it I couldn’t get anywhere, and not because of speed: because of a loop. With the context I had left, the agent would run inference, fill up, need to compact; it would compact, run inference, fill up again, need to compact. Forever. It never finished a feature because it never had room to work between compactions. I needed more than 96K of usable context as a floor to escape that circle, and at f16 KV that doesn’t fit (I only had 24 GB of GPU memory). On architectures that don’t support it, quantized KV cache silently falls back to f16. Setting q8_0 is not a guarantee that you got q8_0. Check your actual memory use with ollama ps instead of assuming. **About ****OLLAMA_HOST=0.0.0.0 with ****OLLAMA_ORIGINS='*':** that leaves the server open to your network. Ollama ships with no authentication on port 11434, and this has a track record: CVE-2025-63389 covers missing authentication that lets a remote attacker perform model-management operations on versions up to and including v0.12.3, and CVE-2026-5530 is an SSRF in the model-pull endpoint affecting versions up to 0.18.1, with active scanning observed in the wild. If you expose this beyond your LAN, put a layer in front of it (an authenticated tunnel, a reverse proxy with auth, anything). This is where the time goes. Qwen3.8–27B uses hybrid attention: of its 64 layers, **only 16 maintain a KV cache that grows with context**. The rest are linear-attention layers with a fixed-size recurrent state. The structure is 16 repetitions of (three Gated DeltaNet blocks + FFN → one Gated Attention block + FFN). Practical consequence: context on this model is abnormally cheap. Those 16 full-attention layers cost roughly **65.5 KB per token at f16** and, at q8_0, a bit under half that. “A bit under half” and not exactly half, because q8_0 isn't one clean byte per element: it stores blocks of 32 values plus a scale, which works out to about 1.06 bytes per element. So the honest number to budget with is **~35 KB per token**, not 32.8. On a conventional dense model this size, 160K tokens in 24 GB would be unthinkable. Here it fits. With that you can compute your own ceiling instead of copying mine: ``` KV ≈ tokens × 35 KB (with q8_0)VRAM_total ≈ weights + KV + compute buffers ``` And a warning that will save you an afternoon: **the KV cache type is invisible in the model name.** My tags say ctx160k and ctx168k, but nothing in them tells you whether that context is being stored at f16 or q8_0 — that lives in the server's environment, not in the Modelfile. Two people running the identical tag can get wildly different VRAM numbers. Any math in this section assumes q8_0 and collapses without it. Here I’m going to get opinionated, because this is a recommendation that gets made a lot and made lightly. Yes, q4_0 on the KV buys you more speed and more context. And yes, published measurements say the impact is small. But going from 16 bits to 4 bits of precision in the cache is, in my opinion, starting to talk about a different model. Not Qwen3.8. I already went to the trouble of not dropping below Q4 on the weights precisely so I wouldn't degrade tool-calling; degrading the KV is opening the same door through the window. **What good is 256K or 2M tokens of context if the model already starts degrading at 128K?** You’d be buying square footage in a zone where the model no longer performs. It’s paying precision for uninhabitable floor space. I ruled out q4 immediately and I don’t regret it. If your case is different (if you need an enormous window and tolerate more noise) go ahead, but treat it as a decision with a cost, not a free adjustment. My criterion, and the reason I ended up where I ended up: **If your context is too short**, the agent has nowhere to put the specs/reqs, its own reasoning, and the code it’s about to write on top of that. It suffocates. **If your context is too long**, you enter the degradation zone. And there it’s not just hallucinations: there are errors. Errors that force you into more loops, or worse, that stay in the code as bugs. Context that doesn’t perform is context that costs you. The ideal, for me, is **the point where degradation begins, plus a little**. That “plus a little” isn’t working context: it’s the cushion for the response when the agent is mid-loop, or when the compactor kicks in. In my case: **160K total = 128K of real work + 32K of output**. It’s no accident that 32K is exactly what I configured as CLAUDE_CODE_MAX_OUTPUT_TOKENS. If max output is 32K and the window is 160K, then 160 − 32 = 128 and everything lines up: the agent can fill its 128K, call the compactor, receive the summary, and carry on fresh with the task it was doing. The numbers have to match or the system jams. **And there is no ideal configuration.** It depends on what system you’re building, how granular your tasks are, your prompting patterns, the software architecture. What’s worth taking is the criterion, not my number. ModelVisionMax context without spillNotesUD-Q4_K_XLNo~160kmy initial configUD-Q4_K_XLYes~160k, ~10% on CPU**impractical, see below** UD-Q4_K_MYes~160kthe one I use for vision workUD-Q4_K_MNo~168–176kif you don’t need vision On disk, the XL builds land about 1 GB above the M builds. In q8_0 KV terms, that gigabyte is roughly 30K tokens of context. Sounds like nothing until you’re 3% from the limit. **I migrated from XL to M for one reason: my project required vision.** With XL plus the projector I couldn’t make it fit. With M I could, at 160K. If your project **doesn’t** need vision, stay on XL (Unsloth’s Dynamic quants allocate bits per tensor and keep more precision where the architecture is sensitive, and you have no reason to pay that back if you don’t have to). The case that cost me a night was unsloth-qwen3.8-27b-q4_k_xl-cc-vision-ctx160k. XL, with projector, at 160K. Ollama loaded it at **90% GPU, 10% CPU**. Decode went from **~46 t/s on a fully-offloaded build to 22–22.5 t/s**. Roughly half the speed for a tenth of the model. When I pushed to 192K context with a 512 batch on XL, CPU use climbed (16%) and decode fell **below 15 t/s**. Every token has to travel through those layers, and the round-trip latency to CPU dominates everything else. **In a local inference stack, plan for zero spill.** Either it all fits on the GPU or you’re paying for it on every single token. I ended up tuning between 10% and 0% CPU, moving context size, batch, and whether it was XL or M. Three coupled variables, chasing zero. This deserves saying because almost nobody does: **I don’t consider myself a GPU expert, but even though every card says 24 GB, everyone should find their own limits experimentally.** The OS, the desktop if you have one running, the driver, the CUDA buffers, whatever you have open in another window: it all eats. My real ceiling isn’t your real ceiling, even if the number printed on the box is the same. Here’s the part I didn’t expect. Look at these two ollama ps outputs, same tag, same context, same machine, different moments: ``` NAME SIZE PROCESSOR CONTEXTunsloth-qwen3.8-27b-q4_k_m-cc-vision-ctx160k:latest 22 GB 100% GPU 163840unsloth-qwen3.8-27b-q4_k_m-cc-vision-ctx160k:latest 23 GB 6%/94% CPU/GPU 163840 ``` One gigabyte of difference and a spill that appears out of nowhere, on an identical configuration. **Your ceiling doesn’t just differ from mine — it differs from yours between restarts.** Which is the real argument for leaving headroom instead of tuning to the last megabyte: a config that sits exactly at 100% GPU on a clean boot is a config that will spill the first time something else on your machine wants memory. The way to find your number: **move context in steps of 4K/8K.** It’s the right granularity. Fine enough to approach the limit without overshooting, coarse enough not to burn an afternoon. Go up until spill appears, drop one step, and that’s your number. Then drop one more, because of the paragraph above. The mechanical part: ``` export ANTHROPIC_BASE_URL=http://your-host:11434export ANTHROPIC_AUTH_TOKEN=ollamaexport ANTHROPIC_MODEL=unsloth-qwen3.8-27b-q4_k_m-cc-vision-ctx160k ``` Ollama accepts any key; it validates nothing. That’s already in twenty tutorials. What isn’t in those twenty tutorials is what follows. ``` export CLAUDE_CODE_MAX_CONTEXT_TOKENS=163840export CLAUDE_CODE_MAX_OUTPUT_TOKENS=32768export CLAUDE_CODE_DISABLE_1M_CONTEXT=1 ``` Claude Code **doesn’t know what’s on the other side of the endpoint**. It assumes cloud-Claude capabilities: enormous windows, long outputs, extended context available. And it will ask your 27B for things it can’t deliver. CLAUDE_CODE_MAX_CONTEXT_TOKENS is the one that declares the real window. This is the variable I had wrong for a while, so be precise: it's documented specifically for gateways and custom model IDs, and it applies directly when the model ID doesn't start with claude- and doesn't contain [1m] — which is exactly the case for an Ollama tag like ours. If your tag did start with claude-, this variable would be ignored unless you also disabled compaction entirely. CLAUDE_CODE_MAX_OUTPUT_TOKENS=32768 is your output budget, the same 32K from the math above. CLAUDE_CODE_DISABLE_1M_CONTEXT=1 is in my config and I'll be honest about what it does: for an unrecognized model ID like ours, Claude Code only assumes a 1M window if the ID contains [1m]. Mine doesn't, so this variable is doing nothing for me. It's harmless insurance, and it becomes strictly necessary the day you put [1m] in a tag name. Keep it, know why it's there. Optional but useful: CLAUDE_CODE_AUTO_COMPACT_WINDOW lets you set the compaction threshold in tokens directly, from 100000 to 1000000, as a plain integer. If you'd rather compact at 128K explicitly than derive it from window minus output, that's the knob. This is, by a wide margin, the part that took me longest to figure out and the part that’s least documented. **Claude Code doesn’t use a single model.** Even with Opus as your main model, it can reach for the Sonnet and Haiku tiers for work that isn’t the main thread — planning, subagents, background summarization for --resume, and various internal operations. Exactly which task routes to which tier is not something you control, and it's not something that's guaranteed to stay the same; it's an implementation detail that Anthropic changes when it makes sense for them to change it. Against an Anthropic endpoint, none of that matters. Against your Ollama, claude-haiku-... doesn't exist, Ollama returns a 404, and Claude Code reports it as a vague "model may not exist" error that points at the model you *did* configure rather than the ones you didn't. So you redirect all of them: ``` export ANTHROPIC_DEFAULT_OPUS_MODEL=unsloth-qwen3.8-27b-q4_k_m-cc-vision-ctx160kexport ANTHROPIC_DEFAULT_SONNET_MODEL=unsloth-qwen3.8-27b-q4_k_m-cc-vision-ctx160kexport ANTHROPIC_DEFAULT_HAIKU_MODEL=unsloth-qwen3.8-27b-q4_k_m-cc-vision-ctx160k ``` And here’s the counterintuitive part. You’d assume the secondary tiers can point at a smaller model — it’s just summarizing, right? **They can’t**, and the reason has two axes: **Window size.** Whatever runs a compaction receives the whole context that needs summarizing. If the context is full, that’s your entire window. A model with a smaller window can’t even read what you’re sending it. **Capabilities.** If your session has images in it, whatever handles it has to be able to see them. Point any tier at a model without a projector and it breaks. My rule, and I’d make it yours: **all three tiers get the same vision capability, or none of them do.** All or nothing. I ran the same K_M build in all three slots for a while, then experimented with XL on Opus and K_M on Sonnet and Haiku. Both work. What doesn’t work is leaving one blank, or mixing a vision build with a text-only one. The worst possible moment for this to surface: **compaction fails exactly when you need it most.** At the end of a long session, with hours of work inside the context. It doesn’t fail at the beginning, when losing the session wouldn’t hurt. Notice how this connects backwards: if in Step 8 you chose K_M with vision, every tier needs vision too. The quantization decision propagates all the way here. With 128K of usable context you are not going to describe the entire system to the agent and expect it to solve it. It’ll blow the context, full stop. What worked for me: **split the project into features or modules and advance one at a time, backed by a state file** (.md or .yaml) referenced from CLAUDE.md. The agent reads that file, knows where it left off, moves forward, updates the state. It can resume at any point (even after a compaction, even after you closed the terminal). Even so, some features required several compactions. Honestly, **128K isn’t a lot. But 64K is worse.** And again: it depends on your project. ``` export API_TIMEOUT_MS=1800000export CLAUDE_BYTE_STREAM_IDLE_TIMEOUT_MS=600000export API_FORCE_IDLE_TIMEOUT=0 ``` Claude Code’s defaults are calibrated for datacenter latency. Your 4090 is not a datacenter, and the client reads “slow” as “dead.” API_TIMEOUT_MS=1800000 is 30 minutes, against a default of 10. It looks excessive until you do the arithmetic on prefill. A 128K prompt is **tens of seconds to several minutes before the first token**, depending on your card and batch size, and that's in a session where the agent is dragging its whole history along. I'm deliberately not publishing a tokens-per-second figure for prefill here, because the only clean measurement I have is on a 42-token prompt, which is pure fixed overhead and measures nothing useful. Measure your own on a real prompt. CLAUDE_BYTE_STREAM_IDLE_TIMEOUT_MS=600000 is 10 minutes for the byte-level streaming watchdog, and it covers the gap during that long prefill. Nothing comes down the stream while the prompt is being processed. Nothing. And the client, on its defaults, decides the connection died. Note the variable is clamped between 10 seconds and 30 minutes, so 10 minutes is comfortably inside the accepted range. API_FORCE_IDLE_TIMEOUT=0 turns off the 5-minute body-idle timeout that aborts a streaming response when no bytes arrive — the documentation calls out slow gateways and local models as exactly the case for it. Important detail: the stream watchdogs run independently of this setting and will still abort a long silent pause even with it at 0. That's why you need both this and the byte-stream variable, not one or the other. If you have a single symptom that says “this is the problem,” it’s this one: **the agent cuts out in the middle of a long task with no clear error.** It’s not your network. It’s the timeouts. ``` NAME ID SIZE MODIFIEDunsloth-qwen3.8-27b-q4_k_xl-cc-vision-ctx160k:latest c5a5877d9a94 34 GB 26 seconds agounsloth-qwen3.8-27b-q4_k_xl-cc-ctx160k:latest 7c3d037195d5 17 GB 12 minutes ago ``` Seventeen GB next to thirty-four. Exactly double. After spending time optimizing 1.1 GB of quantization. **It’s not an Ollama bug and it’s not the projector.** The first time I built the Modelfile with both FROM lines it worked perfectly. This happened when I introduced a typo in the Modelfile during creation: specified wrong, Ollama imported the model twice instead of registering the projector. The irritating part was what came next: redoing ollama create with the correct name didn't fix it. The manifest was stuck. I had to **delete the model and create another one with a different name**. Moral: if you see a size that’s an exact multiple of your GGUF, check the Modelfile character by character before looking for more exotic culprits. And if the name is already burned, don’t fight it: use another one. For reference, here’s what the same build looks like when it’s correct — the XL with vision at 18 GB on disk, not 34: ``` unsloth-qwen3.8-27b-q4_k_xl-cc-vision-ctx160k:latest b79d419fae9d 18 GB ``` Qwen3.8–27B was trained with a multi-token-prediction head (native speculative decoding), where the model drafts its own next tokens and verifies them in a single pass. Published measurements put the gain north of 20%, and some setups report considerably more. On the [official Unsloth page for the model](https://huggingface.co/unsloth/Qwen3.8-27B-GGUF) you can download the MTP module as a separate Q4_0 file of about 1.37 GB. Ollama expose a draft_num_predict parameter you can set in the Modelfile (or pass at runtime) to control how many speculative tokens to use, and a DRAFT instruction for separate drafter models. It's not a llama.cpp-only feature anymore. So why isn’t it in my config? Because it isn’t free. That 1.37 GB has to live in VRAM, and so does the draft context on top of it. Read Step 8 again: I spent days tuning between 10% and 0% CPU spill, and the difference between XL and M (one gigabyte) was worth roughly 30K tokens of context. Adding MTP means paying that same currency again. On 24 GB, with my current context requirements, it was non-negotiable: I’d have to give up either context or weight quality to buy speed. Two things worth knowing if you do try it. The best value is usually low — community measurements often peak at 1 or 2 draft toke If you have more VRAM than me, or shorter context needs, this is probably the single highest-leverage thing you can turn on. My 20–50 t/s is the ceiling of my stack, not the ceiling of the model. num_batch controls how many prompt tokens get pushed through the model per forward pass during prefill. It's a **scheduling knob, not a numeric one**: the attention math and the resulting logits are identical whether you feed 512 tokens at once or 256 twice. The only thing that changes is the size of the intermediate compute buffer. I ran 384, 512, and 768 across contexts from 128K to 176K, with and without vision. **I didn’t observe a significant difference** in speed or in CPU usage. It affects prefill throughput and it nudges GPU memory slightly, but not enough to change any decision I made. I settled on 768 for the text-only build and left it alone. I’d tell you not to bother tuning it for this kind of workload. The value of this section isn’t the savings. It’s learning to tell a memory knob from a quality knob. num_batch doesn't touch quality; touching it isn't going to "make your model worse." But it isn't going to save you either if you're short by gigabytes. Qwen3.8 ships with a high default reasoning effort, and its creators recommend xhigh. You'll be tempted to drop it to low or medium the first time you watch it think for a long stretch before touching a file. Don't, at least not for code. The reasoning I’d apply (and I’ll flag that this is drawn from what others have measured rather than my own benchmarks) is that low doesn't make the model dumber in the sense of having different weights. It makes it less thorough, because it cuts its reasoning short and considers fewer things. It's the same model, so it retains the ability to notice its own mistakes and correct them. Which means what you often get isn't a wrong answer, it's a longer path to the right one: more trial and error, more loops, and more total tokens consumed than if it had just thought properly the first time. So the “faster response” you bought can be a net loss on wall-clock time, and it’s definitely a loss on context, which per Step 8 is the scarcest resource in this entire setup. And beyond the token math: in code, trading quality for speed is a bad trade from any angle you look at it. Leave it at xhigh. The one place this is worth knowing is Step 11. A model at xhigh can think for a long time before the first visible token, especially on the first prompt of a session. If you didn't set the timeouts, that's one more way for the client to conclude your server is dead when it's just being careful. [How to Use unsloth/Qwen3.8–27B-GGUF in Claude Code via Ollama Without Dying in the Process? (2/2)](https://pub.towardsai.net/how-to-use-unsloth-qwen3-8-27b-gguf-in-claude-code-via-ollama-without-dying-in-the-process-2-2-118f4effd428) was originally published in [Towards AI](https://pub.towardsai.net) on Medium, where people are continuing the conversation by highlighting and responding to this story.