How to Use unsloth/Qwen3.8–27B-GGUF in Claude Code via Ollama Without Dying in the Process? (2/2)

A technical guide from Towards AI details how to configure the unsloth/Qwen3.8-27B-GGUF model in Claude Code via Ollama, emphasizing that Ollama's default context is 4096 tokens and must be overridden with PARAMETER num_ctx to use the model's native 262,144-token window. The article provides a Modelfile with specific parameters, notes that repeat_penalty 1.0 disables repetition penalty for code generation, and explains that including the mmproj-F16.gguf projector is mandatory for image support, as requests fail without it. It also clarifies that OLLAMA_FLASH_ATTENTION=1 is an optional override and OLLAMA_NUM_PARALLEL=1 is the default, with memory scaling as OLLAMA_NUM_PARALLEL × OLLAMA_CONTEXT_LENGTH.

In Part 1 https://pub.towardsai.net/how-to-use-unsloth-qwen3-8-27b-gguf-in-claude-code-via-ollama-without-dying-in-the-process-1-2-31acd5105227?source=friends link&sk=043480bd957da697f72701d36d55f92f we patched the template and walked away with a GGUF that no longer blows up with System message must be at the beginning. That was the easy part. Now for the other thing: fitting 27 billion parameters, a context window that’s actually useful, and an overthinking agent into 24 GB of VRAM. Spoiler: there is no ideal configuration. There’s a negotiation, and you’re going to have it with your own hardware. The minimal Modelfile, text inference only: FROM ./Qwen3.8-27B-UD-Q4 K M-cc.ggufPARAMETER num ctx 172032PARAMETER num batch 768PARAMETER temperature 1.0PARAMETER top p 0.95PARAMETER top k 20PARAMETER min p 0.0PARAMETER presence penalty 0.0PARAMETER repeat penalty 1.0 ollama create unsloth-qwen3.8-27b-q4 k m-cc-ctx168k -f Modelfile One thing worth saying out loud before anything else: Ollama’s default context is 4096 tokens. It does not inherit the model’s native window. Qwen3.8–27B ships with 262,144 tokens of native context and Ollama will still hand you 4K unless you say otherwise. PARAMETER num ctx in the Modelfile is what overrides it or OLLAMA CONTEXT LENGTH on the server, which sets the default for every model . If you skip this step, everything else in this article is pointless. The sampling parameters are the ones Qwen recommends for this family. Two are worth understanding instead of copying: repeat penalty 1.0 means disabled . That's not an oversight. In code, legitimate repetition is constant indentation, self. on every line, block closers, imports and penalizing it produces exactly the class of output that looks correct until it doesn't compile. Just in case, bugs aren't legitimate repetitions. min p 0.0 combined with top k 20 and top p 0.95 leaves the filtering to the latter two. If you're coming from tuning min p on chat models, leave it alone here. If your project needs the agent to see images UI screenshots, diagrams, mockups you add a second FROM with the projector: FROM ./Qwen3.8-27B-UD-Q4 K M-cc.ggufFROM ./mmproj-F16.ggufPARAMETER num ctx 163840... The mmproj file is not patched . It goes in exactly as you downloaded it from here https://huggingface.co/unsloth/Qwen3.8-27B-GGUF/tree/main . All the work in Part 1 https://pub.towardsai.net/how-to-use-unsloth-qwen3-8-27b-gguf-in-claude-code-via-ollama-without-dying-in-the-process-1-2-31acd5105227?source=friends link&sk=043480bd957da697f72701d36d55f92f was on the main model's chat template; the projector has no template to break. And this isn’t optional if you need it: if you don’t include the projector and Claude Code tries to send an image, the request fails. It doesn’t degrade gracefully, it doesn’t ignore the image. The request dies :’ OLLAMA HOST=0.0.0.0 \OLLAMA ORIGINS=' ' \OLLAMA FLASH ATTENTION=1 \OLLAMA KV CACHE TYPE=q8 0 \OLLAMA NUM PARALLEL=1 \OLLAMA KEEP ALIVE=-1 \ollama serve Let’s be explicit with two of these variables: OLLAMA FLASH ATTENTION=1 is not a prerequisite you have to remember anymore. Ollama enables flash attention automatically wherever the selected backend and devices support it. The variable is a three-state override: unset means auto, 1 forces it on, 0 forces it off. I set it to 1 because I'd rather force a known state than inherit an auto-detection, but if you leave it out on supported hardware, nothing breaks. OLLAMA NUM PARALLEL=1 is already the documented default. Setting it explicitly is belt and braces, not a fix. The reason it matters is the scaling rule: required memory grows as OLLAMA NUM PARALLEL × OLLAMA CONTEXT LENGTH, so at 160K a second slot is not a rounding error, it's a second 160K KV cache. Pin it so a config change somewhere else can't quietly double your footprint. What the other three do: VariableWithout itOLLAMA KV CACHE TYPE=q8 0the KV cache runs at f16 and weighs twice as much; your context gets cut in halfOLLAMA KEEP ALIVE=-1the model unloads after 5 minutes and you pay a cold load every time you come back from coffeeOLLAMA HOST=0.0.0.0you can't reach it from outside the host The q8 0 KV cache is the entry requirement, not an optimization. Without it I couldn’t get anywhere, and not because of speed: because of a loop. With the context I had left, the agent would run inference, fill up, need to compact; it would compact, run inference, fill up again, need to compact. Forever. It never finished a feature because it never had room to work between compactions. I needed more than 96K of usable context as a floor to escape that circle, and at f16 KV that doesn’t fit I only had 24 GB of GPU memory . On architectures that don’t support it, quantized KV cache silently falls back to f16. Setting q8 0 is not a guarantee that you got q8 0. Check your actual memory use with ollama ps instead of assuming. About OLLAMA HOST=0.0.0.0 with OLLAMA ORIGINS=' ': that leaves the server open to your network. Ollama ships with no authentication on port 11434, and this has a track record: CVE-2025-63389 covers missing authentication that lets a remote attacker perform model-management operations on versions up to and including v0.12.3, and CVE-2026-5530 is an SSRF in the model-pull endpoint affecting versions up to 0.18.1, with active scanning observed in the wild. If you expose this beyond your LAN, put a layer in front of it an authenticated tunnel, a reverse proxy with auth, anything . This is where the time goes. Qwen3.8–27B uses hybrid attention: of its 64 layers, only 16 maintain a KV cache that grows with context . The rest are linear-attention layers with a fixed-size recurrent state. The structure is 16 repetitions of three Gated DeltaNet blocks + FFN → one Gated Attention block + FFN . Practical consequence: context on this model is abnormally cheap. Those 16 full-attention layers cost roughly 65.5 KB per token at f16 and, at q8 0, a bit under half that. “A bit under half” and not exactly half, because q8 0 isn't one clean byte per element: it stores blocks of 32 values plus a scale, which works out to about 1.06 bytes per element. So the honest number to budget with is ~35 KB per token , not 32.8. On a conventional dense model this size, 160K tokens in 24 GB would be unthinkable. Here it fits. With that you can compute your own ceiling instead of copying mine: KV ≈ tokens × 35 KB with q8 0 VRAM total ≈ weights + KV + compute buffers And a warning that will save you an afternoon: the KV cache type is invisible in the model name. My tags say ctx160k and ctx168k, but nothing in them tells you whether that context is being stored at f16 or q8 0 — that lives in the server's environment, not in the Modelfile. Two people running the identical tag can get wildly different VRAM numbers. Any math in this section assumes q8 0 and collapses without it. Here I’m going to get opinionated, because this is a recommendation that gets made a lot and made lightly. Yes, q4 0 on the KV buys you more speed and more context. And yes, published measurements say the impact is small. But going from 16 bits to 4 bits of precision in the cache is, in my opinion, starting to talk about a different model. Not Qwen3.8. I already went to the trouble of not dropping below Q4 on the weights precisely so I wouldn't degrade tool-calling; degrading the KV is opening the same door through the window. What good is 256K or 2M tokens of context if the model already starts degrading at 128K? You’d be buying square footage in a zone where the model no longer performs. It’s paying precision for uninhabitable floor space. I ruled out q4 immediately and I don’t regret it. If your case is different if you need an enormous window and tolerate more noise go ahead, but treat it as a decision with a cost, not a free adjustment. My criterion, and the reason I ended up where I ended up: If your context is too short , the agent has nowhere to put the specs/reqs, its own reasoning, and the code it’s about to write on top of that. It suffocates. If your context is too long , you enter the degradation zone. And there it’s not just hallucinations: there are errors. Errors that force you into more loops, or worse, that stay in the code as bugs. Context that doesn’t perform is context that costs you. The ideal, for me, is the point where degradation begins, plus a little . That “plus a little” isn’t working context: it’s the cushion for the response when the agent is mid-loop, or when the compactor kicks in. In my case: 160K total = 128K of real work + 32K of output . It’s no accident that 32K is exactly what I configured as CLAUDE CODE MAX OUTPUT TOKENS. If max output is 32K and the window is 160K, then 160 − 32 = 128 and everything lines up: the agent can fill its 128K, call the compactor, receive the summary, and carry on fresh with the task it was doing. The numbers have to match or the system jams. And there is no ideal configuration. It depends on what system you’re building, how granular your tasks are, your prompting patterns, the software architecture. What’s worth taking is the criterion, not my number. ModelVisionMax context without spillNotesUD-Q4 K XLNo~160kmy initial configUD-Q4 K XLYes~160k, ~10% on CPU impractical, see below UD-Q4 K MYes~160kthe one I use for vision workUD-Q4 K MNo~168–176kif you don’t need vision On disk, the XL builds land about 1 GB above the M builds. In q8 0 KV terms, that gigabyte is roughly 30K tokens of context. Sounds like nothing until you’re 3% from the limit. I migrated from XL to M for one reason: my project required vision. With XL plus the projector I couldn’t make it fit. With M I could, at 160K. If your project doesn’t need vision, stay on XL Unsloth’s Dynamic quants allocate bits per tensor and keep more precision where the architecture is sensitive, and you have no reason to pay that back if you don’t have to . The case that cost me a night was unsloth-qwen3.8-27b-q4 k xl-cc-vision-ctx160k. XL, with projector, at 160K. Ollama loaded it at 90% GPU, 10% CPU . Decode went from ~46 t/s on a fully-offloaded build to 22–22.5 t/s . Roughly half the speed for a tenth of the model. When I pushed to 192K context with a 512 batch on XL, CPU use climbed 16% and decode fell below 15 t/s . Every token has to travel through those layers, and the round-trip latency to CPU dominates everything else. In a local inference stack, plan for zero spill. Either it all fits on the GPU or you’re paying for it on every single token. I ended up tuning between 10% and 0% CPU, moving context size, batch, and whether it was XL or M. Three coupled variables, chasing zero. This deserves saying because almost nobody does: I don’t consider myself a GPU expert, but even though every card says 24 GB, everyone should find their own limits experimentally. The OS, the desktop if you have one running, the driver, the CUDA buffers, whatever you have open in another window: it all eats. My real ceiling isn’t your real ceiling, even if the number printed on the box is the same. Here’s the part I didn’t expect. Look at these two ollama ps outputs, same tag, same context, same machine, different moments: NAME SIZE PROCESSOR CONTEXTunsloth-qwen3.8-27b-q4 k m-cc-vision-ctx160k:latest 22 GB 100% GPU 163840unsloth-qwen3.8-27b-q4 k m-cc-vision-ctx160k:latest 23 GB 6%/94% CPU/GPU 163840 One gigabyte of difference and a spill that appears out of nowhere, on an identical configuration. Your ceiling doesn’t just differ from mine — it differs from yours between restarts. Which is the real argument for leaving headroom instead of tuning to the last megabyte: a config that sits exactly at 100% GPU on a clean boot is a config that will spill the first time something else on your machine wants memory. The way to find your number: move context in steps of 4K/8K. It’s the right granularity. Fine enough to approach the limit without overshooting, coarse enough not to burn an afternoon. Go up until spill appears, drop one step, and that’s your number. Then drop one more, because of the paragraph above. The mechanical part: export ANTHROPIC BASE URL=http://your-host:11434export ANTHROPIC AUTH TOKEN=ollamaexport ANTHROPIC MODEL=unsloth-qwen3.8-27b-q4 k m-cc-vision-ctx160k Ollama accepts any key; it validates nothing. That’s already in twenty tutorials. What isn’t in those twenty tutorials is what follows. export CLAUDE CODE MAX CONTEXT TOKENS=163840export CLAUDE CODE MAX OUTPUT TOKENS=32768export CLAUDE CODE DISABLE 1M CONTEXT=1 Claude Code doesn’t know what’s on the other side of the endpoint . It assumes cloud-Claude capabilities: enormous windows, long outputs, extended context available. And it will ask your 27B for things it can’t deliver. CLAUDE CODE MAX CONTEXT TOKENS is the one that declares the real window. This is the variable I had wrong for a while, so be precise: it's documented specifically for gateways and custom model IDs, and it applies directly when the model ID doesn't start with claude- and doesn't contain 1m — which is exactly the case for an Ollama tag like ours. If your tag did start with claude-, this variable would be ignored unless you also disabled compaction entirely. CLAUDE CODE MAX OUTPUT TOKENS=32768 is your output budget, the same 32K from the math above. CLAUDE CODE DISABLE 1M CONTEXT=1 is in my config and I'll be honest about what it does: for an unrecognized model ID like ours, Claude Code only assumes a 1M window if the ID contains 1m . Mine doesn't, so this variable is doing nothing for me. It's harmless insurance, and it becomes strictly necessary the day you put 1m in a tag name. Keep it, know why it's there. Optional but useful: CLAUDE CODE AUTO COMPACT WINDOW lets you set the compaction threshold in tokens directly, from 100000 to 1000000, as a plain integer. If you'd rather compact at 128K explicitly than derive it from window minus output, that's the knob. This is, by a wide margin, the part that took me longest to figure out and the part that’s least documented. Claude Code doesn’t use a single model. Even with Opus as your main model, it can reach for the Sonnet and Haiku tiers for work that isn’t the main thread — planning, subagents, background summarization for --resume, and various internal operations. Exactly which task routes to which tier is not something you control, and it's not something that's guaranteed to stay the same; it's an implementation detail that Anthropic changes when it makes sense for them to change it. Against an Anthropic endpoint, none of that matters. Against your Ollama, claude-haiku-... doesn't exist, Ollama returns a 404, and Claude Code reports it as a vague "model may not exist" error that points at the model you did configure rather than the ones you didn't. So you redirect all of them: export ANTHROPIC DEFAULT OPUS MODEL=unsloth-qwen3.8-27b-q4 k m-cc-vision-ctx160kexport ANTHROPIC DEFAULT SONNET MODEL=unsloth-qwen3.8-27b-q4 k m-cc-vision-ctx160kexport ANTHROPIC DEFAULT HAIKU MODEL=unsloth-qwen3.8-27b-q4 k m-cc-vision-ctx160k And here’s the counterintuitive part. You’d assume the secondary tiers can point at a smaller model — it’s just summarizing, right? They can’t , and the reason has two axes: Window size. Whatever runs a compaction receives the whole context that needs summarizing. If the context is full, that’s your entire window. A model with a smaller window can’t even read what you’re sending it. Capabilities. If your session has images in it, whatever handles it has to be able to see them. Point any tier at a model without a projector and it breaks. My rule, and I’d make it yours: all three tiers get the same vision capability, or none of them do. All or nothing. I ran the same K M build in all three slots for a while, then experimented with XL on Opus and K M on Sonnet and Haiku. Both work. What doesn’t work is leaving one blank, or mixing a vision build with a text-only one. The worst possible moment for this to surface: compaction fails exactly when you need it most. At the end of a long session, with hours of work inside the context. It doesn’t fail at the beginning, when losing the session wouldn’t hurt. Notice how this connects backwards: if in Step 8 you chose K M with vision, every tier needs vision too. The quantization decision propagates all the way here. With 128K of usable context you are not going to describe the entire system to the agent and expect it to solve it. It’ll blow the context, full stop. What worked for me: split the project into features or modules and advance one at a time, backed by a state file .md or .yaml referenced from CLAUDE.md. The agent reads that file, knows where it left off, moves forward, updates the state. It can resume at any point even after a compaction, even after you closed the terminal . Even so, some features required several compactions. Honestly, 128K isn’t a lot. But 64K is worse. And again: it depends on your project. export API TIMEOUT MS=1800000export CLAUDE BYTE STREAM IDLE TIMEOUT MS=600000export API FORCE IDLE TIMEOUT=0 Claude Code’s defaults are calibrated for datacenter latency. Your 4090 is not a datacenter, and the client reads “slow” as “dead.” API TIMEOUT MS=1800000 is 30 minutes, against a default of 10. It looks excessive until you do the arithmetic on prefill. A 128K prompt is tens of seconds to several minutes before the first token , depending on your card and batch size, and that's in a session where the agent is dragging its whole history along. I'm deliberately not publishing a tokens-per-second figure for prefill here, because the only clean measurement I have is on a 42-token prompt, which is pure fixed overhead and measures nothing useful. Measure your own on a real prompt. CLAUDE BYTE STREAM IDLE TIMEOUT MS=600000 is 10 minutes for the byte-level streaming watchdog, and it covers the gap during that long prefill. Nothing comes down the stream while the prompt is being processed. Nothing. And the client, on its defaults, decides the connection died. Note the variable is clamped between 10 seconds and 30 minutes, so 10 minutes is comfortably inside the accepted range. API FORCE IDLE TIMEOUT=0 turns off the 5-minute body-idle timeout that aborts a streaming response when no bytes arrive — the documentation calls out slow gateways and local models as exactly the case for it. Important detail: the stream watchdogs run independently of this setting and will still abort a long silent pause even with it at 0. That's why you need both this and the byte-stream variable, not one or the other. If you have a single symptom that says “this is the problem,” it’s this one: the agent cuts out in the middle of a long task with no clear error. It’s not your network. It’s the timeouts. NAME ID SIZE MODIFIEDunsloth-qwen3.8-27b-q4 k xl-cc-vision-ctx160k:latest c5a5877d9a94 34 GB 26 seconds agounsloth-qwen3.8-27b-q4 k xl-cc-ctx160k:latest 7c3d037195d5 17 GB 12 minutes ago Seventeen GB next to thirty-four. Exactly double. After spending time optimizing 1.1 GB of quantization. It’s not an Ollama bug and it’s not the projector. The first time I built the Modelfile with both FROM lines it worked perfectly. This happened when I introduced a typo in the Modelfile during creation: specified wrong, Ollama imported the model twice instead of registering the projector. The irritating part was what came next: redoing ollama create with the correct name didn't fix it. The manifest was stuck. I had to delete the model and create another one with a different name . Moral: if you see a size that’s an exact multiple of your GGUF, check the Modelfile character by character before looking for more exotic culprits. And if the name is already burned, don’t fight it: use another one. For reference, here’s what the same build looks like when it’s correct — the XL with vision at 18 GB on disk, not 34: unsloth-qwen3.8-27b-q4 k xl-cc-vision-ctx160k:latest b79d419fae9d 18 GB Qwen3.8–27B was trained with a multi-token-prediction head native speculative decoding , where the model drafts its own next tokens and verifies them in a single pass. Published measurements put the gain north of 20%, and some setups report considerably more. On the official Unsloth page for the model https://huggingface.co/unsloth/Qwen3.8-27B-GGUF you can download the MTP module as a separate Q4 0 file of about 1.37 GB. Ollama expose a draft num predict parameter you can set in the Modelfile or pass at runtime to control how many speculative tokens to use, and a DRAFT instruction for separate drafter models. It's not a llama.cpp-only feature anymore. So why isn’t it in my config? Because it isn’t free. That 1.37 GB has to live in VRAM, and so does the draft context on top of it. Read Step 8 again: I spent days tuning between 10% and 0% CPU spill, and the difference between XL and M one gigabyte was worth roughly 30K tokens of context. Adding MTP means paying that same currency again. On 24 GB, with my current context requirements, it was non-negotiable: I’d have to give up either context or weight quality to buy speed. Two things worth knowing if you do try it. The best value is usually low — community measurements often peak at 1 or 2 draft toke If you have more VRAM than me, or shorter context needs, this is probably the single highest-leverage thing you can turn on. My 20–50 t/s is the ceiling of my stack, not the ceiling of the model. num batch controls how many prompt tokens get pushed through the model per forward pass during prefill. It's a scheduling knob, not a numeric one : the attention math and the resulting logits are identical whether you feed 512 tokens at once or 256 twice. The only thing that changes is the size of the intermediate compute buffer. I ran 384, 512, and 768 across contexts from 128K to 176K, with and without vision. I didn’t observe a significant difference in speed or in CPU usage. It affects prefill throughput and it nudges GPU memory slightly, but not enough to change any decision I made. I settled on 768 for the text-only build and left it alone. I’d tell you not to bother tuning it for this kind of workload. The value of this section isn’t the savings. It’s learning to tell a memory knob from a quality knob. num batch doesn't touch quality; touching it isn't going to "make your model worse." But it isn't going to save you either if you're short by gigabytes. Qwen3.8 ships with a high default reasoning effort, and its creators recommend xhigh. You'll be tempted to drop it to low or medium the first time you watch it think for a long stretch before touching a file. Don't, at least not for code. The reasoning I’d apply and I’ll flag that this is drawn from what others have measured rather than my own benchmarks is that low doesn't make the model dumber in the sense of having different weights. It makes it less thorough, because it cuts its reasoning short and considers fewer things. It's the same model, so it retains the ability to notice its own mistakes and correct them. Which means what you often get isn't a wrong answer, it's a longer path to the right one: more trial and error, more loops, and more total tokens consumed than if it had just thought properly the first time. So the “faster response” you bought can be a net loss on wall-clock time, and it’s definitely a loss on context, which per Step 8 is the scarcest resource in this entire setup. And beyond the token math: in code, trading quality for speed is a bad trade from any angle you look at it. Leave it at xhigh. The one place this is worth knowing is Step 11. A model at xhigh can think for a long time before the first visible token, especially on the first prompt of a session. If you didn't set the timeouts, that's one more way for the client to conclude your server is dead when it's just being careful. How to Use unsloth/Qwen3.8–27B-GGUF in Claude Code via Ollama Without Dying in the Process? 2/2 https://pub.towardsai.net/how-to-use-unsloth-qwen3-8-27b-gguf-in-claude-code-via-ollama-without-dying-in-the-process-2-2-118f4effd428 was originally published in Towards AI https://pub.towardsai.net on Medium, where people are continuing the conversation by highlighting and responding to this story.