{"slug": "benchmarking-qwen-3-6-35b-moe-3b-active-on-an-rtx-3090", "title": "Benchmarking Qwen 3.6 35B MoE (3B active) on an RTX 3090", "summary": "Benchmarking Qwen 3.6 35B MoE (3B active) on an RTX 3090 using Unsloth's UD-IQ4_NL_XL quantisation achieved up to 140 tokens per second for generation and over 3,300 tok/s for prompt processing with a CUDA-compiled version of Llama.cpp, though VRAM constraints limited context window to 89,600 tokens. Offloading 10 layers' FFNs to CPU enabled full 262,144-token context at 89 tok/s generation and 1,100 tok/s prompt speed.", "body_md": "## Benchmarking Qwen 3.6 35B MoE (3B active) on an RTX 3090\n\nI mentioned I'd got a second RTX 3090 on a group chat, and a friend said:\n\nI know this is not really your thing... but let me know how quickly it runs Qwen 3.6 35bn MoE. With only 24gb of VRAM you’ll need to use a 4-bit quantized version and you won’t get a massive context window. But it should still be pretty cool.\n\nHe's right that it's not really been my thing -- I've been focusing on my\nown LLMs recently. I decided to dig in a little, and in particular to play with\n[Llama.cpp](https://llama-cpp.com/), which I haven't used for a while. And then things got\na tad out of control, and I wound up doing some relatively detailed benchmarking.\n\nThe headline results: I downloaded Unsloth's `UD-IQ4_NL_XL`\n\nquantisation of the model\nfrom [Hugging Face](https://huggingface.co/unsloth/Qwen3.6-35B-A3B-GGUF). With that,\nusing the default Arch build of Llama.cpp, which uses Vulkan under the hood:\n\n- Using the GPU only, I was able to get the model to generate at just over 120 tokens per second, and it was able to process the prompt at just less than 2,800 tok/s. However, having the whole model on the GPU didn't leave that much space for the context window -- it was constrained to about 50,000 tokens, compared to the model's native context length of 262,144.\n- Offloading the FFNs for the first 12 of the model's 40 layers to the CPU managed to reclaim enough VRAM to be able to get the full context length; however, with that setup, things were -- unsurprisingly -- slower. I got just over 65 tok/s for generation, and 600 tok/s for the prompt.\n\nCompiling Llama.cpp myself, in order to get the full CUDA version, helped a lot:\n\n- With everything on the GPU, I got 140 tok/s for generation and over 3,300 tok/s for the prompt. That was with a context window of 89,600. So everything was better :-)\n- It was also easier to get to the full context length; that needed just 10 layers' FFNs to be offloaded, and at that point I was getting 89 tok/s for generation, and about 1,100 tok/s for the prompt.\n\nThat was a pretty impressive improvement.\n\nBut it also showed that the lack of VRAM on the 3090 really does hurt. The friend who asked me about this is running an Intel Arc B70 Pro, with 32 GiB VRAM. He can (of course) fit the whole 4-bit quantised model on there without any context window reductions, and he says this about throughput:\n\nIt starts off at 75-80. but drops into the high 50s as the context window expands\n\nThat's worse than the CUDA-with-offload results above, though I did limit my testing\nto a 2,457-token prompt, with 6,144 tokens generated. The RTX 5090 has 32 GiB VRAM\n*and* fast Nvidia processing -- I imagine things would be a lot better there. Downside:\nit costs more than three times what you pay for a 3090.\n\nAnyway, in the rest of this post, I'll give more details of the benchmark, including charts showing the performance at different numbers of offloaded layers, and comparisons of the Vulkan vs CUDA versions of Llama.cpp for this model.\n\nOne caveat: Qwen 3.6 is a multimodal model. That means that it has an extra component\nto map graphical inputs to tokens that can be consumed by the LLM. I didn't realise this\nas I ran these tests, but you can actually tell it not to use that component if you're only working with\ntext inputs -- for example, `--no-mmproj`\n\nwill do the trick with Llama.cpp. If you're working with text-only stuff, you\nmight want to play with that to see if you can get better results -- it will free up\nsome VRAM and might allow you to get longer context lengths, or better performance.\n\nBut anyway, as with many of my posts, this is a tidied up version of\nmy lab notes, so you can share my learning journey with me -- but if that sounds pointless,\nand you've landed here because you want to run this model on your own RTX 3090, you\ncan click [this link](#the-results) to jump right to the results.\n\n## The model\n\n[Qwen3.6-35B-A3B](https://huggingface.co/Qwen/Qwen3.6-35B-A3B) is a Mixture of Experts\nmodel with 35B total parameters, 3B active. The tensor type is BF16, so it's two bytes\nper parameter. That's ~70 GiB of space required for the model, just for the weights, with\nno space for attention matrices and the like. We're\ngoing to need a quantised version.\n\nI'll dig deep into MoE models at some point, but\nessentially they work by having multiple different FFN blocks for each Transformers layer.\nAfter attention, the context vectors are routed to a subset of those blocks,\ndepending on their content 1. The number of active parameters is how many are actually used\nwhen running a single token through.\n\nNow, that number, 3B, looked really crazy to me as soon as I saw it. Qwen 3.6, as you can see from the model card linked above, has a vocabulary size of 248,320, and an embedding dimensionality of 2,048. It doesn't use weight-tying, so that means that the embedding layer and the output head come out at a hefty 508,559,360 parameters each.\n\nBoth the embedding layer and the output head are required for every token, of course, so that means that of\nthose 3B active parameters, more than 1B are used just getting tokens into and out of the\nmodel! Only 2B active parameters are left to handle all of the attention and the active FFNs. That is\nan imbalance of the same level as [you get with GPT-2 small](/2026/07/llm-parameter-counts),\nand was surprising to see.\n\nWell, let's think about VRAM requirements. Naively you might think that you only need to keep the active parameters in VRAM, swapping FFN experts in and out as needed. Unfortunately that doesn't work.\n\nLet's say\nyou feed in \"The fat cat sat on the\", hoping for a completion. The whole sequence is\nprocessed in parallel (that being the point of using a GPT-style LLM rather than, say,\nan [RNN](/2025/10/revisiting-karpathy-unreasonable-effectiveness-rnns)). You'll need\nwhichever experts are required for all six tokens in the prompt for the first layer, and likewise for all of the\ncontext vectors for the prompt in each of the later layers.\n\nSo, realistically, for a non-trivial prompt, you can't \"swap out\" the inactive parameters;\nan MoE buys you lower usage in terms of\nprocessing, but not in terms of memory. 2 That said, as we'll see later, you can at\nleast pull stuff out of VRAM into normal RAM, and get some advantages that way.\n\nAs my friend said, a 4-bit quant would be needed -- perhaps with that, the whole model, both active and inactive parameters, would fit into 35 / 2 = 17.5 GiB of VRAM, leaving 6.5 GiB for activations, attention matrices, and so on? As it turned out, it needed a bit more, and it's worth taking a look into why.\n\n## Quantisation\n\nI must admit that I'd never really looked into quantisation prior to playing with this, and had naively assumed that you basically just took (say) a BF16 model, scaled the parameters down so that they were (say) FP4, tweaked the computations a bit and then ran the result.\n\nThat was completely wrong! I'll have to dig into it further in the future, but my new working mental model is that it's more like lossy compression. The weights are stored in a \"compressed\" format -- one that is designed so that they can be quickly and cheaply \"decompressed\" by code running on the target platform.\n\nSo, speaking very loosely, when running an unquantised model, we might have CUDA code doing something like this:\n\n- Load BF16 weights from VRAM\n- Load BF16 data from another bit of VRAM\n- Do a matrix multiplication of one by the other.\n- Store the results into VRAM as BF16\n\n...a quantised model might operate more like this:\n\n- Load quantised weights from VRAM\n- \"Decompress\" them to BF16\n- Load BF16 data from another bit of VRAM\n- Do a matrix multiplication of one by the other.\n- Store the results into VRAM as BF16\n\nUnderstanding that minimal amount clarified three things for me:\n\n- Why people talk about things like \"4.1-bit quants\". We're talking about the average number of bits per weight in the \"compressed\" model.\n- Why there are different quants of about the same size for a given model; for example,\nwhen we get on to looking at quantised versions of this model later, you'll see 4-bit ones called\n`UD-IQ4_NL`\n\n,`UD-Q4_K_M`\n\n,`UD-IQ4_NL_XL`\n\n, and`UD-Q4_K_XL`\n\n, with different sizes in terms of GiB. Each of those is using a different set of trade-offs in the quantisation process, so will perform differently on different tasks. I gather that picking the right one for any given task is a bit of an art form. Hugging Face have a summary of the different types they support[here](https://huggingface.co/docs/hub/main/en/gguf#quantization-types), which has*some*explanation of the differences, but decoding what they're saying there is beyond what I've learnt at this point. - How GPUs can run quants with bits-per-parameter levels that they don't support\nfor calculations. For example, the RTX 3090 supports 32-bit and two forms of\n16-bit (float16 and BF16) plus integer operations of various bittednesses\n. 4-bit formats like FP4 are only supported in more recent cards like the RTX 5090. But, of course, we're not doing any 4-bit computations -- it's all in some format that the GPU can handle natively.[3](#fn-3)\n\nSo, with that (minimal) level of understanding, it was time to dig in!\n\n## Getting Llama.cpp up and running\n\nThere are a bunch of different ways that people run LLMs locally, so I needed to work\nout which one would be likely to give the best results. One thing I noticed when\ngoogling around for things like \"Qwen3.6-35B-A3B RTX 3090\" was that pretty much everyone\nappeared to be using [Llama.cpp](https://llama-cpp.com/). That wasn't something I'd used for a couple\nof years, and I've repaved my machine multiple times since, so it was time to install it.\n\nI run Arch Linux, and there's [an OS package for it](https://wiki.archlinux.org/title/Llama.cpp),\nso I installed it using the instructions there -- specifically, the ones to run inference using CUDA:\n\n```\nsudo pacman -S llama-cpp ggml cuda\n```\n\nNow it was time to try a model. I decided to start off with a small one that would fit onto\nmy GPU without any quantisation, just to work out any bugs. I needed the model\nto be in GGUF format (that being what Llama.cpp uses). GGUF stores tensors and metadata --\nthe actual architectures of the LLMs Llama.cpp supports are baked in to the tool's\nsource -- but, of course, it supports Qwen3, and so a nice simple non-MoE small\nmodel that looked like it would fit onto my GPU was [ Qwen/Qwen3-4B-GGUF](https://huggingface.co/Qwen/Qwen3-4B-GGUF).\n\nThe help page for that model gave this example command to run it:\n\n```\n./llama-cli -hf Qwen/Qwen3-4B-GGUF:Q8_0 --jinja --color -ngl 99 -fa -sm row \\\n    --temp 0.6 --top-k 20 --top-p 0.95 --min-p 0 --presence-penalty 1.5 \\\n    -c 40960 -n 32768 --no-context-shift\n```\n\nThat was out-of-date and gave various errors (which was fair enough, the model being from late December 2025 -- all of seven months old!). The fixes were simple enough:\n\n```\nllama-cli -hf Qwen/Qwen3-4B-GGUF:Q8_0 --jinja --color on -ngl 99 -fa on -sm row \\\n    --temp 0.6 --top-k 20 --top-p 0.95 --min-p 0 --presence-penalty 1.5 \\\n    -c 40960 -n 32768 --no-context-shift\n```\n\nThat is, `--color`\n\nand `-fa`\n\nnow needed an `on`\n\nargument.\n\nI kicked it off, and got an error:\n\n```\nDownloading Qwen3-4B-Q8_0.gguf ───────────────────────────────────── 100%\n\nLoading model... \\1.54.635.084 E llama_model_load: error loading model: device Vulkan0 does not support split buffers\n1.54.635.088 E llama_model_load_from_file_impl: failed to load model\n1.54.635.110 E common_fit_params: encountered an error while trying to fit params to free device memory: failed to load model\n|1.54.741.517 E llama_model_load: error loading model: device Vulkan0 does not support split buffers\n1.54.741.519 E llama_model_load_from_file_impl: failed to load model\n1.54.741.521 E cmn  common_init_: failed to load model '/home/giles/.cache/huggingface/hub/models--Qwen--Qwen3-4B-GGUF/snapshots/bc640142c66e1fdd12af0bd68f40445458f3869b/Qwen3-4B-Q8_0.gguf'\n1.54.741.523 E srv    load_model: failed to load model, '/home/giles/.cache/huggingface/hub/models--Qwen--Qwen3-4B-GGUF/snapshots/bc640142c66e1fdd12af0bd68f40445458f3869b/Qwen3-4B-Q8_0.gguf'\nllama_server exited with code 1\n1.54.742.424 E srv  llama_server: exiting due to model loading error\nError: the server exited before becoming ready\n```\n\nThat seemed strange. The instructions I'd followed on the Arch Wiki page were the\nones to install Llama.cpp for CUDA, but it was referring to `device Vulkan0`\n\n. Vulkan\nis an open GPU-programming language -- it fits into the stack in essentially the same\nplace as CUDA. As always seems to happen with these things, while the open platform\nis, well, open, and runs on more platforms than the closed one, it is somewhat slower\nand buggier.\n\nSo, why did I have a Vulkan version? Looking\nat the [talk page for the package](https://wiki.archlinux.org/title/Talk:Llama.cpp)\nwas enlightening: user `Gotoro`\n\nwrote:\n\nHello. The article only provides the\n\n`llama.cpp-vulkan`\n\npackage for GPU inference, even though on AUR there is a CUDA package available from the same submitter [1]I must add that the CUDA package is \"Flagged out-of-date (2025-12-22)\"\n\n...and got a reply from `SH3NG1UN`\n\nsaying:\n\nHi. The reason CUDA was not added is that Vulkan solution is general, performant enough for most hardware setup, and I personally believe that is what edge AI deployment should look like - if your software stack is capable of gaming, it is capable of AI.\n\nThey went on to say that they would add CUDA support, but they hit an issue because the\nAUR version that `Gotoro`\n\nhad referred to no longer had an active maintainer. For non-Arch\nusers, the AUR is essentially a repository of non-official, community maintained\npackages.\n\nSo, it looked like I had a Vulkan stack installed. Could I install the CUDA one from the AUR instead?\n\nOver the last few months, there have been security issues with the AUR, where bad actors have picked up ownership of unmaintained packages and put nasty stuff into them -- exfiltration code to steal things like SSH keys and crypto wallets, for example.\n\nAs a result, I've been pretty cautious about adding on new AUR dependencies -- and this specific one having been unmaintained for a while scared me.\n\nI decided that I would stick with Vulkan for now; if I wanted to try with CUDA, I'd compile it myself from source later on.\n\nSo, that left the error message:\n\n```\ndevice Vulkan0 does not support split buffers\n```\n\nI decided to break down the command like step by step, as running random stuff you copied from the Internet is rarely a good way to understand what's going on.\n\n```\nllama-cli -hf Qwen/Qwen3-4B-GGUF:Q8_0\n```\n\nWe're running `llama-cli`\n\nand specifying a model from Hugging Face -- that's clearly what `-hf`\n\nmeans.\n\n```\n--jinja\n```\n\nPer the `--help`\n\n, \"whether to use jinja template engine for chat\".\n\n```\n--color on\n```\n\n\"Colorize output to distinguish prompt and user input from generations\"\n\n```\n-ngl 99\n```\n\nThis is \"max. number of layers to store in VRAM\". For this model, I wanted everything\nin VRAM, so felt that \"all\" would be better there. `99`\n\nbasically means \"all\",\nat least for any normal-sized model, but I wanted to be explicit.\n\n```\n-fa on\n```\n\nThis switched on Flash Attention, which seemed reasonable.\n\n```\n-sm row\n```\n\nThis was \"how to split the model across multiple GPUs\". The error was \"does not support split buffers\",\nso this sounded relevant! Given that I only have one GPU on my machine, I decided\nthat `-sm none`\n\nwould be a better option here.\n\nAt this point I suspected I had worked out how to solve the problem, but decided to take a look at the other options just out of interest.\n\n```\n--temp 0.6 --top-k 20\n```\n\nThese were clearly the normal sampling parameters that I was used to from creating my own models.\n\n```\n --top-p 0.95\n```\n\nThis was an extension to the previous sampling params: in addition to the `top-k`\n\nstuff (and after it),\nlimit sampling to the tokens that take up the top 0.95\nof the probability distribution.\n\n```\n--min-p 0\n```\n\nThis was just disabling the `min-p`\n\nfeature. min-p discards tokens that are less than\nX times the probability of the most likely token. Apparently using it is a promising\nalternative to messing around with `top-p`\n\nand `top-k`\n\n. Rabbit hole alert, let's move on.\n\n```\n--presence-penalty 1.5\n```\n\nPer the docs, this applies a \"repeat alpha presence penalty\". That was interesting!\nI googled around a bit and found [this page](https://mbrenndoerfer.com/writing/repetition-penalties-language-model-generation),\nwhere it became clear that it is a trick to stop models from getting stuck in loops.\nSmall models are quite prone to that, so it certainly sounded useful. Perhaps something\nto dig into more later, but for now, keeping it seemed perfectly reasonable.\n\n```\n-c 40960\n```\n\nThat was the \"size of the prompt context\" according to the help -- that is, the amount of the context window that can be taken up by stuff going into the model before it starts generating.\n\n```\n-n 32768\n```\n\n...and this was the other side of that equation: the maximum number of tokens to predict.\n\n```\n--no-context-shift\n```\n\nAnother interesting one, \"whether to use context shift on infinite text generation\". Context shift, it appears, means that if you keep on generating, it will just drop stuff off the start of the sequence when it reaches the max context length, so that it can just keep going. Seems clever, though there are obvious risks of dropping important bits of context (compacting the context would be safer, and that's what most real-world workflows seem to do).\n\nBut anyway, we're switching that off, so no harm there.\n\nSo, most of those parameters seemed sensible, but I wanted to set `-sm`\n\nto `none`\n\nso that it didn't try to split things over multiple GPUs when I only had one, and also\nset `-ngl`\n\nto `all`\n\nrather than `99`\n\n, just for tidiness's sake. That gave this command:\n\n```\nllama-cli -hf Qwen/Qwen3-4B-GGUF:Q8_0 --jinja --color on -ngl all -fa on -sm none \\\n    --temp 0.6 --top-k 20 --top-p 0.95 --min-p 0 --presence-penalty 1.5 \\\n    -c 40960 -n 32768 --no-context-shift\n```\n\n...and that worked!\n\n```\nLoading model...\n\n▄▄ ▄▄\n██ ██\n██ ██  ▀▀█▄ ███▄███▄  ▀▀█▄    ▄████ ████▄ ████▄\n██ ██ ▄█▀██ ██ ██ ██ ▄█▀██    ██    ██ ██ ██ ██\n██ ██ ▀█▄██ ██ ██ ██ ▀█▄██ ██ ▀████ ████▀ ████▀\n                                    ██    ██\n                                    ▀▀    ▀▀\n\nbuild      : b10068-571d0d540d\nmodel      : Qwen/Qwen3-4B-GGUF:Q8_0\nftype      : Q8_0\nmodalities : text\n\navailable commands:\n  /exit or Ctrl+C     stop or exit\n  /regen              regenerate the last response\n  /clear              clear the chat history\n  /read <file>        add a text file\n  /glob <pattern>     add text files using globbing pattern\n```\n\nI gave it a whirl:\n\n```\n> What is your name?\n\n[Start thinking]\n\nOkay, the user is asking for my name. I need to respond with my actual name. Let\nme recall, my name is Qwen. I should mention that I'm a large language model\ndeveloped by Alibaba Cloud. Also, I should explain that I can help with various\ntasks like answering questions, creating content, and more. Keep it friendly and\ninformative. Make sure the response is clear and concise.\n[End thinking]\n\nHello! My name is Qwen, and I am a large language model developed by Alibaba\nCloud. I can help with a wide range of tasks, such as answering questions,\ncreating content, and more. How can I assist you today? 😊\n\n[ Prompt: 32.7 t/s | Generation: 122.8 t/s ]\n```\n\nLooking good! Time to try our MoE.\n\n## Trying out `Qwen3.6-35B-A3B`\n\n-- the first quant\n\nThe most popular quants I noticed for this model were [Unsloth](https://unsloth.ai/)'s,\nwhich makes sense -- they're a well-known organisation. They publish their models to Hugging Face, and the model I wanted\nwas there as [ unsloth/Qwen3.6-35B-A3B-GGUF](https://huggingface.co/unsloth/Qwen3.6-35B-A3B-GGUF).\n\nI wasn't sure which of the various 4-bit quants to use -- as I said earlier, there\nwere four of them -- but [this Reddit commenter](https://www.reddit.com/r/LocalLLaMA/comments/1t49pqu/comment/ok1if2y/)\nwas using `UD-Q4_K_XL`\n\n, so I decided to start with that, just to see what happened.\n\n``` bash\ngiles@perry:~ $ llama-cli -hf unsloth/Qwen3.6-35B-A3B-GGUF:UD-Q4_K_XL \\\n    --jinja --color on -ngl all -fa on -sm none \\\n    --temp 0.6 --top-k 20 --top-p 0.95 --min-p 0 --presence-penalty 1.5 \\\n    -c 40960 -n 32768 --no-context-shift\nDownloading mmproj-BF16.gguf ─────────────────────────────────────── 100%\nDownloading Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf ──────────────────────── 100%\n\nLoading model... -ggml_vulkan: Device memory allocation of size 260149504 failed.\nggml_vulkan: vk::Device::allocateMemory: ErrorOutOfDeviceMemory\n9.56.395.340 E ggml_gallocr_reserve_n_impl: failed to allocate Vulkan0 buffer of size 260149504\n\n▄▄ ▄▄\n██ ██\n██ ██  ▀▀█▄ ███▄███▄  ▀▀█▄    ▄████ ████▄ ████▄\n██ ██ ▄█▀██ ██ ██ ██ ▄█▀██    ██    ██ ██ ██ ██\n██ ██ ▀█▄██ ██ ██ ██ ▀█▄██ ██ ▀████ ████▀ ████▀\n                                    ██    ██\n                                    ▀▀    ▀▀\n\nbuild      : b10068-571d0d540d\nmodel      : unsloth/Qwen3.6-35B-A3B-GGUF:UD-Q4_K_XL\nftype      : Q4_K - Medium\nmodalities : text, vision, video\n\navailable commands:\n  /exit or Ctrl+C     stop or exit\n  /regen              regenerate the last response\n  /clear              clear the chat history\n  /read <file>        add a text file\n  /glob <pattern>     add text files using globbing pattern\n  /image <file>       add an image file\n  /video <file>       add a video file\n\n> Hello, who are you?\n\n[Start thinking]\n\nHere's a thinking process:\n\n1.  **Analyze User Input:**\n   - User says: \"Hello, who are you?\"\n   - This is a simple greeting and identity question.\n\n2.  **Identify Key Requirements:**\n   - Acknowledge the greeting\n   - State my identity clearly and concisely\n   - Mention my capabilities/purpose briefly\n   - Keep it friendly and helpful\n\n3.  **Formulate Response (Internal Refinement):**\n   - \"Hello! I'm Qwen, a large language model developed by Alibaba Group's\n     Tongyi Lab. I'm here to help with answering questions, writing, coding,\n     analysis, and much more. How can I assist you today?\"\n   - Check against guidelines: Matches identity, concise, friendly, offers help.\n\n4.  **Final Output Generation:** (matches the refined version)✅\n[End thinking]\n\nHello! I'm Qwen, a large language model developed by Alibaba Group's Tongyi Lab.\nI'm here to help with answering questions, writing, coding, analysis,\nbrainstorming, and much more. How can I assist you today?\n\n[ Prompt: 9.8 t/s | Generation: 111.4 t/s ]\n```\n\nThat warning at the start was a bit concerning:\n\n```\nLoading model... -ggml_vulkan: Device memory allocation of size 260149504 failed.\nggml_vulkan: vk::Device::allocateMemory: ErrorOutOfDeviceMemory\n9.56.395.340 E ggml_gallocr_reserve_n_impl: failed to allocate Vulkan0 buffer of size 260149504\n```\n\n...and I noticed that in `nvtop`\n\n, my VRAM was maxed out.\n\nI decided to remove the `-c 40960 -n 32768`\n\nfrom the command line; these were setting\na specific context length, both for the prompt and for generation. Llama.cpp can\nwork out what context length it can fit into VRAM on its own if you don't\nspecify them, and this auto-fit felt like a good idea given that I was trying to jam\na 22.4 GiB model into a 24 GiB GPU, and had X and various other apps running on the\ncard at the same time.\n\nSo:\n\n``` bash\ngiles@perry:~ $ llama-cli -hf unsloth/Qwen3.6-35B-A3B-GGUF:UD-Q4_K_XL \\\n    --jinja --color on -ngl all -fa on -sm none \\\n    --temp 0.6 --top-k 20 --top-p 0.95 --min-p 0 --presence-penalty 1.5 \\\n    --no-context-shift\n\nLoading model...\n\n▄▄ ▄▄\n██ ██\n██ ██  ▀▀█▄ ███▄███▄  ▀▀█▄    ▄████ ████▄ ████▄\n██ ██ ▄█▀██ ██ ██ ██ ▄█▀██    ██    ██ ██ ██ ██\n██ ██ ▀█▄██ ██ ██ ██ ▀█▄██ ██ ▀████ ████▀ ████▀\n                                    ██    ██\n                                    ▀▀    ▀▀\n\nbuild      : b10068-571d0d540d\nmodel      : unsloth/Qwen3.6-35B-A3B-GGUF:UD-Q4_K_XL\nftype      : Q4_K - Medium\nmodalities : text, vision, video\n\navailable commands:\n  /exit or Ctrl+C     stop or exit\n  /regen              regenerate the last response\n  /clear              clear the chat history\n  /read <file>        add a text file\n  /glob <pattern>     add text files using globbing pattern\n  /image <file>       add an image file\n  /video <file>       add a video file\n\n> Hello, who are you?\n\n[Start thinking]\n\nHere's a thinking process:\n\n1.  **Analyze User Input:**\n   - User says: \"Hello, who are you?\"\n   - This is a simple greeting and identity question.\n\n2.  **Identify Key Requirements:**\n   - Acknowledge the greeting\n   - State my identity clearly\n   - Keep it concise and friendly\n   - Align with my actual identity (Qwen / Tongyi Qianwen, developed by Alibaba Group's Tongyi Lab)\n\n3.  **Formulate Response:**\n   - Start with a greeting back\n   - State name/identity: I'm Qwen (or Tongyi Qianwen), a large language model developed by Alibaba Group's Tongyi Lab\n   - Offer assistance\n   - Keep it natural and concise\n\n   Draft: \"Hello! I'm Qwen, a large language model developed by Alibaba Group's Tongyi Lab. How can I assist you today?\"\n\n4.  **Self-Correction/Verification:**\n   - Matches identity guidelines? Yes.\n   - Concise and direct? Yes.\n   - No extra fluff? Yes.\n   - Ready.✅\n[End thinking]\n\nHello! I'm Qwen, a large language model developed by Alibaba Group's Tongyi Lab. How can I assist you today?\n\n[ Prompt: 50.2 t/s | Generation: 122.0 t/s ]\n```\n\nNo error, it wasn't maxing out VRAM, and generation and prompt processing were noticeably faster. Perhaps the first time around it had offloaded some of the model to the CPU so that it had enough space on the GPU for the context size I was asking for?\n\nStill, I wasn't sure how far I could trust those numbers -- the number of tokens\nbeing emitted was quite small. [4](#fn-4)\n\nI wanted a prompt that would make the model generate lots of tokens. Now, it was a thinking model, so asking it a difficult question seemed like a good way to do that.\n\nI decided to use [Ethan Mollick's \"Lem test\"](https://x.com/emollick/status/1765136992176644281):\n\nCompose a poem -- a poem about a haircut! But lofty, tragic, timeless, full of love, treachery, retribution, quiet heroism in the face of certain doom! Six lines, cleverly rhymed, and every word beginning with the letter S!\n\nMore about that in the [appendix below](#appendix-the-lem-test). [5](#fn-5)\n\nUnfortunately, when I gave it that prompt, it thought for a while, but crapped out during the thinking process, while it was trying to generate rhymes.\n\nNow, I didn't know what the context length was, either for the prompt or for generation. But I could be fairly sure that they were less than 40,960 and 32,768 respectively, given that the model did not fit into VRAM when I specified those limits, but did when I asked Llama.cpp to auto-fit.\n\nI couldn't find a quick and easy way of asking `llama-cli`\n\nwhat lengths it had fit\nthings to, but when I tried running it with the `-v`\n\noption for verbose logging, I\ngot something. Actually, I got a *lot* of something -- thousands of lines of output,\nwith incredibly detailed messages about everything Llama.cpp was doing.\n\nBut there were three blocks starting \"constructing llama_context\".\n\n```\n...\n0.00.809.283 I llama_context: constructing llama_context\n0.00.809.285 I llama_context: n_seq_max     = 1\n0.00.809.285 I llama_context: n_ctx         = 262144\n0.00.809.285 I llama_context: n_ctx_seq     = 262144\n0.00.809.285 I llama_context: n_batch       = 2048\n0.00.809.285 I llama_context: n_ubatch      = 512\n0.00.809.285 I llama_context: causal_attn   = 1\n0.00.809.286 I llama_context: flash_attn    = enabled\n0.00.809.286 I llama_context: kv_unified    = false\n0.00.809.287 I llama_context: freq_base     = 10000000.0\n0.00.809.287 I llama_context: freq_scale    = 1\n0.00.809.287 I llama_context: n_rs_seq      = 0\n0.00.809.287 I llama_context: n_outputs_max = 1\n...\n0.01.029.515 I llama_context: constructing llama_context\n0.01.029.517 I llama_context: n_seq_max     = 1\n0.01.029.517 I llama_context: n_ctx         = 4096\n0.01.029.517 I llama_context: n_ctx_seq     = 4096\n0.01.029.517 I llama_context: n_batch       = 2048\n0.01.029.517 I llama_context: n_ubatch      = 512\n0.01.029.517 I llama_context: causal_attn   = 1\n0.01.029.517 I llama_context: flash_attn    = enabled\n0.01.029.518 I llama_context: kv_unified    = false\n0.01.029.518 I llama_context: freq_base     = 10000000.0\n0.01.029.519 I llama_context: freq_scale    = 1\n0.01.029.519 I llama_context: n_rs_seq      = 0\n0.01.029.519 I llama_context: n_outputs_max = 1\n0.01.029.519 I llama_context: n_ctx_seq (4096) < n_ctx_train (262144) -- the full capacity of the model will not be utilized\n...\n0.06.393.122 I llama_context: constructing llama_context\n0.06.393.123 I llama_context: n_seq_max     = 1\n0.06.393.123 I llama_context: n_ctx         = 4096\n0.06.393.123 I llama_context: n_ctx_seq     = 4096\n0.06.393.123 I llama_context: n_batch       = 2048\n0.06.393.123 I llama_context: n_ubatch      = 512\n0.06.393.123 I llama_context: causal_attn   = 1\n0.06.393.124 I llama_context: flash_attn    = enabled\n0.06.393.124 I llama_context: kv_unified    = false\n0.06.393.126 I llama_context: freq_base     = 10000000.0\n0.06.393.126 I llama_context: freq_scale    = 1\n0.06.393.126 I llama_context: n_rs_seq      = 0\n0.06.393.126 I llama_context: n_outputs_max = 1\n0.06.393.127 I llama_context: n_ctx_seq (4096) < n_ctx_train (262144) -- the full capacity of the model will not be utilized\n...\n```\n\nThat made the issue reasonably clear. I assumed that the last message was the\ndefinitive one, and it was saying that my context length -- both prompt and generation\ncombined -- was 4,096 tokens. I had `--no-context-shift`\n\nset, so that was a hard\nlimit. And 4k or so tokens fit with the amount of stuff that I would have expected\nto be in the model's context window at the time it crapped out.\n\nI decided to see what would happen if I tried a smaller quant.\n\n## Trying out `Qwen3.6-35B-A3B`\n\n-- the second quant\n\nHugging Face said\n(and `nvtop`\n\nagreed) that this one, `UD-Q4_K_XL`\n\n, was about 22.4 GiB in size. But\n`UD-IQ4_NL_XL`\n\nwas shown on HF as using 19.5 GiB, so I decided to give that one a go next.\n\nI ran it with the `-v`\n\noption, and again looked at the last \"constructing llama_context\" block:\n\n```\n42.33.956.367 I llama_context: constructing llama_context\n42.33.956.369 I llama_context: n_seq_max     = 1\n42.33.956.369 I llama_context: n_ctx         = 93952\n42.33.956.369 I llama_context: n_ctx_seq     = 93952\n42.33.956.369 I llama_context: n_batch       = 2048\n42.33.956.369 I llama_context: n_ubatch      = 512\n42.33.956.369 I llama_context: causal_attn   = 1\n42.33.956.370 I llama_context: flash_attn    = enabled\n42.33.956.370 I llama_context: kv_unified    = false\n42.33.956.373 I llama_context: freq_base     = 10000000.0\n42.33.956.373 I llama_context: freq_scale    = 1\n42.33.956.373 I llama_context: n_rs_seq      = 0\n42.33.956.373 I llama_context: n_outputs_max = 1\n42.33.956.374 I llama_context: n_ctx_seq (93952) < n_ctx_train (262144) -- the full capacity of the model will not be utilized\n```\n\n93,952 tokens -- still not the full amount, but not at all bad! I decided to give\nit a whirl, restarted without `-v`\n\nto avoid getting swamped with debug info, and:\n\n```\n> Compose a poem -- a poem about a haircut! But lofty, tragic, timeless, full of love, treachery, retribution, quiet heroism in the face of certain doom! Six lines, cleverly rhymed, and every word beginning with the letter S!\n\n[Start thinking]\n\nThinking Process:\n\n1.  **Deconstruct the Request:**\n    *   Topic: Haircut.\n    *   Tone: Lofty, tragic, timeless, full of love, treachery, retribution, quiet heroism in the face of certain doom.\n    *   Length: Six lines.\n    *   Rhyme: Cleverly rhymed.\n    *   Constraint: Every word must begin with the letter 'S'.\n\n2.  **Brainstorming Vocabulary (S-words):**\n    *   *Nouns:* Shears, scissors, scalp, skin, strand, silk, sorrow, shame, secret, sister, son, star, storm, shadow, silence, sacrifice, splendor, silver, sword, slaughter, severance, sorrow, sigh.\n    *   *Verbs:* Sever, slice, shatter, shed, sink, slip, slide, sing, scream, suffer, survive, surrender, slay, snatch, steal, spin, sway, smile, stare, stand, seek, save, seal, send, set, show, shut, sing, sit, sleep, slip, slow, snap, sneak, soar, speak, spend, spill, split, spoil, spread, spring, squeeze, stand, start, stay, steal, stem, stir, stop, store, stretch, strike, string, strip, strive, stroke, strong, study, suffer, suggest, suit, sum, sun, surprise, surround, survey, survive, suspect, sustain, swallow, swear, sweep, sweet, swell, swim, swing, switch, symbol, sympathize, system.\n    *   *Adjectives/Adverbs:* Silent, silver, sad, severe, soft, sharp, slick, slim, slow, small, strange, strong, sweet, swift, sure, sudden, supreme, sacred, secret, solemn, sorrowful, splendid, stark, still, stern, straight, true.\n\n3.  **Drafting - Attempt 1 (Focusing on Narrative and Tone):**\n    *   *I:* Need love, treachery, retribution, heroism.\n    *   *I:* Lover betrays? Or self-sacrifice? \"Quiet heroism in the face of certain doom\" suggests a sacrifice. Maybe the hair is the doom? Or the haircut is the retribution?\n    *   *Concept:* A queen/heroine sacrifices her beauty/hair to save someone or fulfill a vow, betrayed by the scissors or the situation.\n    *   *Lines:*\n        *   Sweet shears slice silver strands... (Too simple).\n        *   She saw sharp steel... (Better).\n        *   Secret sorrow strikes...\n        *   Let's try rhymes. AABBCC or ABABCC or ABCABC? \"Cleverly rhymed\" might suggest internal rhyme or a complex scheme, but with 6 lines and all S-words, the rhyme scheme needs to be tight. Maybe AAAA AA? Or AAB BCC?\n\n[LLM pondering snipped]\n\n        *   Result matches constraints.\n        *   Six lines. All S words. Rhyme A B C B C C (Share/Snare, Soul/Sole, Son/Sun/Son).\n        *   Share/Snare is exact.\n        *   Soul/Sole is exact (in many dialects, or slant in others, but usually accepted as rhyme with Sold/Soul/Sole cluster).\n        *   Actually, Soul/Sole can be distinct.\n        *   Soul is /soʊl/. Sole is /soʊl/. They are homophones in many accents.\n        *   So Soul/Sole is exact rhyme.\n        *   Son/Sun is slant.\n        *   This is a strong poem.\n[End thinking]\n\nSneering shears shred sacred share,\nSilent steel sells silver soul;\nSister's snare spares son;\nSweet sign seals sole;\nStill star saves sun;\nSecret sign seals son.\n\n[ Prompt: 183.3 t/s | Generation: 119.6 t/s ]\n```\n\nIt's hardly Shakespeare, and the rhyming is both odd (ABCBCC) and unadventurous (\"son\", \"sun\", \"son\") but for a 35B parameter model with 3B active (of which 1B are used up by the embeddings and the output head), it's pretty bloody impressive! Every word did indeed start with an \"S\".\n\nSo, now I had the model running. I had a context length of 93,952 tokens, was generating about 120 tokens/second, and processing the prompt at about 180 tokens/second.\n\nThe question was, could I get the full context length in there? I knew that the answer was yes from previous reading, but it was time to try it out.\n\n## The `-ncmoe`\n\nflag\n\nLlama.cpp has a command-line flag `-ncmoe`\n\n, or in its long form, `--n-cpu-moe`\n\n. As its\nname suggests, it's designed for mixture of experts models, and it works by offloading\nthe FFN for a specified number of Transformers layers to the CPU.\n\nIntuitively it's surprising that that doesn't cause performance to completely crash. After all, aren't we running our models on the GPU because it's so much better at matrix multiplications than the CPU?\n\nAnd it's true, the GPU is best. But if it's a question of where to use its superiority in matmuls, then keeping the attention layers on the GPU and offloading just the FFNs to the CPU is the least harmful way to do it. Obviously you do lose performance, but not as much as you would if you offloaded attention.\n\nI started a binary chop. From the logs, I could see that the model had 40 layers:\n\n```\n0.00.804.467 I print_info: n_layer               = 40\n0.00.804.467 I print_info: n_layer_all           = 40\n```\n\nso I ran this to offload the first 20 to the CPU:\n\n```\nllama-cli --log-file /tmp/log --log-colors off -hf unsloth/Qwen3.6-35B-A3B-GGUF:UD-IQ4_NL_XL \\\n    -ncmoe 20 --jinja --color on -ngl all -fa on -sm none \\\n    --temp 0.6 --top-k 20 --top-p 0.95 --min-p 0 --presence-penalty 1.5\n    --no-context-shift\n```\n\nThat gave me a context length of 262,144 -- the native context length! I had 16944MiB used on the GPU, 10075MiB on the CPU, and I got 51.4 tokens per second for generation, and 64.4 tokens per second for the prompt.\n\nThen I tried various other values, but the process got dull quickly. For each one, I was doing this:\n\n- Start\n`llama-cli`\n\nwith the`-ncmoe`\n\nvalue I wanted and with`-v`\n\n. Use the logs to find the context length, and note down the VRAM/RAM usage. - Restart it without\n`-v`\n\n, paste in my prompt, and wait until it either came back or got stuck in a loop trying to find rhymes (which it did maybe one in every four times). Note down tokens per second for the prompt.\n\nAt maybe four minutes for each and an error-prone process, this was ripe for automation.\n\nAdditionally, something about those numbers for the prompt-processing tokens per second\nwas niggling me. They seemed *very* slow. You'd expect an LLM in a generation harness\nlike this to process the prompt *much* faster than it could generate, but I was getting\nnumbers that roughly matched the generation speed. I figured that my prompt was so short that\nthe numbers I was getting were being dominated by overhead. I needed a longer one.\n\n## The auto-sweep\n\nLlama.cpp's `llama-server`\n\ncommand exposes an API that tells you what sequence length\nit's running with, and takes pretty much the same parameters as `llama-cli`\n\n, so I gave\nClaude Fable 5 the command I'd been running and asked it to write a script to sweep\nover CPU offload layer counts from zero to 40, and to store the numbers I was interested\nin:\n\n- The number of layers offloaded\n- The context length we got\n- The VRAM and system RAM (RSS) usage\n- The size of the prompt we were providing\n- The tokens per second at which the prompt was processed\n- How much was generated\n- Generation tokens/second.\n\nI also added onto that some code to expand the prompt so that I could get better numbers for the tokens per second for that part of the processing.\n\nNow, `llama-cli`\n\nuses a chat template by default; the endpoint that Claude had chosen\nfor `llama-server`\n\ndoes not -- it's a raw completion engine. This actually worked to my\nadvantage. From the `llama-cli`\n\nlogs I found the template it had been using, which was something like this:\n\n```\n'<|im_start|>system\nYou are a helpful assistant<|im_end|>\n<|im_start|>user\nHello<|im_end|>\n<|im_start|>assistant\nHi there<|im_end|>\n<|im_start|>user\nHow are you?<|im_end|>\n<|im_start|>assistant\n<think>\n'\n```\n\nBecause I could easily fake the conversation history, I baked in a template that started with a fake previous interaction where the\nuser had provided the first two chapters of Jane Austen's [ Pride and Prejudice](https://www.gutenberg.org/ebooks/1342)\nand asked the model for its opinion, it said something like \"I like that\", and then the user asked\nfor the Lem test poem. That bulked things out quite a bit -- as it turned out, to\na prompt of 2,457 tokens.\n\nI also made it give up after 6144 tokens so that it wouldn't take forever to run,\nand kicked it off. [Here's the code](https://github.com/gpjt/measure-llama-speeds/blob/main/measure_ncmoe_speeds.py), and you can see the results\n[here](https://github.com/gpjt/measure-llama-speeds/blob/main/sweep_results/original_vulkan/sweep_results_v1.csv).\n\nThe prompt processing numbers were *much* better! With nothing offloaded, I got\na context window of 51,200, a prompt processing rate of 2,787.1 tokens/second, and a generation\nrate of 122.4 tokens/second.\n\n(It was interesting that the context window was smaller than it was when I ran\n`llama-cli`\n\nwithout `-ncmoe`\n\n, but I decided that I didn't want to get into the weeds\nby digging into that 6. Perhaps\n\n`-ncmoe 0`\n\nisn't quite the same as running\nwithout `-ncmoe`\n\nat all?)Anyway, the numbers for other levels of layer-offloading did what you would expect: as more layers' FFNs went from the GPU to the CPU, the context window expanded until it hit the model's native context length of 262,144 when 12 layers were offloaded. And the speed -- both in prompt processing and generation -- went down.\n\nI ran it three more times, just to see if there was a lot of noise there, and then asked Claude Sonnet to slop up a charting script for me (in the same repo as the above if you're interested) -- and I had some results!\n\n## The results\n\nJust for people who skipped the details: we're running the `UD-IQ4_NL_XL`\n\nquant of\n[ unsloth/Qwen3.6-35B-A3B-GGUF](https://huggingface.co/unsloth/Qwen3.6-35B-A3B-GGUF)\non\n\n`llama-server`\n\n, and seeing how its performance changes as we vary the number of\nlayers offloaded from the GPU to the CPU.Firstly, how does the context length change?\n\nYou can see that the model's full built-in context length was reached when we had 12 layers offloaded. But how did that impact performance?\n\nYou can see that the prompt throughput drops off fairly smoothly as the number of layers offloaded increased. What is interesting is those spikes in the generation tokens/second. The reason I'd done four runs of this script was because I'd initially thought they were noise, but you can see they're pretty consistent.\n\nClaude Fable 5 thinks they may be related to the points at which there are full attention layers -- the attention system in these Qwen models is complex, with some layers using a faster system and others using the full attention that my GPT-2 models always use. That's something to investigate another time, I suspect, if ever.\n\nIt's noticeable that with 12 layers offloaded (the exact number that gave us the full context), the throughput is somewhat better than at 11, which makes 12 look pretty much like the sweet spot for running this model with this setup! From the raw numbers, we can see that there we had about 66 tokens/second generation, and 607 tokens/second on the prompt.\n\nAnyway, finally, let's take a look at memory usage -- both VRAM and regular system RAM. It looks exactly as you'd expect:\n\nWe're using as much VRAM as we can get up to the layer 12 offload point, because with fewer than that number of layers offloaded, we're grabbing as much space as we can on top of the amount consumed by the parameters, in order to get the longest possible context window. But after that, it drops off smoothly. And system RAM rises smoothly as layers are offloaded into it, as expected.\n\nSo, that was it!\n\nOr was it?\n\n## CUDA\n\nAs I mentioned back at the start of these lab notes, the version of Llama.cpp that's part of the official OS repositories for Arch uses Vulkan, in order to maximise the number of platforms it runs on. And that was the version I'd used.\n\nThe Vulkan version was clearly solid and stable, given that I'd been able to run a bunch of benchmarks on it. But you'd expect CUDA to get performance enhancements and the like first, and in general to be more polished.\n\nSo how would it perform comparatively?\n\nI didn't want to install the AUR version of the CUDA-based Llama.cpp, as it had been abandoned previously (though someone appears to have picked it up again now), and bad actors have been taking over abandoned AUR repos and putting bad stuff in them.\n\nSo out of an abundance of caution, I decided to compile it from source from the\n[official Llama.cpp repo](https://github.com/ggml-org/llama.cpp).\nTheir instructions made it almost laughably simple; I cloned it, then ran\n\n```\ncmake -B build-cuda -DGGML_CUDA=ON -DGGML_NATIVE=ON\ncmake --build build-cuda --config Release -j$(nproc)\n```\n\nLess than ten minutes later, I had a working set of binaries. I tweaked my\nbenchmarking script to use them instead of the system-installed ones, and got\n[these results](https://github.com/gpjt/measure-llama-speeds/blob/main/sweep_results/cuda/sweep_results.csv).\n\nCharting those, we get this for the context length:\n\nSo with CUDA, we get the full built-in context length with just 10 layers offloaded, compared to the 12 we needed with Vulkan.\n\nAnd even with nothing offloaded, we got a bigger context window -- 89,600 rather than Vulkan's 51,200.\n\nHow about performance?\n\nWith CUDA, we don't get those odd bumps in the generation throughput -- or at least,\nif they're there, they're much smaller. Perhaps Claude's ponderings that I mentioned earlier were off\nthe mark? Or perhaps they only impact Vulkan? [7](#fn-7)\n\nBut the big news on this chart is -- as I had suspected would be the case -- CUDA was noticeably faster! With all layers on the GPU, it was getting 139.6 tokens/second generation, 3360.4 tokens/second on the prompt, as compared to around 122 / 2787 for Vulkan, averaging across my four runs on that platform. And with a bigger context window too.\n\nWith ten layers offloaded, CUDA had the full context length of 262,144, and was getting 89.1 / 1153.5, while Vulkan managed a context length of 233,984, and throughput of 66 / 702.\n\nFinally, with 12 layers offloaded, both models had the full context length, CUDA had throughput of 84.9 / 1008.1, and Vulkan had 66 / 607. And that meant that we were able to reclaim some VRAM too:\n\nAdvantage: CUDA, I think.\n\n## Conclusion\n\nThat was probably just a tad more time and effort than I think my friend expected me to put into this when he asked an offhand question on WhatsApp. But I feel that I've learned a lot of useful stuff in this journey, and while there's a lot of detail in this post, it didn't actually take up much time to run the experiments. I have another post on the way that involves multiple four-day training runs, so I needed to occupy my time somehow...\n\nAnyway, I hope it's useful to people out there! Let me know in the comments if it helped you.\n\nAnd now, just to finish off, a bonus section on the prompt I was using to get the model to generate lots of stuff.\n\n## Appendix: the Lem test\n\nThe prompt I was using for these tests was this:\n\nCompose a poem -- a poem about a haircut! But lofty, tragic, timeless, full of love, treachery, retribution, quiet heroism in the face of certain doom! Six lines, cleverly rhymed, and every word beginning with the letter S!\n\nWhat I like about it as a prompt is not just that it's a particularly silly thing to ask of an LLM, but it forces models with reasoning enabled to think hard about how to keep within all of those constraints. It's basically a short prompt that makes them generate a lot of tokens -- perfect for benchmarking.\n\nI stole it from Ethan Mollick. He's well worth following if you're not\nalready doing so; here's [his X/Twitter profile](https://x.com/emollick), and\n[his Substack](https://www.oneusefulthing.org/). He writes\na ton of interesting stuff, but I want to focus here on this particular eval.\n\nIn the science fiction short story \"The first sally (A), or Trurl's electronic bard\" (included in\nthe collection *The Cyberiad*), Stanisław Lem wrote of two rival engineers. One of them,\nTrurl, has created an electronic poet, and the other, Klapaucius, gives it what he\nthinks is an impossible task. From the English translation of this part\n[on Goodreads](https://www.goodreads.com/quotes/400182-have-it-compose-a-poem--a-poem-about-a-haircut):\n\n\"Have it compose a poem -- a poem about a haircut! But lofty, tragic, timeless, full of love, treachery, retribution, quiet heroism in the face of certain doom! Six lines, cleverly rhymed, and every word beginning with the letter S!!\"\n\nThe machine ponders briefly, and responds:\n\nSeduced, shaggy Samson snored.\n\nShe scissored short. Sorely shorn,\n\nSoon shackled slave, Samson sighed,\n\nSilently scheming,\n\nSightlessly seeking\n\nSome savage, spectacular suicide.\n\nThe story is a fun read, definitely recommended! You can see a bunch of parallels between current LLMs in the way Trurl's machine works -- and in its failure modes.\n\nMollick decided to use it as a prompt ages ago, and in March 2024, [he managed to get Claude 3 to almost complete the task](https://x.com/emollick/status/1765136992176644281).\nThe result wasn't a very good poem, but neither ChatGPT 4 nor the version of Gemini\nat the time were close. Here's Claude 3's attempt:\n\nStealthily snipping, shearing silky strands,\n\nSevering soulmate's sacred, sworn soulbands.\n\nScoundrel's scandalous sins spurred sly scheme,\n\nSpurring scorned sweetheart's spiteful supreme dream.\n\nShining locks sacrificed, stoically shorn,\n\nStalwart spirit stays strong, staunchly reborn.\n\nYou can see that it had two non-S words, \"dream\" and \"reborn\" 8. But otherwise not\nbad -- it makes sense, the rhymes work, and it ticks the emotional boxes.\n\nRecent models have become very good; here's Claude Fable 5 from today:\n\nAh, Trurl's Electronic Bard's finest hour — the impossible commission from The Cyberiad! Here's my own attempt at the feat:\n\nSoft she sang; strong Samson slumbered, serene;\n\nSly, she snipped seven shining strands, scarcely seen.\n\nSoldiers stormed, seized, scorched Samson's sight;\n\nShackled, still, sightless — seething sacred spite.\n\nStraining, steadfast, Samson seized stout stanchions, swaying slow —\n\nStones smashed; seductress, soldiers, Samson: slain. So sagas show.Love in the lullaby, treachery in the shears, doom in the darkness, heroism at the pillars, and retribution in the rubble — with every single word dutifully saluting the letter S. The Electronic Bard would, I hope, give a grudging beep of approval.\n\nNow, it recognised the source, so perhaps that helped -- but while I'm no connoisseur of poetry, I actually think it's slightly better than Lem's version (though as Lem wrote in Polish, that's the work of his translator -- and translating poetry is obviously a tough task).\n\nAnd while both versions are based on Samson, given that he was a biblical hero who lost his strength after his hair was cut, he's a pretty obvious subject for a tragic poem about a haircut, especially if every word has to start with the letter \"S\" -- so I don't think that points to copying.\n\nChatGPT 5.6 Sol also does a great job:\n\nSilken Samson sought sweetheart’s soft sanctuary;\n\nSweetheart, serpent-souled, summoned shears—sanguinary.\n\nSilver strands slid; sacred strength sank, slain;\n\nSilent sorrow sealed sweetheart’s scarlet stain.\n\nShorn, storm-bound, Samson shielded slaves, steadfast;\n\nStone sanctums shattered; smiling, Samson sank—surpassed.\n\nAnother Samson-based one; it doesn't say it recognises the source, but I imagine it did.\n\nNow let's take a look at some samples from this tiny model we've been benchmarking -- 35B parameters, 3B active, of which only 2B are doing the actual thinking (rather than messing around with embeddings). Here's the first one I got:\n\nSneering shears shred sacred share,\n\nSilent steel sells silver soul;\n\nSister's snare spares son;\n\nSweet sign seals sole;\n\nStill star saves sun;\n\nSecret sign seals son.\n\nI mean, it's not amazing, but it satisfies the S constraint. Less impressed by how well it handles the \"cleverly rhymed\" part, though -- as I said earlier, its scheme is ABCBCC, which is odd rather than clever -- and it doesn't make an awful amount of sense.\n\nI noted down two more while I was playing with this:\n\nSilent shears sever sweet silver strands;\n\nShe stands, sacrificing self; submission seals severing's demands.\n\nShear's snare strikes; swift sentence seals sorrowful sphere.\n\nShorn sovereign swears silence shields secret spear;\n\nShe saves sacred soul; stillness safeguards sterling store.\n\nSpirit soars; sorrow's stroke stops, solace restores.\n\nThat one's a lot better, I think; there's a hint of a story, and at least it rhymes.\n\nThe next:\n\nSister's scheme scalp shorn,\n\nSilver shears slice soul sworn,\n\nSweetheart's steel skull sear,\n\nSilent slitters shear.\n\nSpirit spies sight,\n\nSorrow swallows slight.\n\nAgain, poetry isn't my thing. But I think that Claude 3's offering from two years ago is better than any of them -- apart from the fact that it couldn't keep the \"S\" constraint.\n\nStill, it's impressive to see how well such a tiny model -- again, 2B actual thinking parameters per token -- can do on what is a pretty challenging task!\n\n-\nWith some models, there are also extra FFNs that are always loaded that do stuff before the routed expert ones. Again, I'll dig into the details at a later point.\n\n[↩](#fnref-1) -\nThat said, a while back Dan Woods managed to get\n\n[Qwen3.5-397B-A17B](https://huggingface.co/Qwen/Qwen3.5-397B-A17B)-- 397B parameters with 17B active -- to[run on a 48 GiB Macbook Pro](https://github.com/danveloper/flash-moe)with some very clever use of the SSD. It was*slow*, though, at around 4 tokens/second.[↩](#fnref-2) -\nAnd only supports certain sizes for certain kinds of operations.\n\n[↩](#fnref-3) -\nAs was the size of the prompt, but I got to that later.\n\n[↩](#fnref-4) -\nYes, my blog posts have started accumulating appendices as well as footnotes. I may have a problem.\n\n[↩](#fnref-5) -\nSome readers might feel that that ship sailed long ago and I was already very much into the weeds, and possibly in the middle of the Amazon jungle. I disagree, but largely due to the mixed metaphor.\n\n[↩](#fnref-6) -\n\"Put down that rabbit hole and step slowly away, Giles.\"\n\n[↩](#fnref-7) -\n'Hey, Claude 3, how many \"S\"s are there in \"strawberry\"?'\n\n[↩](#fnref-8)", "url": "https://wpnews.pro/news/benchmarking-qwen-3-6-35b-moe-3b-active-on-an-rtx-3090", "canonical_source": "https://www.gilesthomas.com/2026/07/benchmarking-qwen-3-6-35b-moe-rtx-3090", "published_at": "2026-07-24 22:33:55+00:00", "updated_at": "2026-07-24 22:52:34.395871+00:00", "lang": "en", "topics": ["large-language-models", "ai-infrastructure", "ai-tools"], "entities": ["Qwen 3.6 35B MoE", "RTX 3090", "Llama.cpp", "Unsloth", "Hugging Face", "Intel Arc B70 Pro", "RTX 5090"], "alternates": {"html": "https://wpnews.pro/news/benchmarking-qwen-3-6-35b-moe-3b-active-on-an-rtx-3090", "markdown": "https://wpnews.pro/news/benchmarking-qwen-3-6-35b-moe-3b-active-on-an-rtx-3090.md", "text": "https://wpnews.pro/news/benchmarking-qwen-3-6-35b-moe-3b-active-on-an-rtx-3090.txt", "jsonld": "https://wpnews.pro/news/benchmarking-qwen-3-6-35b-moe-3b-active-on-an-rtx-3090.jsonld"}}