# Running qwen 3.6 / 3.8 on 3090+3080 over RPC? > Source: > Published: 2026-08-26 14:12:47+00:00 Hi I am trying to setup a local model machine. I just got qwen 3 coder 30B A3B and qwen 3.6 27B and 3.8 27B running on my LLM box machine. I’m new to this and it seems to run okay-ish after applying someone’s advice to quantize and err do something. Lol. I don’t know how it works. He said it’s new but I could add a second 3090 in my T5810 and run them together over RPC. I am on a budget so I was thinking of getting a 3080 10GB to put in my server and perhaps it would allow for more context for my qwen’s as it keeps running out. Also I really don’t know what I’m doing and probably haven’t configured anything properly yet. I think I get about 70 tokens a second running Qwen 3.8 27B at the moment. Old mate says he gets 70~ tokens a second too on his 3090. It just seems misconfigured and slow. Going to have a second look around the forums for guides on how to configure this while I wait for responses on it. But I thought it was worth asking about dual boxes and RPC config. My LLM box is 7800X3D, 64GB DDR5, 3090 24GB. Running ubuntu and ollama. My server is a 14 core intel, 32GB DDR4, 1650 4GB ( at the moment) 1 Like Running a model distributed over RPC is a feature of llama.cpp as far as I know and buggy, because the developers often don’t pay that much attention if RPC works correctly with a model. Additionally you need a really fast network connection to get any usable speeds out of this. 10Gbits absolute minimum, probably you’d need more then 40Gbits to use it well. However you are on Linux and you do have a 3090 already. I think the best experience here is to use the ninfer for for the 3090. ninfer is a small inference engine, like llama.cpp or ollama but specifically made for Nvidia Blackwell and then Qwen models. Someone did a fork of that and optimized it for the 3090. This should give you by far the best speed you can achive on a 3090. Other then that, if you want to get another GPU the most plausible scenario would be if you put it into the same computer as the 3090 you already got. 1 Like In Ollama, set the num_gpu parameter or use the num-gpu command-line option to force the model to use multiple GPUs, and configure distributed inference using rpc. Oh wow. Thank you dude. I’m going to use this ASAP. Sorry if this is a dumb question. I’m going to read that after I get my server running again just excited. Can this optimiser thing work on other local models for like 3d model generation etc? 1 Like Ninfer is basically only for the Qwen model line. Other models won’t work with it, without heavy modification. For general use llama.cpp is still the way to go. I wonder how much better it is. I get 35-40 tokens a second on 3.8. definitely put that ninfer thing on later. 1 Like I use it on my Blackwell card, so I have no experience with the fork for the 3090. But trying it is definitely worth a shot. 1 Like I got ninfer to run and it’s roughly double the tokens per second. (From 35-40 up to 55-60) Qwen 3.8 27B on Ninfer runs fantastically. It’s soo good. I’m going to run it on my server and have it automate everything. I couldn’t find an abliterated qwen 3.8 though. How hard would it be for me to make my own abliterated version for this ninfer? Very hard I’m guessing? 1 Like Sorry that is a question I am not knowledgeable on. Certainly people create these ninfer based quants on their own, so you might look up how they do it and then quantize an existing abliterated model maybe?