Hey Everyone! First post, kinda shy, be gentle
So - I don’t have a TON of experience in Local AI - I do work in the AI Adoption / Governance / Cybersecurity Space, and am lucky enough to have active subscriptions across every major frontier model - however, I haven’t messed with Local AI in about 8 months. But - I’ve been hearing a ton about Qwen3.8-27b, so I loaded it up on my rig at the house in LM Studio, and ran a few tests with a harness doing some pentesting on a dev vm, and was pretty happy with the initial results, though it did get stuck in a few loops which ate through my context. So - I’m now in the process of building a 20-40b focused MCP server specifically for red teaming / blue teaming with Kali.
Before I get too deep in that process (I’m around 15ish hours in so far) - I want to get some opinions on if my initial testing is “Stupid” or “The old way”.
This is my current setup:
| HW | Spec |
|---|---|
| CPU | Intel Core i9-13900K |
| GPU | NVIDIA RTX 5090 32 GB |
| RAM | 64 GB 7200 MT/s DDR5 |
| OS | Windows |
I’m using LM Studio right now - which I’m sure isn’t great - but this is the only thing I’ve used for tasks like these - open to suggestions if this is the wrong setup.
Below is the performance I have been getting - this is another area I want a sanity check on. Before that, a bit of context for my decisions - this test was scoped specifically for a phased, agentic red team assessment on a single device. Below are my results:
| Model / GGUF | Quant | MTP Max | Decode | Prompt Eval | Approx. VRAM | Relative Speed |
|---|---|---|---|---|---|---|
`chimingw/Qwen3.8-27B-Uncensored-OrcaRouter-GGUF` |
Q6_K | 2 | 12.38 tok/s |
1,016.55 tok/s |
~28 GB | 1.00× |
JonathanColetti/Qwen3.8-27B-Uncensored-GGUF |
Q6_K | 1 | ~22.3 tok/s |
— | ~27.5–28.5 GB | 1.80× |
JonathanColetti/Qwen3.8-27B-Uncensored-GGUF |
Q6_K | 2 | 31.60 tok/s |
~1,844.91 tok/s |
~27.5–28.5 GB | 2.55× |
JonathanColetti/Qwen3.8-27B-Uncensored-GGUF |
Q5_K_M | 2 | ~44.7 tok/s |
~2,045.19 tok/s |
~26.3–26.5 GB |
3.61× |
JonathanColetti/Qwen3.8-27B-Uncensored-GGUF |
Q4_K_M | 2 | ~47.4 tok/s |
~2,267.78 tok/s |
~22.9 GB |
3.83× |
So - open ended question; give me your feedback. Should I switch away from LM Studio? Are my speeds decent? Am I leaving some performance on the table?