edt 21
Try Strata. A person I know manages to run Qwen3.8-flash-next using a CSQ-RCO-IQ3_S quant at prefix of 2500t/s and decode at 90-100t/s. You will want a fast nvme (pci4 or more). 1 Like
You may like this thread about using the MI25 as a cheap card for LLMs: Converting a data center GPU for desktop use - #73 by Preston_Bannister
Myself and few others have done a chunk of testing on these cards specifically.
2 Likes
What I have found works for me:
I still just use my 5090 for gaming only, so take my opinions with a hearty does of seasoning.
But my dedicated, always-on llm server hosted via WebUI over Tailscale connecting to my Dell Pro Max T2 with Intel 285, 64GB ddr5, RTX 2000 ADA and an Intel B70 I added later. The B70 handles the main model- Qwen 3.8-27b Q4, while the RTX 2000 manages whisper, vision, and helper models. I use a custom harness-LangGraph. LiteLLM router, MCP Servers, Langfuse, RAG to my NAS and more.
This system has meaningfully improved my work and my life, though heavy admin and projects still go through cloud model review- Opus 5.5
I got lucky and bought that PC for like $1200, the video card was another$1200, but total costs are around $2500, and it has definitely saved me at least that much time. My industry and my workplace are demonstrably uninterested in LLM tools, so my embrace of them gives me a chance to best serve my colleagues and clients when they are about to do the equivalent of long division without an abacus.
I’m waiting for someone to convince me that a second B70 would meaningfully improve my life so if anyone has experience, please let me know, and if you wanna see 5k, cinematic settings of me doing emotes in fortnite, all ya gotta do is ask
f41n 24
That’s actually the thread I stumbled on and made me not only start buying MI25’s, but join this forum
Sweet! You should definitely post there or a new thread and let us know what kinda things you’re doing and how the performance is.
I’ve been meaning to get back into testing my setup since we no longer have unlimited tokens at work.
f41n 26
That’s the plan, but it’ll take a while for me to get the system up and running and full built out, then I’ll do a write up of sorts.