A $2,000 Radeon R9700 with extra system RAM takes on dual RTX Pro 6000 cards to run a 124B MoE model locally. Here's what actually matters.
What matters more for running big LLMs locally: GPU or RAM? #
For running a single large language model, VRAM decides whether the model fits and how fast it generates tokens. But for running agentic workloads, where multiple coding or research agents run at once and their context has to stick around between turns, system RAM is often the real bottleneck. A budget GPU like AMD’s $2,000 Radeon R9700, paired with enough system RAM, can run a 124 billion parameter mixture-of-experts model and sustain more concurrent agents than a machine with two RTX Pro 6000 cards and less RAM.
TL;DR #
- System RAM, not VRAM, caps how many AI agents you can run at once , because each agent’s growing conversation and tool-call history has to live somewhere between turns, and that somewhere is system memory.
- A 124 billion parameter mixture-of-experts model needed only about 5.5 GB of VRAM to run on a Radeon R9700, with the remaining ~90GB sitting in system RAM, because MoE architectures only activate a handful of experts per token instead of the whole network.
- Doubling RAM from 128GB to 256GB didn’t make a single agent faster, but it changed how many agents could finish at all , taking a 12-agent job from 2 completed out of 12 to all 12 completing, and cutting total job time from 85 minutes to 59 minutes.
- The dual RTX Pro 6000 setup (192GB of VRAM) hit its RAM ceiling before its GPU ceiling , sustaining 12 concurrent coding agents on 128GB of system RAM before throughput collapsed at 16 agents.
- The R9700 with 128GB of RAM sustained 8 concurrent agents before hitting its GPU limit (only 32GB of VRAM for live inference lanes), but jumped to 16 sustained agents once RAM doubled to 256GB.
- Single-agent speed still favors the expensive card : running one agent on one model, the RTX Pro 6000 hit roughly 63 tokens per second versus about 28-29 on the R9700, so the budget card trades raw speed for capability per dollar.
- A big model is not automatically a slow model : because of the mixture-of-experts design, the 124B parameter model ran at a usable ~38 tokens per second on the cheap card even though it dwarfed the GPU’s own VRAM.
__>__add rate limiting to the API
● Reading src/server.ts
● Editing 3 files
+ import{ limit } from'rate'
- app.use(cors()) ✓ Tests passed · ready for PR
● Live
Approved
Approved
Approved
How does system RAM affect running AI agents locally? #
When you run a single prompt against a local LLM, the model weights load into VRAM and the GPU does the work. Agentic workloads are different. Each agent maintains an ongoing conversation: tool calls, retrieved context, and a growing cache of everything it has already processed. That state has to be stored somewhere between turns, and it’s stored in system RAM, not on the GPU.
This matters because if RAM runs low, the system has to evict an agent’s context to make room for another one. When that agent comes back around for its next turn, its conversation isn’t there anymore, so it has to be rebuilt from scratch. That rebuild costs time and compute. In testing on a Threadripper Pro workstation with dual RTX Pro 6000 GPUs (192GB of VRAM total) and 128GB of system RAM, running 8 concurrent coding agents worked fine and 12 agents kept climbing in throughput. At 16 agents, throughput fell off sharply and 10 of the 16 jobs failed to complete, because conversations were getting pushed out of RAM repeatedly. Swap usage stayed at zero and CPU usage stayed around 6% throughout, confirming that neither of those was the constraint. RAM was.
Can a cheap GPU actually run a 124B parameter model? #
Yes, if the model is a mixture-of-experts (MoE) architecture and the system has enough RAM to hold the rest of it. The test case here was a 124 billion parameter model (Llama 3.0 Flash, running with a 131,000 token context window) on a Radeon R9700, an AMD professional workstation GPU with 32GB of VRAM.
By VRAM math alone, a 124B model has no business running on a 32GB card. But MoE models don’t activate every parameter for every token. They route each token through a small subset of “expert” subnetworks, so the active compute footprint per token is much smaller than the total parameter count. In this case, only about 5.5GB of VRAM was needed for the active portion of the model, while roughly 90GB sat in system RAM. The result was about 38 tokens per second, which is usable for real work despite the model being enormous relative to the GPU’s own memory.
How did the $2,000 card compare to the $16,000+ dual-GPU setup? #
The comparison isn’t a simple win or loss, because the two setups hit different bottlenecks.
On the RTX Pro 6000 dual-GPU machine, the limiting factor was system RAM, not VRAM. With 128GB of RAM, it sustained 12 concurrent agents before falling apart at 16. On the R9700, the limiting factor flipped: with only 32GB of VRAM, the GPU could only support so many “live lanes” of inference at once, capping sustained performance at 8 concurrent agents, even though system RAM had plenty of headroom left.
When the R9700’s system RAM was doubled from 128GB to 256GB, nothing about the GPU changed, but the agent ceiling jumped. All 12 agents finished with zero context rebuilds, and the system then sustained 16 concurrent agents with RAM peaking around 150GB, more than the old 128GB setup even had available. For comparison: the RTX Pro 6000 setup with 128GB of RAM handled 12 agents, the R9700 with 128GB of RAM handled 8, and the R9700 with 256GB of RAM handled 16. That’s not a claim that the cheap card “beat” the expensive one outright, since the RTX Pro 6000 setup was never tested with 256GB of RAM either. It shows that RAM, not the GPU tier, was the variable driving how much agent work each machine could sustain.
Is the Radeon R9700 worth it for local LLM work? #
It depends on what you’re optimizing for. On raw single-agent speed, the R9700 loses clearly: running one agent against one model fully loaded on the card, the RTX Pro 6000 produced around 63 tokens per second against roughly 28-29 on the R9700, a bit under half the speed. If low latency on a single task is the priority, the more expensive card wins.
But if the goal is running the largest possible models and stacking agent workloads over time rather than racing a single response, the calculus changes. The R9700 costs a fraction of a single RTX Pro 6000, let alone two, and when paired with enough system RAM, it can still load and usably run a 124B parameter MoE model and sustain more concurrent agents than a RAM-starved flagship setup. The tradeoff is speed, not capability: you’re accepting slower per-agent throughput in exchange for being able to run bigger models and more of them in parallel, for a much smaller GPU budget.
Why would anyone want terabytes of system RAM for AI? #
Because RAM, not VRAM, is what determines how much agent work a machine can hold in flight at once and how big a model it can run at all. AMD’s upcoming Halo Station workstation, announced with up to 2TB of system memory alongside 576GB of HBM3 GPU memory, reflects this: nearly four times more “slow” memory than fast memory, by design. For workloads that keep many agents’ contexts alive simultaneously, or that need to run models whose total size dwarfs any single GPU’s VRAM, bulk RAM capacity is what makes the work possible in the first place. VRAM still governs parallelism and speed, but RAM is what decides whether a model runs at all and how much agentic throughput a machine can sustain before it starts falling over.
Frequently Asked Questions #
Does VRAM or system RAM matter more for local LLM inference?
It depends on the workload. For a single model generating responses to one prompt at a time, VRAM speed and capacity drive performance. For agentic workloads running multiple concurrent agents, system RAM capacity determines how many agents can run before contexts get evicted and have to be rebuilt, which is often the actual bottleneck.
How can a 124 billion parameter model run on a 32GB GPU?
It works if the model uses a mixture-of-experts architecture, which only activates a small subset of its total parameters for each token. In the tested case, only about 5.5GB of VRAM was needed for the active computation, while the bulk of the model’s weights (roughly 90GB) sat in system RAM.
What happens when an AI agent’s context doesn’t fit in RAM?
When system RAM runs low, an agent’s stored conversation and context can get evicted to make room for others. The next time that agent needs to continue, its state has to be rebuilt from scratch instead of resuming, which adds significant time and reduces overall throughput.
regulatory compliance and security.
Is the Radeon R9700 a good choice for running large local LLMs?
It’s a reasonable choice if the priority is running large models and many concurrent agents on a limited budget, since it costs far less than high-end workstation GPUs. It’s a weaker choice if the priority is maximum single-response speed, where higher-end cards with more VRAM still perform roughly twice as fast per agent.
Does adding more RAM make individual AI agents respond faster?
Not directly. In testing, per-agent token throughput stayed roughly the same after doubling RAM from 128GB to 256GB. What improved was the number of agents that could run concurrently without failing, and the total time to complete a batch of agent jobs, which dropped significantly because fewer agents needed their context rebuilt from scratch.