The $2,000 Inference Server: Standing Up Local AI on Ten-Year-Old Hardware

A developer built a local AI inference server for about $2,000 using ten-year-old hardware, including two NVIDIA Tesla P40 GPUs and an AMD EPYC processor. The system handles thousands of agent requests daily, running 12 different GPU model stacks behind a single OpenAI-compatible endpoint. The developer found that most online advice for LLM performance is written for newer hardware and often fails on older cards, leading to lessons in measuring on actual hardware and changing one variable at a time.

I run a local inference server that handles thousands of agent requests a day. It cost about $2,000 in used parts, and the newest silicon in it taped out around 2016. This series is the story of standing it up, and more honestly, the story of how much of what I "knew" about it turned out to be wrong. I didn't pick this hardware to prove a point. I picked it because it's what I could afford. It turned out to be the best teacher I could have bought. Server guts used from the enterprise-surplus market eBay receipts, January 2026 ; case, PSU, and cooler new from Newegg: | Component | Cost | |---|---| | AMD EPYC 7302P + Supermicro H11SSL-i 16 cores / 32 threads | $555 | | 128 GB DDR4-2666 ECC 8x 16 GB | $551 | | 2x NVIDIA Tesla P40 24 GB each | $403 | | 1 TB Intel DC P4510 U.2 NVMe | $100 | | U.2 adapter and power cabling | $63 | | Case, PSU, CPU cooler Newegg, approximate | ~$330 | | Total | ~$2,000 | For context, that's roughly two months of what an always-on agent workload would cost me in frontier API bills. The machine paid for itself before I finished tuning it. The P40 is the heart of the build and the source of most of the pain. It's a Pascal datacenter card from 2016: 24 GB of VRAM, compute capability 6.1, no Tensor Cores, no NVLink. Two of them give you 48 GB of VRAM on paper. One of the first hard lessons in my notes is titled "a 47GB model does not fit in 48GB." The driver reserves about 6%, so usable is 45 GiB. Budgeting at the spec-sheet number OOMs. Knowing the "no" list up front would have saved me weeks. Here it is: No vLLM. Compute capability 6.1 is too old. I didn't take the documentation's word for it; I have an experiment directory proving it. Ruled out for real. No Tensor Cores means FP16 is a trap. On Pascal, FP16 math runs at 1/64th the rate of FP32. The card's strength is INT8 through the dp4a instruction, about 47 TOPS. Everything about a working Pascal config flows from that one fact: quantized models, integer matmul kernels, and skepticism toward any advice written for newer cards. No concurrent GPU models. One large model resident at a time. Swaps take about 30 seconds through the Portainer API. You design around it or you fight it forever. No fast cold starts. 20 to 40 seconds to load a model, depending on size. Most advice doesn't apply. This one cost me the most. The internet's LLM performance guidance is written on Ampere and newer. Some of it transfers to Pascal, some of it is irrelevant, and some of it is actively destructive. A "40% faster" split mode I found recommended in a vendor blog crashes Pascal outright with an illegal memory access. Telling those categories apart is most of the work, and it's the subject of this whole series. Here's what those constraints actually bought, with measured numbers from my own logs, not estimates: Twelve different GPU model stacks are compiled, deployed, and benchmarked on this machine today, swapped on demand behind a single OpenAI-compatible endpoint. Agents talk to it all day. Most requests never touch a paid API. None of that requires modern hardware. It requires knowing the machine you actually have. On an H100, plenty of mistakes just cost you a little throughput you never notice. On a P40, mistakes fail loudly. The wrong split mode crashes. The wrong precision runs 64 times slower. The wrong context budget OOMs. The hardware gives you honest feedback because there's no headroom to hide in. That feedback forced habits I now think of as the real payoff of the project: Measure on your own hardware. Community numbers are hypotheses, not facts. Every claim in my notes carries a date and the file the measurement lives in. Change one variable at a time. I learned this by breaking it. A "modernized" rebuild changed four things at once and prompt processing collapsed from 153 tokens/sec to 29. Nothing was attributable until a clean A/B isolated each variable. That incident became a standing rule. Verdicts expire; mechanisms survive. The most useful things I know about this machine aren't rules like "flag X is always right." They're conditions: "flag X wins on these architectures because of this mechanism, and loses on those." Over six months, roughly half of my March conclusions were overturned or narrowed by September. Each reversal has a measurement behind it. That's not embarrassing; that's the discovery process working. That last point is the spine of this series. Performance guidance for a moving target like llama.cpp is perishable. Flags I tuned around got deleted upstream. An environment variable I exported religiously turned out to be dead code that was never read. A "critical" recompile fixed a problem the running binary didn't have. Every one of those stories is a post. The homelab behind this goes back to 2016: Docker, networking, self-hosted services, a decade of running my own infrastructure because I wanted to know how things work. The AI chapter is recent. I started messing with LLMs in the fall of 2025, and the substrate was already there, which is the only reason the timeline in this series is months instead of years. I work in SQL and data pipelines by day, on donor management systems for nonprofits. Nights and weekends I pointed the homelab at inference. This series is the honest record of what happened next: what I believed, what the machine proved, and what I had to unlearn. Everything in it comes from dated, recorded measurements. Where something was never tested, it's labeled as never tested. Where I was wrong, the original wrong belief is quoted, because the reversal is the content. $2,000, ten-year-old silicon, and more real systems education than any cloud bill has ever bought me.