Llmspeed Benchmark: Pick the Perfect On‑Device LLM Hardware Fast

A developer released Llmspeed, an open-source Python CLI that benchmarks LLM inference across CPUs, Apple Silicon, and NVIDIA/AMD GPUs, reporting cold- and warm-start latency, throughput, utilization, memory, and estimated power draw. The tool runs all inference locally and can execute a YAML-defined matrix of models, devices, and batch sizes, with JSON output feeding a companion spreadsheet that converts results into TCO, electricity cost, and SLA feasibility. Internal tests reportedly showed median latency within 5% of a FastAPI service under identical load.

If you’re tired of guessing whether a single RTX 4090 can sustain 100 req/s for Llama‑2‑7B—or how much electricity a Jetson‑Orin inference server will burn each year—this guide is for you. Llmspeed is an open‑source CLI that lets you run reproducible, hardware‑agnostic benchmarks in minutes, so you can make data‑driven purchasing decisions before you spend a fortune. In the next few sections we’ll walk through installing llmspeed , running benchmarks on CPUs, GPUs, and edge devices, and turning the raw numbers into actionable insights with a ready‑to‑use spreadsheet calculator. By the end you’ll be able to answer questions like: Install the CLI Python 3.10+ required pip install llmspeed Verify the installation llmspeed --version The CLI auto‑detects the best device CPU, Apple M‑series, NVIDIA/AMD GPU llmspeed run --model meta-llama/Llama-2-7b-chat-hf \ --prompt "Explain quantum tunneling in two sentences." \ --output results.json Run on the NVIDIA RTX 4090 explicitly llmspeed run --model meta-llama/Llama-2-7b-chat-hf \ --device cuda \ --prompt "Summarize the plot of Inception." \ --output results.json Create a YAML config with a list of devices and batch sizes cat benchmark.yaml <<'EOF' models: - meta-llama/Llama-2-7b-chat-hf devices: - cpu - cuda - rocm - metal Apple Silicon batch sizes: 1, 8, 32 prompts: - "Translate to French: The quick brown fox jumps over the lazy dog." EOF Execute the whole matrix llmspeed batch --config benchmark.yaml --output matrix results.json All measurements are performed locally; no data ever leaves your machine unless you enable the optional telemetry flag. | Question | Answer | |---|---| | Do I need a GPU to run llmspeed benchmarks? | No. llmspeed works on pure CPU, Apple M‑series GPUs, and external NVIDIA/AMD GPUs. Use --device to select a backend or let the CLI auto‑detect. | | How close are the latency numbers to production performance? | In our internal tests the median latency differed by ≤ 5 % from a FastAPI service under identical load, because llmspeed uses the same HF‑Transformers pipeline with torch.compile when available. | | Can I benchmark proprietary models safely? | Absolutely. All inference runs stay on‑device. The CLI only contacts external servers if you enable --telemetry . | | What metrics does llmspeed report? | Cold‑start latency, warm‑start latency, throughput tokens / sec , GPU/CPU utilization, memory consumption, and estimated power draw via NVIDIA‑SMI or macOS powermetrics . | | Is there a way to compare results across hardware? | Export the JSON output and drop it into the companion Google Sheet calculator link in the repo . The sheet converts raw numbers into TCO, electricity cost, and SLA feasibility. | | Metric | What to Look For | Example RTX 4090 | |---|---|---| | Cold‑start latency | Indicates first‑request penalty; important for bursty traffic. | 420 ms | | Warm‑start latency | Steady‑state response time; the key SLA number. | 68 ms | | Throughput tokens / sec | Determines how many concurrent requests you can sustain. | 2,850 t/s | | GPU Utilization | Should stay below ~85 % for headroom; spikes indicate bottlenecks. | 78 % | | Power W | Feed into your electricity‑cost model. | 250 W peak | | Estimated yearly electricity cost | Power W × hours/year × $/kWh . | $219 US avg $0.13/kWh | Step‑by‑step calculation using the spreadsheet : results.json . | Scenario | Recommended Hardware | Reasoning | |---|---|---| | Low‑budget research notebook | Apple M2 or AMD Ryzen 7950X CPU‑only | Acceptable warm latency ~120 ms for 7B models; zero GPU cost. | | High‑throughput SaaS ≥ 100 req/s | Dual RTX 4090 NVLink | Warm latency ≤ 70 ms, throughput 5 k tokens/s, amortized electricity cost <$300/yr. | | Edge robot with 5 W budget | NVIDIA Jetson Orin 8 TOPS | Power‑aware inference; latency ~250 ms, fits < 5 W envelope. | | Compliance‑first enterprise | On‑premise AMD EPYC + Radeon 7900 XTX | Strong FP16 performance, no vendor lock‑in, easy to audit. | examples/ folder for ready‑made benchmark scripts. Happy benchmarking — The Llmspeed Team Herramienta mencionada: Groq Cloud https://groq.com