# Llmspeed Benchmark: Pick the Perfect On‑Device LLM Hardware Fast

> Source: <https://dev.to/leojulieta/llmspeed-benchmark-pick-the-perfect-on-device-llm-hardware-fast-6j1>
> Published: 2026-10-17 08:16:46+00:00

If you’re tired of guessing whether a single RTX 4090 can sustain 100 req/s for Llama‑2‑7B—or how much electricity a Jetson‑Orin inference server will burn each year—this guide is for you. **Llmspeed** is an open‑source CLI that lets you run reproducible, hardware‑agnostic benchmarks in minutes, so you can make data‑driven purchasing decisions before you spend a fortune.  

In the next few sections we’ll walk through installing `llmspeed`, running benchmarks on CPUs, GPUs, and edge devices, and turning the raw numbers into actionable insights with a ready‑to‑use spreadsheet calculator. By the end you’ll be able to answer questions like:  

```
# Install the CLI (Python 3.10+ required)
pip install llmspeed

# Verify the installation
llmspeed --version
# The CLI auto‑detects the best device (CPU, Apple M‑series, NVIDIA/AMD GPU)
llmspeed run --model meta-llama/Llama-2-7b-chat-hf \
             --prompt "Explain quantum tunneling in two sentences." \
             --output results.json
# Run on the NVIDIA RTX 4090 explicitly
llmspeed run --model meta-llama/Llama-2-7b-chat-hf \
             --device cuda \
             --prompt "Summarize the plot of Inception." \
             --output results.json
# Create a YAML config with a list of devices and batch sizes
cat > benchmark.yaml <<'EOF'
models:
  - meta-llama/Llama-2-7b-chat-hf
devices:
  - cpu
  - cuda
  - rocm
  - metal   # Apple Silicon
batch_sizes: [1, 8, 32]
prompts:
  - "Translate to French: The quick brown fox jumps over the lazy dog."
EOF

# Execute the whole matrix
llmspeed batch --config benchmark.yaml --output matrix_results.json
```

All measurements are performed locally; no data ever leaves your machine unless you enable the optional telemetry flag.

| **Question** | **Answer** | 
|---|---|
| **Do I need a GPU to run `llmspeed` benchmarks?** | No. `llmspeed` works on pure CPU, Apple M‑series GPUs, and external NVIDIA/AMD GPUs. Use`--device` to select a backend or let the CLI auto‑detect. | 
| **How close are the latency numbers to production performance?** | In our internal tests the median latency differed by **≤ 5 %** from a FastAPI service under identical load, because`llmspeed` uses the same HF‑Transformers pipeline with`torch.compile` when available. | 
| **Can I benchmark proprietary models safely?** | Absolutely. All inference runs stay on‑device. The CLI only contacts external servers if you enable `--telemetry` . | 
| **What metrics does `llmspeed` report?** | Cold‑start latency, warm‑start latency, throughput (tokens / sec), GPU/CPU utilization, memory consumption, and estimated power draw (via NVIDIA‑SMI or macOS `powermetrics` ). | 
| **Is there a way to compare results across hardware?** | Export the JSON output and drop it into the companion Google Sheet calculator (link in the repo). The sheet converts raw numbers into TCO, electricity cost, and SLA feasibility. | 

| Metric | What to Look For | Example (RTX 4090) | 
|---|---|---|
| **Cold‑start latency** | Indicates first‑request penalty; important for bursty traffic. | 420 ms | 
| **Warm‑start latency** | Steady‑state response time; the key SLA number. | 68 ms | 
| **Throughput (tokens / sec)** | Determines how many concurrent requests you can sustain. | 2,850 t/s | 
| **GPU Utilization** | Should stay below ~85 % for headroom; spikes indicate bottlenecks. | 78 % | 
| **Power (W)** | Feed into your electricity‑cost model. | 250 W (peak) | 
| **Estimated yearly electricity cost** | `Power (W) × hours/year × $/kWh` . | $219 (US avg $0.13/kWh) | 

**Step‑by‑step calculation** (using the spreadsheet):

`results.json`.
| Scenario | Recommended Hardware | Reasoning | 
|---|---|---|
| **Low‑budget research notebook** | Apple M2 or AMD Ryzen 7950X (CPU‑only) | Acceptable warm latency (~120 ms) for 7B models; zero GPU cost. | 
| **High‑throughput SaaS (≥ 100 req/s)** | Dual RTX 4090 (NVLink) | Warm latency ≤ 70 ms, throughput > 5 k tokens/s, amortized electricity cost <$300/yr. | 
| **Edge robot with 5 W budget** | NVIDIA Jetson Orin (8 TOPS) | Power‑aware inference; latency ~250 ms, fits < 5 W envelope. | 
| **Compliance‑first enterprise** | On‑premise AMD EPYC + Radeon 7900 XTX | Strong FP16 performance, no vendor lock‑in, easy to audit. | 

`examples/` folder for ready‑made benchmark scripts.
*Happy benchmarking!* 

*— The Llmspeed Team* 

*Herramienta mencionada: [Groq Cloud](https://groq.com)*