If you’re tired of guessing whether a single RTX 4090 can sustain 100 req/s for Llama‑2‑7B—or how much electricity a Jetson‑Orin inference server will burn each year—this guide is for you. Llmspeed is an open‑source CLI that lets you run reproducible, hardware‑agnostic benchmarks in minutes, so you can make data‑driven purchasing decisions before you spend a fortune.
In the next few sections we’ll walk through installing llmspeed, running benchmarks on CPUs, GPUs, and edge devices, and turning the raw numbers into actionable insights with a ready‑to‑use spreadsheet calculator. By the end you’ll be able to answer questions like:
pip install llmspeed
llmspeed --version
llmspeed run --model meta-llama/Llama-2-7b-chat-hf \
--prompt "Explain quantum tunneling in two sentences." \
--output results.json
llmspeed run --model meta-llama/Llama-2-7b-chat-hf \
--device cuda \
--prompt "Summarize the plot of Inception." \
--output results.json
cat > benchmark.yaml <<'EOF'
models:
- meta-llama/Llama-2-7b-chat-hf
devices:
- cpu
- cuda
- rocm
- metal # Apple Silicon
batch_sizes: [1, 8, 32]
prompts:
- "Translate to French: The quick brown fox jumps over the lazy dog."
EOF
llmspeed batch --config benchmark.yaml --output matrix_results.json
All measurements are performed locally; no data ever leaves your machine unless you enable the optional telemetry flag.
| Question | Answer |
|---|---|
Do I need a GPU to run llmspeed benchmarks? |
No. llmspeed works on pure CPU, Apple M‑series GPUs, and external NVIDIA/AMD GPUs. Use--device to select a backend or let the CLI auto‑detect. |
| How close are the latency numbers to production performance? | In our internal tests the median latency differed by ≤ 5 % from a FastAPI service under identical load, becausellmspeed uses the same HF‑Transformers pipeline withtorch.compile when available. |
| Can I benchmark proprietary models safely? | Absolutely. All inference runs stay on‑device. The CLI only contacts external servers if you enable --telemetry . |
What metrics does llmspeed report? |
Cold‑start latency, warm‑start latency, throughput (tokens / sec), GPU/CPU utilization, memory consumption, and estimated power draw (via NVIDIA‑SMI or macOS powermetrics ). |
| Is there a way to compare results across hardware? | Export the JSON output and drop it into the companion Google Sheet calculator (link in the repo). The sheet converts raw numbers into TCO, electricity cost, and SLA feasibility. |
| Metric | What to Look For | Example (RTX 4090) |
|---|---|---|
| Cold‑start latency | Indicates first‑request penalty; important for bursty traffic. | 420 ms |
| Warm‑start latency | Steady‑state response time; the key SLA number. | 68 ms |
| Throughput (tokens / sec) | Determines how many concurrent requests you can sustain. | 2,850 t/s |
| GPU Utilization | Should stay below ~85 % for headroom; spikes indicate bottlenecks. | 78 % |
| Power (W) | Feed into your electricity‑cost model. | 250 W (peak) |
| Estimated yearly electricity cost | Power (W) × hours/year × $/kWh . |
$219 (US avg $0.13/kWh) |
Step‑by‑step calculation (using the spreadsheet):
results.json.
| Scenario | Recommended Hardware | Reasoning |
|---|---|---|
| Low‑budget research notebook | Apple M2 or AMD Ryzen 7950X (CPU‑only) | Acceptable warm latency (~120 ms) for 7B models; zero GPU cost. |
| High‑throughput SaaS (≥ 100 req/s) | Dual RTX 4090 (NVLink) | Warm latency ≤ 70 ms, throughput > 5 k tokens/s, amortized electricity cost <$300/yr. |
| Edge robot with 5 W budget | NVIDIA Jetson Orin (8 TOPS) | Power‑aware inference; latency ~250 ms, fits < 5 W envelope. |
| Compliance‑first enterprise | On‑premise AMD EPYC + Radeon 7900 XTX | Strong FP16 performance, no vendor lock‑in, easy to audit. |
examples/ folder for ready‑made benchmark scripts.
Happy benchmarking!
— The Llmspeed Team
Herramienta mencionada: Groq Cloud