Your training run died at epoch 47. The logs show nothing useful — just a silent hang and a CUDA out-of-memory error three hours later. Meanwhile, your GPU has been sitting at 91°C for the last twenty minutes, quietly throttling itself into uselessness.
If you're renting GPU time on a remote box, you need visibility into two numbers: utilization percentage and temperature in Celsius. This guide walks through monitoring both from the command line, building a lightweight dashboard, and automating alerts so you find out about problems before they cost you a day of compute.
Why Remote GPU Monitoring Is Different #
On your local workstation, you can hear the fans spin up. On a remote server, you get nothing — no noise, no heat, no visual feedback. A GPU that's thermally throttling still reports "running" in your process list. It just runs at 40% speed.
Thermal throttling is when a GPU reduces its clock speed to avoid damage from heat. Most data center GPUs (A100, H100, RTX 4090) start throttling between 83°C and 90°C. Above 95°C, you risk hardware degradation or an emergency shutdown.
The fix is simple: poll nvidia-smi on a schedule, log the output, and alert when thresholds break.
The Foundation: nvidia-smi #
nvidia-smi (NVIDIA System Management Interface) is the command-line tool that ships with every NVIDIA driver. It reads sensor data directly from the GPU.
The default output is human-readable but painful to parse:
bash nvidia-smi
For scripting, use the query flag with CSV output:
bash nvidia-smi --query-gpu=index,name,utilization.gpu,memory.used,memory.total,temperature.gpu,power.draw \ --format=csv,noheader,nounits
Sample output:
Fields map in order: index, name, GPU utilization %, memory used (MB), memory total (MB), temperature (°C), power draw (W). This one command gives you everything you need for a basic monitoring loop.
Step 1: A One-Liner Health Check #
Before building anything elaborate, run this every time you SSH in:
bash nvidia-smi --query-gpu=index,utilization.gpu,memory.used,memory.total,temperature.gpu \ --format=csv,noheader | \ awk -F', ' '{printf "GPU %s: util=%s%% mem=%s/%s MB temp=%s°C\n", $1, $2, $3, $4, $5}'
It prints a clean per-GPU summary. If any GPU shows 0% utilization while your training script claims to be running, something is wrong — probably a dead data or a hung NCCL collective.
Step 2: Log Metrics to a File #
For historical data, append to a CSV on a cron schedule. Create a script: bash #!/bin/bash # /usr/local/bin/gpu-log.sh
if [ ! -f "$LOG" ]; then echo "timestamp,gpu_index,util_pct,mem_used_mb,mem_total_mb,temp_c,power_w" > "$LOG" fi nvidia-smi --query-gpu=index,utilization.gpu,memory.used,memory.total,temperature.gpu,power.draw \ --format=csv,noheader,nounits | \ awk -v ts="$TS" -F', ' '{print ts","$1","$2","$3","$4","$5","$6}' >> "$LOG"
Schedule it every 30 seconds:
bash crontab -e
Cron's minimum granularity is one minute, so the two-line trick gives you 30-second sampling. For finer resolution, use a systemd timer or just run a while true; do ...; sleep 5; done loop inside tmux.
Step 3: Alert on Temperature Spikes #
Logging is useless if nobody reads the log. Add a threshold check:
bash #!/bin/bash # /usr/local/bin/gpu-alert.sh
nvidia-smi --query-gpu=index,temperature.gpu,utilization.gpu \ --format=csv,noheader,nounits | while IFS=', ' read -r idx temp util; do if [ "$temp" -ge "$THRESHOLD" ]; then
curl -s -X POST "$WEBHOOK_URL" \ -H "Content-Type: application/json" \ -d "{\"text\":\"$MSG\"}" fi done
Swap the webhook for Slack, Discord, or PagerDuty — all accept a simple JSON POST. Run this every minute via cron. If you're on a shared host, keep alert frequency low to avoid rate limits.
Step 4: A Lightweight Dashboard with nvitop #
If you want a live view without leaving the terminal, install nvitop:
bash pip install nvitop nvitop
It's an interactive TUI (terminal user interface) that shows per-process GPU usage, memory, temperature, and power — similar to htop but for GPUs. It also works over SSH without any port forwarding. For most developers, this replaces the need for a browser-based dashboard entirely.
For multi-server fleets, nvitop --monitor runs in a non-interactive mode suitable for piping into logging systems.
Where to Run This #
You can run any of the above on a local workstation with a GPU. But the setup matters most when you're paying for remote compute, because that's where silent failures get expensive.
I've tested this monitoring stack on two providers that give you full nvidia-smi access and root SSH:
Both expose real hardware sensors, which is the whole point — some managed platforms hide temperature data behind their own dashboards, which defeats the purpose of nvidia-smi.
If you're still comparing options, Server Rental Guide has a breakdown of GPU server rental providers by price, region, and hardware tier. Worth a read before you commit to a monthly plan.
Step 5: Watch for the Silent Killers #
Two failure modes don't show up in a simple temperature check:
Memory creep. A training loop that leaks tensors will slowly fill VRAM until it OOMs. Track memory.used over time and alert when it grows monotonically without dropping. A quick check:
bash nvidia-smi --query-gpu=memory.used --format=csv,noheader,nounits
If this number only goes up across epochs, you have a leak.
Utilization gaps. If utilization.gpu drops below 20% for more than a minute while your job is "running," you're likely bottlenecked on data or stuck in a deadlock. Add a check that flags sustained low utilization.
bash # Alert if GPU 0 sits under 20% for 3 consecutive samples
while true; do UTIL=$(nvidia-smi --query-gpu=utilization.gpu --format=csv,noheader,nounits -i 0) if [ "$UTIL" -lt 20 ]; then [ "$LOW_COUNT" -ge 3 ] && echo "GPU 0 idle for 3 samples — investigate" && LOW_COUNT=0 else
fi sleep 60 done
Run this in tmux alongside your training job. It's crude but catches the deadlock case that kills multi-day runs.
Putting It Together #
A complete monitoring setup is four pieces:
nvidia-smiquery — the raw data source, runs in milliseconds. 2. CSV logging on a cron schedule — builds a history you can graph later. 3. Threshold alerts via webhook — pushes problems to where you'll see them. 4.nvitopfor live inspection — the interactive fallback when something looks off.
Total setup time: about fifteen minutes. Total cost: zero. Total savings: potentially an entire training run.
Conclusion #
Remote GPU monitoring comes down to polling nvidia-smi on a schedule and acting on the numbers. Log utilization, memory, and temperature to a CSV. Alert when temperature crosses 85°C or utilization flatlines. Use nvitop when you need to see what's happening right now.
The developers who lose days to silent GPU failures are the ones who never set this up. Don't be one of them. Pick a provider that gives you real hardware access — PowerVPS or Immers Cloud both work — and wire up the scripts above before your next long run.