{"slug": "how-to-monitor-gpu-usage-and-temperature-on-a-remote-server", "title": "How to Monitor GPU Usage and Temperature on a Remote Server", "summary": "A developer published a command-line guide for monitoring GPU utilization and temperature on remote servers, using nvidia-smi queries, cron-scheduled CSV logging, webhook-based temperature alerts, and the nvitop terminal dashboard. The guide notes that data center GPUs such as the A100, H100 and RTX 4090 begin thermal throttling between 83°C and 90°C, and that remote GPUs can throttle to roughly 40% speed while still appearing to run.", "body_md": "Your training run died at epoch 47. The logs show nothing useful — just a silent hang and a CUDA out-of-memory error three hours later. Meanwhile, your GPU has been sitting at 91°C for the last twenty minutes, quietly throttling itself into uselessness.\n\nIf you're renting GPU time on a remote box, you need visibility into two numbers: utilization percentage and temperature in Celsius. This guide walks through monitoring both from the command line, building a lightweight dashboard, and automating alerts so you find out about problems before they cost you a day of compute.\n\n## Why Remote GPU Monitoring Is Different\n\nOn your local workstation, you can hear the fans spin up. On a remote server, you get nothing — no noise, no heat, no visual feedback. A GPU that's thermally throttling still reports \"running\" in your process list. It just runs at 40% speed.\n\nThermal throttling is when a GPU reduces its clock speed to avoid damage from heat. Most data center GPUs (A100, H100, RTX 4090) start throttling between 83°C and 90°C. Above 95°C, you risk hardware degradation or an emergency shutdown.\n\nThe fix is simple: poll `nvidia-smi` on a schedule, log the output, and alert when thresholds break.\n\n## The Foundation: nvidia-smi\n\n`nvidia-smi` (NVIDIA System Management Interface) is the command-line tool that ships with every NVIDIA driver. It reads sensor data directly from the GPU.\n\nThe default output is human-readable but painful to parse:\n\nbash nvidia-smi\n\nFor scripting, use the query flag with CSV output:\n\nbash nvidia-smi --query-gpu=index,name,utilization.gpu,memory.used,memory.total,temperature.gpu,power.draw \\ --format=csv,noheader,nounits\n\nSample output:\n\nFields map in order: index, name, GPU utilization %, memory used (MB), memory total (MB), temperature (°C), power draw (W).\n\nThis one command gives you everything you need for a basic monitoring loop.\n\n## Step 1: A One-Liner Health Check\n\nBefore building anything elaborate, run this every time you SSH in:\n\nbash nvidia-smi --query-gpu=index,utilization.gpu,memory.used,memory.total,temperature.gpu \\ --format=csv,noheader | \\ awk -F', ' '{printf \"GPU %s: util=%s%% mem=%s/%s MB temp=%s°C\\n\", $1, $2, $3, $4, $5}'\n\nIt prints a clean per-GPU summary. If any GPU shows 0% utilization while your training script claims to be running, something is wrong — probably a dead data loader or a hung NCCL collective.\n\n## Step 2: Log Metrics to a File\n\nFor historical data, append to a CSV on a cron schedule. Create a script:\n\nbash #!/bin/bash # /usr/local/bin/gpu-log.sh\n\nif [ ! -f \"$LOG\" ]; then echo \"timestamp,gpu_index,util_pct,mem_used_mb,mem_total_mb,temp_c,power_w\" > \"$LOG\" fi\n\nnvidia-smi --query-gpu=index,utilization.gpu,memory.used,memory.total,temperature.gpu,power.draw \\ --format=csv,noheader,nounits | \\ awk -v ts=\"$TS\" -F', ' '{print ts\",\"$1\",\"$2\",\"$3\",\"$4\",\"$5\",\"$6}' >> \"$LOG\"\n\nSchedule it every 30 seconds:\n\nbash crontab -e\n\nCron's minimum granularity is one minute, so the two-line trick gives you 30-second sampling. For finer resolution, use a systemd timer or just run a `while true; do ...; sleep 5; done` loop inside `tmux`.\n\n## Step 3: Alert on Temperature Spikes\n\nLogging is useless if nobody reads the log. Add a threshold check:\n\nbash #!/bin/bash # /usr/local/bin/gpu-alert.sh\n\nnvidia-smi --query-gpu=index,temperature.gpu,utilization.gpu \\ --format=csv,noheader,nounits | while IFS=', ' read -r idx temp util; do if [ \"$temp\" -ge \"$THRESHOLD\" ]; then\n\ncurl -s -X POST \"$WEBHOOK_URL\" \\ -H \"Content-Type: application/json\" \\ -d \"{\\\"text\\\":\\\"$MSG\\\"}\" fi done\n\nSwap the webhook for Slack, Discord, or PagerDuty — all accept a simple JSON POST. Run this every minute via cron. If you're on a shared host, keep alert frequency low to avoid rate limits.\n\n## Step 4: A Lightweight Dashboard with nvitop\n\nIf you want a live view without leaving the terminal, install `nvitop`:\n\nbash pip install nvitop nvitop\n\nIt's an interactive TUI (terminal user interface) that shows per-process GPU usage, memory, temperature, and power — similar to `htop` but for GPUs. It also works over SSH without any port forwarding. For most developers, this replaces the need for a browser-based dashboard entirely.\n\nFor multi-server fleets, `nvitop --monitor` runs in a non-interactive mode suitable for piping into logging systems.\n\n## Where to Run This\n\nYou can run any of the above on a local workstation with a GPU. But the setup matters most when you're paying for remote compute, because that's where silent failures get expensive.\n\nI've tested this monitoring stack on two providers that give you full `nvidia-smi` access and root SSH:\n\nBoth expose real hardware sensors, which is the whole point — some managed platforms hide temperature data behind their own dashboards, which defeats the purpose of `nvidia-smi`.\n\nIf you're still comparing options, [Server Rental Guide](https://serverrental.store) has a breakdown of GPU server rental providers by price, region, and hardware tier. Worth a read before you commit to a monthly plan.\n\n## Step 5: Watch for the Silent Killers\n\nTwo failure modes don't show up in a simple temperature check:\n\n**Memory creep.** A training loop that leaks tensors will slowly fill VRAM until it OOMs. Track `memory.used` over time and alert when it grows monotonically without dropping. A quick check:\n\nbash nvidia-smi --query-gpu=memory.used --format=csv,noheader,nounits\n\nIf this number only goes up across epochs, you have a leak.\n\n**Utilization gaps.** If `utilization.gpu` drops below 20% for more than a minute while your job is \"running,\" you're likely bottlenecked on data loading or stuck in a deadlock. Add a check that flags sustained low utilization.\n\nbash # Alert if GPU 0 sits under 20% for 3 consecutive samples\n\nwhile true; do UTIL=$(nvidia-smi --query-gpu=utilization.gpu --format=csv,noheader,nounits -i 0) if [ \"$UTIL\" -lt 20 ]; then\n\n[ \"$LOW_COUNT\" -ge 3 ] && echo \"GPU 0 idle for 3 samples — investigate\" && LOW_COUNT=0 else\n\nfi sleep 60 done\n\nRun this in `tmux` alongside your training job. It's crude but catches the deadlock case that kills multi-day runs.\n\n## Putting It Together\n\nA complete monitoring setup is four pieces:\n\n1. **`nvidia-smi` query** — the raw data source, runs in milliseconds. 2. **CSV logging on a cron schedule** — builds a history you can graph later. 3. **Threshold alerts via webhook** — pushes problems to where you'll see them. 4. **`nvitop` for live inspection** — the interactive fallback when something looks off.\n\nTotal setup time: about fifteen minutes. Total cost: zero. Total savings: potentially an entire training run.\n\n## Conclusion\n\nRemote GPU monitoring comes down to polling `nvidia-smi` on a schedule and acting on the numbers. Log utilization, memory, and temperature to a CSV. Alert when temperature crosses 85°C or utilization flatlines. Use `nvitop` when you need to see what's happening right now.\n\nThe developers who lose days to silent GPU failures are the ones who never set this up. Don't be one of them. Pick a provider that gives you real hardware access — PowerVPS or Immers Cloud both work — and wire up the scripts above before your next long run.", "url": "https://wpnews.pro/news/how-to-monitor-gpu-usage-and-temperature-on-a-remote-server", "canonical_source": "https://dev.to/big_mazzy_06d057cc24398c5/how-to-monitor-gpu-usage-and-temperature-on-a-remote-server-55ce", "published_at": "2026-09-25 09:00:32+00:00", "updated_at": "2026-09-25 09:30:34.525483+00:00", "lang": "en", "topics": ["ai-infrastructure", "mlops", "ai-chips", "developer-tools"], "entities": ["NVIDIA", "nvidia-smi", "nvitop", "A100", "H100", "RTX 4090", "Slack", "PagerDuty"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/how-to-monitor-gpu-usage-and-temperature-on-a-remote-server", "markdown": "https://wpnews.pro/news/how-to-monitor-gpu-usage-and-temperature-on-a-remote-server.md", "text": "https://wpnews.pro/news/how-to-monitor-gpu-usage-and-temperature-on-a-remote-server.txt", "jsonld": "https://wpnews.pro/news/how-to-monitor-gpu-usage-and-temperature-on-a-remote-server.jsonld"}}