CPU rightsizing is a solved, well-documented practice. GPU rightsizing is where the real money is now, and almost nobody writes about it, because GPU instances are expensive enough that people are scared to touch them and unsure how. Given how much a GPU box costs per hour, an over-provisioned one is the single most expensive rightsizing mistake in your account. Here is how to rightsize AWS GPU instances without breaking the workload, including the compatibility check that quietly bites people.
Rightsizing starts with using the right family, not just the right size. On AWS:
The most common GPU waste is running a training-class P-family instance for an inference workload that a G-family instance would serve fine at a fraction of the cost. Wrong family is a bigger error than wrong size.
GPU workloads have several resources that can be the bottleneck, and CPU utilization, the thing you would check for a normal instance, is often the least relevant:
nvidia-smi
telemetry.)The rightsizing signal is a GPU sitting at low utilization or using a fraction of its VRAM over a sustained window (a 90-day-style baseline, same idea as CPU rightsizing). That is your candidate to move down a size or across to a cheaper family.
Here is the gotcha that turns a clean rightsizing into an outage. Different GPU families use different NVIDIA architectures, which means different CUDA driver and library requirements. Move a workload from, say, an A100-based P4 to an L4-based G6, and the driver version, CUDA toolkit, and framework build that worked on one may not match the other.
If you rightsize the instance but do not verify CUDA compatibility, the box comes up and the workload fails to initialize the GPU, or silently falls back to CPU and runs at a crawl. Before any GPU family change: This step is the difference between "rightsized and saving money" and "rightsized and paged at 2am because inference is down." It is also why teams avoid GPU rightsizing entirely, they got burned once and never went back. Do the compatibility check and it is safe.
Rightsizing the instance matters, but the bigger GPU waste is often time, not size. GPU utilization on "always-on" fleets is routinely far below what people assume, and eval, dev, and training-experiment pools have no reason to run overnight or on weekends.
GPU rightsizing is the highest-value, least-covered cost work in most AI-touched accounts, because the per-hour price makes every mistake expensive. Use the right family (G5/G6 for inference, P4/P5 for heavy training), rightsize on GPU utilization and VRAM rather than CPU, and always run the CUDA compatibility check before a family change so you do not trade savings for an outage. Then make sure the thing is not idling overnight, which is often the bigger win.
Have you rightsized GPU instances, and did the driver/CUDA compatibility catch you the way it caught a lot of us? That check is the step every GPU-cost article skips and every GPU incident includes.