# Observability for AI Infrastructure: What to Monitor Beyond CPU and Memory > Source: > Published: 2026-08-18 10:00:00+00:00 Thanks for taking the time to read. If you’ve worked with AI infrastructure or observability, I’d love to hear your experience in the comments. In Part 2 of **AI Infrastructure for Cloud Engineers**, we looked at how GPUs, scheduling, autoscaling, and model serving change the way AI workloads run on Kubernetes. Read Part 2: [Running AI Workloads on Kubernetes: GPUs, Scheduling, Scaling, and Model Serving] Getting an AI workload into production is only the beginning. Once users depend on it, the questions change: A Kubernetes dashboard showing healthy Pods cannot answer all of these questions. Production AI systems need visibility across **infrastructure, accelerators, inference, and the full request path**. Traditional infrastructure metrics still matter. For Kubernetes, we still need to watch: ``` CPU Memory Pod availability Pod restarts Node health Network Storage Request rate Error rate Latency ``` But consider this: ``` Pods Running: 5/5 CPU Usage: 42% Memory Usage: 58% Pod Restarts: 0 ``` Everything looks healthy. Now look at the AI workload: ``` GPU utilization: 99% Inference latency: Increasing Queue depth: Growing Time to first token: Increasing ``` The platform is technically running. The user experience is still getting worse. That is the main difference with AI infrastructure observability: **you need to connect infrastructure health with model behavior.** I find it useful to think about AI observability in four layers: ``` 1. Kubernetes Infrastructure ↓ 2. GPU / Accelerator ↓ 3. Model Inference ↓ 4. End-to-End Request ``` Each layer answers a different question. This is the foundation. Monitor: ``` Pod availability Pod restarts Node health CPU utilization Memory utilization Network Storage Deployment health ``` If inference suddenly becomes slow, you first need to know whether the problem is actually inside the model. Maybe the Pod is under memory pressure. Maybe a node has a networking issue. Maybe the application cannot reach a dependency. Kubernetes metrics provide that first layer of context. GPUs are often among the most expensive resources in an AI platform. Useful signals include: ``` GPU utilization GPU memory usage Temperature Power consumption Device health GPU errors ``` For NVIDIA environments, **DCGM Exporter** can expose GPU telemetry in a Prometheus-compatible format. A simple monitoring flow might look like this: ``` GPU Nodes ↓ DCGM Exporter ↓ Prometheus ↓ Grafana ``` The goal is not just to ask: Is the GPU busy? A better question is: Is the GPU being used efficiently while keeping inference healthy? For example: ``` GPU: 95% Queue: Low Latency: Stable Throughput: High ``` That may be perfectly healthy. But: ``` GPU: 95% Queue: Growing Latency: Increasing Errors: Increasing ``` tells a very different story. The value comes from correlating signals rather than looking at one metric in isolation. Inference metrics tell us what the AI service is actually doing. The most useful ones include: ``` Request rate Inference latency Time to first token Tokens per second Queue depth Concurrent requests Model errors Timeouts ``` A request may pass through several stages: ``` Request ↓ Queue ↓ Model Processing ↓ First Token ↓ Response Generation ↓ Complete Response ``` That gives us several useful timings: Queue depth is especially useful as an early warning signal. Imagine: ``` 09:00 → 2 waiting requests 09:05 → 18 09:10 → 64 09:15 → 140 ``` Nothing has crashed. But demand is arriving faster than the available inference capacity can handle. That signal can also feed autoscaling: ``` Queue grows ↓ Scaling signal ↓ More inference capacity ↓ Queue decreases ``` This is where AI-specific metrics become operational signals, not just dashboard numbers. A production AI application is rarely just a model. A request might travel through: ``` User ↓ API Gateway ↓ AI Application ↓ Model Server ↓ Vector Database ↓ External Tool ↓ Response ``` If the request takes eight seconds, we need to know where those eight seconds were spent. Without tracing: ``` Request duration: 8.2 seconds ``` With tracing: ``` API Gateway 40 ms Application 70 ms Vector Search 420 ms Model Inference 6.4 sec External Tool 950 ms ``` Now the bottleneck is much easier to identify. Each observability signal answers a different question. Is something wrong? Example: ``` Inference latency increased 40%. ``` What happened? Example: ``` Model request timed out after 10 seconds. ``` Where did it happen? Example: ``` Most of the delay occurred during vector retrieval. ``` Together: ``` Metric Alert ↓ Latency increased ↓ Trace investigation ↓ Vector search is slow ↓ Logs ↓ Database connection pool exhausted ``` That is much more useful than looking at disconnected dashboards. A correlation ID also helps connect these signals. ``` { "request_id": "req-a91f82", "service": "model-server", "model": "model-v2", "latency_ms": 1840, "status": "success" } ``` Now the same request can be followed through logs and traces across multiple services. Avoid logging: ``` API keys Access tokens Passwords Sensitive prompts Private customer data Confidential model responses ``` Observability should improve visibility without becoming a security risk. A simple cloud-native setup could look like: ``` Kubernetes │ ├── Application Metrics ├── GPU Metrics ├── Logs └── Traces │ ▼ OpenTelemetry / Exporters │ ├── Prometheus ├── Log Backend └── Trace Backend │ ▼ Grafana Dashboards Alerts ``` The tools may differ between organizations. The pattern is what matters: ``` Collect ↓ Correlate ↓ Visualize ↓ Alert ↓ Investigate ``` A dashboard with dozens of graphs can still be difficult to use. Instead, build dashboards around operational questions. Monitor: ``` Available replicas Pod restarts Node health Request success rate ``` Monitor: ``` GPU utilization GPU memory Temperature Power Device health ``` Monitor: ``` Request latency Time to first token Tokens per second Queue depth Concurrent requests ``` Monitor: ``` Errors by model Errors by provider Timeouts Retries Failed requests ``` That gives engineers somewhere useful to start during an incident. Not every metric needs an alert. For example, 90% GPU utilization does not automatically mean there is a problem. If throughput is high and latency is stable, the system may simply be using its resources efficiently. A more useful alert might be: ``` Queue depth increasing AND Inference latency increasing ``` That points to something users are actually experiencing. The best alerts are actionable. Otherwise, teams eventually start ignoring them. Multi-tenant AI platforms also need visibility by tenant. Suppose the overall error rate reaches 15%. That sounds serious. But the breakdown might be: ``` Tenant A: 1% Tenant B: 2% Tenant C: 78% Tenant D: 1% ``` Now the problem looks isolated rather than platform-wide. This matters when tenants use different: ``` Model providers API keys Quotas Tools Workloads ``` Tenant-level visibility also helps with rate limiting, failure isolation, and capacity planning. GPU utilization is not only a performance metric. It is also a cost signal. Consider: ``` Cluster A GPU utilization: 82% Cluster B GPU utilization: 19% ``` If both clusters use similar hardware, Cluster B deserves investigation. Maybe that spare capacity is intentional. Or maybe the organization is paying for GPUs that spend most of their time idle. The same applies to: ``` GPU hours Tokens generated Requests served Model usage ``` These signals begin to connect infrastructure behavior with spend. That leads directly into the next part of this series: **FinOps for AI**. Before running an AI workload in production, make sure you can answer: If several of these questions cannot be answered quickly, there is probably an observability gap. Observability tells us how the infrastructure behaves. It also reveals something else: How efficiently are we using the infrastructure we are paying for? GPU hours, token usage, inference volume, idle capacity, and model selection all affect the economics of an AI platform. In Part 4, we will look at: FinOps for AI: Understanding GPU, Token, and Inference Costs We will break down where AI infrastructure costs come from, why GPU utilization matters financially, and what cloud teams can measure to avoid unnecessary spend. For AI infrastructure, CPU and memory are still important. They are simply no longer enough. A production AI platform needs visibility across: ``` Kubernetes ↓ GPU Infrastructure ↓ Model Serving ↓ Inference ↓ Dependencies ↓ User Experience ``` The real value comes from connecting those layers. Instead of asking: Why does the AI feel slow? we want to be able to say: Queue depth increased because the inference workers reached GPU capacity, which pushed time to first token above our target. That is the difference between simply monitoring infrastructure and actually understanding the system. This article is **Part 3 of my AI Infrastructure for Cloud Engineers series**: I regularly share what I learn about cloud infrastructure, Kubernetes, DevOps, SRE, observability, and the engineering behind production AI systems. **LinkedIn:** [Connect with me on LinkedIn](https://www.linkedin.com/in/sushyamnagallapati/) If you're operating AI workloads in production, which signal has been most useful for you: GPU utilization, inference latency, time to first token, queue depth, or something else?