Observability Design for the AI Era – App, Infra, CI, LLM (Part 1) Ryan, CTO at airCloset, details how his team shaped observability for AI consumption by splitting monitoring into four axes—application, infrastructure, CI, and LLM—each with a tailored data shape, using OpenTelemetry and Grafana Cloud as the foundation. Hi, I'm Ryan https://x.com/ryantsuji , CTO at airCloset. In the previous series, code-graph deep dive Part 2 /posts/code-graph-46-repos-part2 , I wrote about making a 46-repo codebase semantically searchable for AI. The final issue I left open in that piece was the absence of dynamic analysis : What lives on the graph is the fact that "this edge exists statically." How often that edge actually gets used in production isn't recorded. A graph that gives you static facts is one thing. Telling AI what's actually happening in production right now is a separate problem. So the same shaping discipline I applied to the static graph needs to apply to the observability stack too. This post is the first half of that story. I split it into two: Part 1 this post covers how I shape four different monitoring surfaces application / infrastructure / CI / LLM . Part 2 covers PII handling, the integration surface, and Self-Healing — coming a week later. What Does "Observable to AI" Even Mean? what-does-observable-to-ai-even-mean The biggest lesson from the code-graph series was: the data has to be shaped before AI can consume it . Throwing 46 repositories of source at a model blows past the context window and invites hallucination. So we shaped it — static analysis into a graph, boundary nodes given meaning, SAME ENTITY joins between graphs — and only then handed it over. The observability stack has the exact same problem. Throw raw production logs at AI and you get: - Sheer log volume that drowns the context window - No way for the model to tell errors from noise - Metrics, logs, and traces that don't link to each other - Questions like "what are we spending right now" that raw logs don't answer at all In other words, logs have to be reshaped before AI can use them. Same problem, different domain. The catch is that the right shape depends on what you want AI to answer . At cortex the internal AI platform , I split the monitoring surface into four axes and let each one settle into its own form: Note: "cortex" here refers to airCloset's internal AI platform codename. Unrelated to Snowflake Cortex, Palo Alto Networks Cortex, etc. | Monitoring target | What you want AI to answer | Shape | |---|---|---| | Application | "What's happening in production right now?" exploration | log + trace | | Infrastructure | "Do we have enough resources? Anything down?" time series | metric | | CI | "What broke? Since when?" alert + history | log + alert | | LLM | "How much are we spending? Who's using how much?" real-time + structured aggregation | metric + structured records | "Just push everything through OTel and dump it all in Loki" is an option. But the moment you do, you're asking one backend to answer wildly different kinds of questions — real-time "what's spending right now" alongside "monthly cost broken down by team via SQL" — and one of them is going to suffer. Splitting by purpose is the choice I made. Let me walk through each of the four axes. Application and infrastructure are the foundation, so I'll keep those brief. CI and LLM are where the AI-era design judgments actually surface, so I'll dig into those. Application — OTel + Loki + Tempo, the Standard Stack application--otel--loki--tempo-the-standard-stack The foundation is unremarkable. Every cortex application is instrumented with OpenTelemetry https://opentelemetry.io/ , with traces going to Tempo, logs to Loki, and metrics to Mimir — the standard Grafana Cloud setup. There's no special trick here. What matters is the discipline: every app emits logs and traces in the same shape . That uniformity is what lets AI later run something like {service name="