LLMOps for Compound AI Systems — Observability & Cost

A developer outlines an LLMOps playbook for compound AI systems, emphasizing the need for a gateway, observability, caching, validation, and autoscaling to manage complexity, cost, and safety. The article reports a 38% reduction in token spend and 25% lower median latency after implementing these controls in a Q&A pipeline.

Most GenAI pilots don't fail because the models are bad — they fail because the surrounding system wasn't built for production. In 2026 an "LLM call" is rarely a single model invocation. Real systems are compound: embedders, retrievers, vector stores, re-rankers, validators, tool calls, and multiple LLMs wired together. Without a focused LLMOps strategy, that complexity explodes into latency spikes, runaway token bills, and safety gaps once real traffic arrives. This article outlines an actionable LLMOps playbook for compound AI systems that keeps them fast, safe, and affordable. Put a gateway between your application and providers. The gateway is the single control plane for routing, budgeting, caching, and basic guardrails. Route by task complexity and confidence — don’t throw a 13B model at every query. Benefits: Example routing heuristic: Instrument every stage as a span: embed, search, rerank, prompt build, LLM call, tool call. Correlate retriever hit rates, embedder latency, and token usage in one trace so you can find the slow or expensive stage instantly. Use OpenTelemetry-compatible spans and capture: model, prompt version, embedding version, token counts, retrieval scores, and cache hit/miss. Why it matters: when cost or hallucination spikes, the root cause is usually a retrieval or chunking issue — not the final decoder. Cache embeddings + responses keyed by semantic vector of the query and tenant or privacy namespace where appropriate . That lets you short-circuit expensive LLM calls for paraphrases or repeated queries. Key knobs: Typical gains: 15–60% reduction in API calls for repetitive workloads; latency drops from seconds to milliseconds on cache hits. Run lightweight, automated checks before exposing outputs downstream: relevance, faithfulness to retrieved context, hallucination score, and safety filters. Use a small, cheap judge model or heuristic validators to accept/reject or escalate results. Pattern: attempt cheap route - validate output - if validator fails, escalate to stronger model or human review. Autoscale vector DBs, embedder workers, and large-model serving pools separately from front-door routers. Heavy tiers vector search, GPU inference should be monitored and scaled by the metrics they care about: query latency, queue depth, and token consumption, not CPU alone. This avoids the common pattern where a few expensive escalations push the whole stack into failure. Last quarter I inherited a Q&A pipeline that spiked costs during business hours. We implemented three LLMOps controls: Result: 38% reduction in token spend, 25% lower median latency, and a single trace that revealed a misconfigured retriever returning low-quality chunks. python simplified pseudo-implementation from embeddings import embed text from vector store import qdrant search, qdrant upsert from models import small model, large model, validator SIMILARITY THRESHOLD = 0.93 CACHE TTL = 60 60 24 1 day async def handle request tenant id, user query : q emb = embed text user query, model='embed-small' semantic cache lookup hit = qdrant search collection=tenant id, vector=q emb, top k=1 if hit and hit.score = SIMILARITY THRESHOLD: return hit.payload 'response' cache hit complexity classifier cheap heuristic if is simple lookup user query : response = await small model.complete user query else: response = await large model.complete user query eval gate: lightweight judge before returning score = validator.score response, context=q emb if score < 0.7: escalate to stronger model or human queue response = await large model.complete user query, system='escalate' store in semantic cache asynchronously qdrant upsert collection=tenant id, vector=q emb, payload={'response': response}, ttl=CACHE TTL return response This pattern is intentionally simple: embed first, check cache, route, validate, then writeback. In production you’ll add tracing spans around each step and per-request cost attribution. LLMOps for compound AI systems is not a single checklist you run once. It’s an operating model you iterate on as traffic reveals new failure modes: new query types, escalations, or cost drivers. Start with a gateway + tracing + semantic cache and expand to eval pipelines and tiered autoscaling. What single LLMOps control would have saved your team the most pain when you moved from demo to production? Share a painful incident and the control that would have caught it earlier — that’s where the next optimization usually hides.