{"slug": "a-practical-framework-for-governing-ai-costs-on-azure-the-hidden-cost-of-ai-on", "title": "A Practical Framework for Governing AI Costs on Azure — The Hidden Cost of Generative AI on Azure", "summary": "A production team running generative AI on Azure found that Azure Cost Management reported its entire Azure OpenAI Service spend as a single $38,400 line with no per-application or per-use-case breakdown, even after a 34% month-over-month jump, and built its own cost-governance framework over three months to fix it. The team placed Azure API Management in front of its Azure OpenAI endpoints to log application name, agent ID, run ID, step number, model, input tokens and output tokens to Azure Monitor, then built a Power BI dashboard showing average cost per agent interaction — which revealed one agent at £0.34 per interaction versus £2.17 for another with the same user-facing experience. The team also warned that teams buying provisioned throughput units (PTUs) off pilot volumes often land at 40–50% utilisation and pay more per token than pay-as-you-go, recommending a PTU baseline with overflow routing to consumption pricing.", "body_md": "Eight months into running AI in production on Azure, our finance team asked a question I should have been able to answer easily.\n\n*Which use case is costing the most?*\n\nI opened Azure Cost Management — the same tool we’d used for years to manage cloud spend across the organisation. I expected a breakdown. By application, by team, by use case.\n\nWhat I got was one line.\n\n*Azure OpenAI Service — $38,400.*\n\nThe same number finance already had from the invoice. No detail underneath it. No explanation of why it had jumped 34% the previous month.\n\nThat moment was the start of a three-month journey to build what Azure doesn’t give you — a framework for actually governing AI costs, not just receiving the bill at the end of the month.\n\nThis is what that framework looks like.\n\nBefore the framework, it’s worth being honest about why Azure Cost Management falls short — because it’s not a criticism of the tool. It’s a tool that was designed for a different problem.\n\nTraditional cloud cost management works because resources consume predictably. A virtual machine running 24 hours costs roughly the same 24 hours every day. Storage costs per gigabyte. Costs attach to resources, resources attach to tags, tags attach to teams. It works.\n\nGenerative AI breaks every one of these assumptions.\n\nThis is not a solvable problem with better tagging. It requires a different architecture for cost visibility — one that sits at the inference layer, not the resource layer.\n\nWith regular cloud services — servers, storage, databases — the cost is fairly predictable. More users, more cost. It scales roughly in line with what you’d expect.\n\nAI is different. Here’s why:\n\nBefore anything else, you need to understand how Azure OpenAI bills you.\n\n**The trap most teams fall into:** They buy PTUs based on how much they used during the pilot. Pilots run at steady, predictable volumes. Production doesn’t — there are quiet nights, busy Monday mornings, seasonal spikes. Many teams end up with PTU reservations sitting at 40–50% utilisation, which means they’re actually paying *more* per token than if they’d stayed on pay-as-you-go.\n\n**The fix:** Use a mix. PTU for your steady baseline — the volume you’re confident you’ll always hit. Pay-as-you-go for anything above that. Azure OpenAI supports this natively. It’s called overflow routing. Set it up so when your PTU capacity is full, extra requests automatically go to consumption pricing.\n\nWe were using Azure AI Agent Service — Microsoft’s managed agent platform within Azure AI Foundry — which simplifies a lot of the orchestration. But it doesn’t solve the cost problem. It just wraps it differently.\n\nOnce we understood the problem, we stopped relying on what Azure gave us natively and built the visibility and controls ourselves.\n\n**A gateway with proper logging.** We deployed Azure API Management in front of our Azure OpenAI endpoints. Every AI call — whether from a simple chatbot or an agent mid-run — passes through APIM, which logs the application name, agent ID, run ID, step number, model, input tokens, and output tokens to Azure Monitor. The step number and run ID were the additions that made agents visible. We could now see not just that an agent ran, but how many steps it took and how much context it was carrying by the end.\n\n**A cost dashboard that shows agent-level spend.** With token data per agent run in Log Analytics, we built a Power BI report that shows average cost per agent interaction, not just total spend. When we first ran this, we found that our most-used agent was costing an average of £0.34 per interaction. Our second most-used agent was costing £2.17. Same user-facing experience. Very different underlying behaviour. The expensive one was running nearly twice as many steps and retrieving much more data than necessary.\n\n**Real-time circuit breakers for agents.** In APIM, we set a maximum token budget per agent run — not per call, but per entire session identified by run ID. If an agent run accumulates more than a threshold of tokens across its calls, the next call returns a “budget exceeded” response that terminates the run. This killed the runaway loop problem immediately. We also set a maximum step count per run — if an agent makes more than 25 LLM calls for a single task, something has gone wrong and the run stops.\n\n**Context pruning at the agent layer.** For agents maintaining state across multi-turn conversations, we implemented a summarisation step every five turns. Rather than passing the full conversation history forward, the agent calls a cheap model (GPT-4o mini) to summarise the last five turns into a compact summary. That summary replaces the raw history in subsequent calls. Context stayed manageable. The cost of the summarisation step was tiny compared to the cost it was preventing.\n\n**Tool output filtering before context injection.** Before any tool result gets injected into the agent’s context, we added a filtering step that extracts only the relevant portions. A search result that returns three pages of text becomes three bullet points by the time the agent sees it. This was the single highest-impact change we made for agent cost — it reduced the average context size per agent run by 61%.\n\n**Step budgets per agent type.** Different agent tasks have different expected step counts. A document classification agent should complete in 3–5 steps. A research and synthesis agent might legitimately take 15. Define expected step ranges per agent type, and flag — or terminate — runs that exceed them. This is different from the circuit breaker: the circuit breaker stops runaway loops, the step budget catches agents that are working harder than they should be.\n\n**Model routing by complexity.** Not every query needs GPT-4o. GPT-4o mini is roughly 15 times cheaper per token and handles the majority of routine tasks — standard questions, simple classifications, common responses — with no meaningful quality difference for those use cases. Sort your use cases: simple, standard, complex. Route through APIM to the appropriate model deployment. In our deployment, this reduced inference spend by 44% in the first month.\n\n**Semantic caching for repeated queries.** Customer-facing AI applications get asked the same questions repeatedly. APIM’s semantic caching capability (backed by Azure Cache for Redis) returns cached responses when a new query is sufficiently similar to something already answered. We hit a 31% cache rate on our customer service application — roughly a third of inference calls now cost nothing.\n\nAzure is investing in this problem. Azure AI Foundry’s project-level tracking and AI Tracing are genuinely useful additions. The gap between what the platform provides and what enterprise finance teams need is narrowing.\n\nIt has not closed. The chargeback attribution, the real-time guardrails, the agent-level controls, the model routing logic — these still need to be built. The build is maybe three to four weeks for a team that hasn’t done it before. The savings are immediate.\n\nThe enterprises that are managing Azure AI spend well are not the ones with the most sophisticated optimisation architecture. They are the ones who built visibility first and made decisions from actual data.\n\n**Start here:** Deploy APIM. Define your logging schema. Get token data flowing. Build one dashboard.\n\nYou will find something in that first dashboard that changes what you work on next.\n\nYou can’t govern what you can’t see.\n\n[A Practical Framework for Governing AI Costs on Azure — The Hidden Cost of Generative AI on Azure](https://pub.towardsai.net/a-practical-framework-for-governing-ai-costs-on-azure-the-hidden-cost-of-generative-ai-on-azure-bbee78fc6bba) was originally published in [Towards AI](https://pub.towardsai.net) on Medium, where people are continuing the conversation by highlighting and responding to this story.", "url": "https://wpnews.pro/news/a-practical-framework-for-governing-ai-costs-on-azure-the-hidden-cost-of-ai-on", "canonical_source": "https://pub.towardsai.net/a-practical-framework-for-governing-ai-costs-on-azure-the-hidden-cost-of-generative-ai-on-azure-bbee78fc6bba?source=rss----98111c9905da---4", "published_at": "2026-11-20 18:01:04+00:00", "updated_at": "2026-11-20 18:13:02.608243+00:00", "lang": "en", "topics": ["ai-infrastructure", "ai-agents", "mlops"], "entities": ["Azure", "Azure Cost Management", "Azure OpenAI Service", "Azure API Management", "Azure Monitor", "Azure AI Agent Service", "Azure AI Foundry", "Power BI"], "also_reported_by": [], "takeaway": "Route every Azure OpenAI call through API Management to log run ID, step number and token counts, then track average cost per agent interaction rather than total spend — one agent cost £2.17 per interaction versus £0.34 for another with the same user experience because it ran nearly twice as many st", "recommendation": {"advice": "Log per-run token and step data at an API gateway and alert on average cost per agent interaction; cap tokens per run ID, and pair a PTU baseline with overflow routing to pay-as-you-go to avoid 40–50% PTU utilisation.", "playbook": "llm-cost", "playbook_url": "https://wpnews.pro/playbooks/llm-cost"}, "alternates": {"html": "https://wpnews.pro/news/a-practical-framework-for-governing-ai-costs-on-azure-the-hidden-cost-of-ai-on", "markdown": "https://wpnews.pro/news/a-practical-framework-for-governing-ai-costs-on-azure-the-hidden-cost-of-ai-on.md", "text": "https://wpnews.pro/news/a-practical-framework-for-governing-ai-costs-on-azure-the-hidden-cost-of-ai-on.txt", "jsonld": "https://wpnews.pro/news/a-practical-framework-for-governing-ai-costs-on-azure-the-hidden-cost-of-ai-on.jsonld"}}