{"slug": "14-years-of-enterprise-asp-net-part-4-azure-observability-ai-in-real-systems", "title": "14 Years of Enterprise ASP.NET, Part 4: Azure, Observability & AI in Real Systems", "summary": "A developer with 14 years of enterprise ASP.NET experience details architectural decisions for a .NET 9 system serving 110,000 monthly active users, emphasizing right-sizing Azure compute to save $2,000 per month and implementing OpenTelemetry-based observability to cut mean-time-to-diagnose from 35 minutes to 4. The project treats AI as a production component, using grounded RAG to deflect about 520 support tickets monthly and classical ML via ML.NET for in-process predictions.", "body_md": "*Originally published at prepstack.co.in*\n\nPart 4 of 4 —Where the system actually runs: choosing Azure architecture by cost and scaling profile, making the system observable, and treating AI as a real architectural component — not a demo.14 Years of Enterprise ASP.NET(finale).\n\nRunning example: **Mattrx** — .NET 9 / ASP.NET Core, 110k MAU, Azure SQL, ~3,200 req/sec peak.\n\n**Pick the compute by your scaling and operational profile, then right-size — don't default to the biggest box or the trendiest platform.** Most enterprise .NET runs perfectly on Azure App Service; you reach for Container Apps or AKS when you have a specific reason, not because Kubernetes is on your résumé.\n\nThe decision framework: **App Service** for standard web/API (default), **Container Apps** when you want containers + scale-to-zero without running a cluster, **AKS** only when you genuinely need its control plane and have the ops capacity. A 5-person team has no business running Kubernetes.\n\nOver-provisioning is the most common and most invisible cloud waste — it never pages anyone, so nobody fixes it.\n\nRight-sizing the web tier (P2v3×6 always-on → P1v3×2 + autoscale), moving to managed Redis, and tuning the SQL tier saved roughly **$2,000/month total** — with *better* peak headroom, because autoscale handles the month-end burst the fixed fleet was over-sized for.\n\nFor years I \"had logging\" and was still blind in production. The shift from logging to **observability** — answering *new* questions about a running system without shipping new code — is the difference between a 4-minute incident and a 4-hour one.\n\nYou can't fix what you can't see, and you can't see what you didn't instrument.\n\nThree pillars, tied by a correlation ID: **logs** (what happened), **metrics** (how much/how often), **traces** (where the time went).\n\n```\n// structured fields + a correlation scope so every line in the request is linkable\nusing (logger.BeginScope(new Dictionary<string, object> { [\"CorrelationId\"] = correlationId }))\n{\n    logger.LogInformation(\"Fetched {Count} campaigns for {TenantId} in {Ms}ms\",\n        count, tenantId, sw.ElapsedMilliseconds);\n}\n\n// OpenTelemetry: traces + metrics out of the box, exported to App Insights\nbuilder.Services.AddOpenTelemetry()\n    .WithTracing(t => t.AddAspNetCoreInstrumentation().AddSqlClientInstrumentation())\n    .WithMetrics(m => m.AddAspNetCoreInstrumentation().AddRuntimeInstrumentation())\n    .UseAzureMonitor();\n```\n\nThis took **mean-time-to-diagnose a production incident from ~35 minutes to ~4** — you jump straight to the failing span instead of grepping by timestamp and hoping.\n\nAI stopped being a side experiment and became a *component* — with the same rigor as any dependency. **Treat an LLM like an untrusted, probabilistic service: wrap it, ground it, validate its output, measure it.** Three distinct shapes, NOT interchangeable: **RAG** (answer from your docs, grounded + cited), **agentic** (tools + reasoning loop), **classical ML** (ML.NET for churn/forecast — no LLM needed).\n\n```\n// AFTER — retrieve context, ground the prompt, validate the output, measure it\npublic async Task<HelpAnswer> AskAsync(string question, CancellationToken ct)\n{\n    var context = await search.RetrieveAsync(question, topK: 5, ct);   // ground in OUR docs\n    if (context.Count == 0)\n        return HelpAnswer.NoAnswer(\"I don't have docs on that.\");       // refuse, don't hallucinate\n\n    var prompt = Prompt.Grounded(question, context);\n    var raw = await llm.CompleteAsync(prompt, ct);\n    var answer = Guardrails.Validate(raw, context);                     // citations + safety\n    metrics.RecordAiCall(tokens: raw.Usage, grounded: true);           // cost + quality tracking\n    return answer;\n}\n```\n\nThe grounded RAG help system deflects **~520 support tickets/month** — but only because it's grounded and has a refuse-path; the naive ungrounded version hallucinated answers that *created* tickets. Classical-ML predictions run in-process via ML.NET at ~3 ms with no LLM cost at all.\n\nFourteen years, twelve lessons, one theme: **enterprise software rewards restraint and fundamentals over novelty.** Clean boundaries (Part 1), a data layer that respects the database (Part 2), ceilings removed and architecture split only when warranted (Part 3), and infrastructure sized to reality with eyes-on observability and AI treated as an engineered component (Part 4). None of it is flashy. All of it is what keeps a system alive — and a team shipping — past year five.\n\n*That's all four parts. Full finale with every before/after, the diagrams, and the numbers — and Parts 1–3 — on PrepStack.*", "url": "https://wpnews.pro/news/14-years-of-enterprise-asp-net-part-4-azure-observability-ai-in-real-systems", "canonical_source": "https://dev.to/kirandeepjassalcrypto/14-years-of-enterprise-aspnet-part-4-azure-observability-ai-in-real-systems-36da", "published_at": "2026-08-14 18:40:21+00:00", "updated_at": "2026-08-14 19:05:37.046187+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "mlops", "ai-products", "developer-tools"], "entities": ["Azure", "ASP.NET", ".NET 9", "Mattrx", "OpenTelemetry", "ML.NET", "Azure SQL", "App Insights"], "alternates": {"html": "https://wpnews.pro/news/14-years-of-enterprise-asp-net-part-4-azure-observability-ai-in-real-systems", "markdown": "https://wpnews.pro/news/14-years-of-enterprise-asp-net-part-4-azure-observability-ai-in-real-systems.md", "text": "https://wpnews.pro/news/14-years-of-enterprise-asp-net-part-4-azure-observability-ai-in-real-systems.txt", "jsonld": "https://wpnews.pro/news/14-years-of-enterprise-asp-net-part-4-azure-observability-ai-in-real-systems.jsonld"}}