Semantic Kernel in a Legacy .NET Product: What Surprised Us After 6 Months in Production

Blackthorn Vision, a Microsoft-partnered .NET and AI development company, replaced a custom prompt orchestration layer with Semantic Kernel in a production healthcare SaaS product. After six months, the team found that Semantic Kernel's plugin system and context management improved code testability and maintainability, but real user inputs in production caused unexpected argument formats and exceptions that staging had missed. The team added explicit input validation to every plugin function to handle noisy production data.

We have been running Semantic Kernel in a production .NET SaaS product in the healthcare data space for six months. The integration replaced a custom prompt orchestration layer that the team had built directly against the Azure OpenAI SDK. It was not a greenfield project. It was a system with real users, real data, and real production constraints from day one. At Blackthorn Vision https://blackthorn-vision.com/ , a Microsoft-partnered .NET and AI development company helping enterprise teams build and modernize complex software products, we have now run several Azure OpenAI and Semantic Kernel integrations using our AI and machine learning development https://blackthorn-vision.com/machine-learning-and-ai-development/ practice and .NET development services https://blackthorn-vision.com/technologies/net-development-services/ past the demo stage and into sustained production. This post covers what we found after six months that we did not fully anticipate going in: the things that staging missed, the things we had to rebuild, and the things that worked better than we expected. The honest answer is that we did not start with Semantic Kernel. The initial integration used direct Azure OpenAI SDK calls wired into ASP.NET Core controllers. It worked for the first three months of limited rollout and covered the basic prompt-response pattern the feature needed. Two things broke it at broader rollout. The first was context management: as conversations grew longer, token costs increased significantly with every additional turn because the full history was being sent with each request. The second was plugin orchestration: the product needed the AI feature to call .NET business logic during inference, and managing that through manual function-calling patterns in the raw SDK produced code that was difficult to test and increasingly fragile. Semantic Kernel solved both. The value of Semantic Kernel is not the SDK itself. The value is giving enterprise .NET teams an orchestration layer that fits naturally into existing dependency injection, logging, security, and service boundaries. Semantic Kernel provided a structured foundation for managing context, The plugin system let us expose existing C service methods to the model as callable functions, using the dependency injection registration the application already had. The migration from raw SDK to Semantic Kernel took about two weeks and the resulting code was significantly more testable and maintainable. Most proof-of-concept implementations never encounter the issues below because they are tested with a handful of users rather than sustained production traffic. What we did not fully appreciate was how much Semantic Kernel's production behavior would differ from its staging behavior, and specifically where it would differ. In staging, we tested plugins with a defined set of representative prompts. The model called the right functions with the right arguments in almost every case. Production looked different within the first week. Real user inputs are noisier than test inputs. Users phrase requests in ways that the model interprets ambiguously. In several cases, the model called a plugin function with an unexpected argument format: an empty string where an integer was expected, a partial value where a full record identifier was required, or a null where the function assumed a populated object. The functions had no input validation because they had never needed it in staging. In production, invalid arguments caused exceptions that surfaced as generic AI feature errors, with no visibility into which plugin had been called or what argument it had received. The fix was two-part. First, we added explicit input validation to every plugin function before any business logic executes: KernelFunction, Description "Retrieves account summary for a given tenant" public async Task<string GetAccountSummaryAsync Description "The tenant identifier, must be a non-empty GUID string" string tenantId { if string.IsNullOrWhiteSpace tenantId || Guid.TryParse tenantId, out return "Unable to retrieve account summary: invalid tenant identifier provided."; return await accountService.GetSummaryAsync tenantId ; } Second, we implemented idempotency for every plugin function with side effects. The model occasionally calls the same function twice in a planning loop, and a function that creates a record or triggers a workflow needs to handle a duplicate call without duplicating the action. After these changes, plugin reliability in production stabilized. Before them, we were seeing roughly one invalid argument error per hundred AI feature interactions. After, such errors became rare. Plugin input validation and idempotency are now part of Blackthorn Vision's standard Semantic Kernel implementation checklist for every enterprise .NET product. We had Application Insights configured for the rest of the product. We assumed the Semantic Kernel integration would surface naturally in the existing telemetry. It did not. Semantic Kernel emits logs, metrics, and traces compatible with OpenTelemetry, but connecting them to the existing Application Insights workspace required explicit configuration that we had not fully completed before rollout. The result was a system where AI feature interactions were visible in the product's usage analytics but invisible in the diagnostic telemetry. When something went wrong with an AI interaction, we knew it happened because the user saw an error. We did not know which plugin was called, what the rendered prompt contained, how many tokens the request consumed, or where in the orchestration chain the failure occurred. The minimum observability setup that made production problems actually diagnosable: // Register Semantic Kernel with OpenTelemetry builder.Services.AddSingleton<Kernel sp = { var kernelBuilder = Kernel.CreateBuilder ; kernelBuilder.AddAzureOpenAIChatCompletion deploymentName: config "AzureOpenAI:DeploymentName" , endpoint: config "AzureOpenAI:Endpoint" , credentials: new DefaultAzureCredential ; kernelBuilder.Services.AddLogging logging = logging.AddOpenTelemetry otel = { otel.IncludeFormattedMessage = true; otel.IncludeScopes = true; } ; return kernelBuilder.Build ; } ; Beyond the SDK configuration, we added a filter that logs prompt inputs with PII fields redacted , function call results, token counts broken down by input and output, and latency at each orchestration step: public class ProductionObservabilityFilter : IPromptRenderFilter, IFunctionInvocationFilter { private readonly ILogger logger; private readonly TelemetryClient telemetry; public async Task OnPromptRenderAsync PromptRenderContext context, Func<PromptRenderContext, Task next { await next context ; logger.LogInformation "Prompt rendered. Template: {Template}, TokenEstimate: {Tokens}", context.Function.Name, EstimateTokens context.RenderedPrompt ; } public async Task OnFunctionInvocationAsync FunctionInvocationContext context, Func<FunctionInvocationContext, Task next { var sw = Stopwatch.StartNew ; await next context ; sw.Stop ; telemetry.TrackDependency "SemanticKernel", context.Function.Name, context.Function.PluginName, DateTimeOffset.UtcNow, sw.Elapsed, context.Result.ValueType = null ; } } With this in place, diagnosing production problems that previously took hours started taking minutes. Every Blackthorn Vision Azure OpenAI engagement now starts with observability infrastructure fully configured before the first production user reaches the system. Single-user testing does not reveal how Azure OpenAI rate limits behave under concurrent multi-tenant load. We discovered this at month two when a cluster of high-activity users exhausted the deployment's token-per-minute quota and caused AI feature failures for all users simultaneously. The immediate fix, for workloads where regulatory requirements permitted cross-region deployment, was provisioning a second Azure OpenAI deployment in a different Azure region and implementing client-side load balancing. The longer-term fix was adding per-tenant throttling at the application layer before requests reach Azure OpenAI, so that no single tenant can consume a disproportionate share of the shared quota: public class TenantRateLimiter { private readonly IMemoryCache cache; private const int MaxRequestsPerTenantPerMinute = 20; public async Task<bool AllowRequestAsync string tenantId { var key = $"rate limit:{tenantId}:{DateTime.UtcNow:yyyyMMddHHmm}"; var count = cache.GetOrCreate key, e = { e.AbsoluteExpirationRelativeToNow = TimeSpan.FromMinutes 2 ; return 0; } ; if count = MaxRequestsPerTenantPerMinute return false; cache.Set key, count + 1, TimeSpan.FromMinutes 2 ; return true; } } Per-tenant throttling is now a mandatory requirement in every multi-tenant Azure OpenAI system Blackthorn Vision builds. One effective approach was implementing it through IMemoryCache. In a single-node deployment this works correctly, but in a multi-instance environment behind a load balancer, in-memory rate limiting operates per instance rather than across the fleet. In production we back this with Azure Cache for Redis to ensure limits apply consistently across all running instances. The code below shows the pattern; substitute the cache implementation based on your deployment topology. We also implemented retry logic that respects the Retry-After header Azure OpenAI returns with 429 responses, using Polly: js var retryPolicy = HttpPolicyExtensions .HandleTransientHttpError .OrResult r = r.StatusCode == HttpStatusCode.TooManyRequests .WaitAndRetryAsync retryCount: 3, sleepDurationProvider: attempt, response, = { var retryAfter = response?.Result?.Headers.RetryAfter?.Delta; return retryAfter ?? TimeSpan.FromSeconds Math.Pow 2, attempt ; }, onRetryAsync: , timespan, attempt, = { logger.LogWarning "Azure OpenAI throttled. Retry {Attempt} in {Delay}s", attempt, timespan.TotalSeconds ; return Task.CompletedTask; } ; The built-in chat history mechanism in Semantic Kernel accumulates conversation turns and sends the full history with every request. In a copilot feature with multi-turn conversations, cumulative token usage grows significantly as conversation length increases. This did not appear in staging because test conversations were short. In production, conversations regularly reached 20 to 30 turns for engaged users. The token cost per request at turn 25 was substantially higher than at turn 5, and the context window limit became a practical constraint for the longest conversations. We implemented a sliding window approach using SharpToken for local token counting before each request: js var encoding = GptEncoding.GetEncodingForModel "gpt-4o" ; const int MaxContextTokens = 8000; while history.Messages.Count 2 { var totalTokens = history.Messages .Sum m = encoding.Encode m.Content ?? string.Empty .Count ; if totalTokens <= MaxContextTokens break; // Remove oldest non-system message var oldest = history.Messages .FirstOrDefault m = m.Role = AuthorRole.System ; if oldest = null history.Messages.Remove oldest ; } For conversations where context continuity matters beyond the window, we added Azure AI Search as a vector memory store. Key facts from earlier turns are embedded and retrieved at inference time, giving the model access to important earlier context without including the full message history. This allowed us to preserve business context without continually expanding prompt size or exposing the full conversation history, which matters particularly in enterprise products where conversations contain sensitive operational data. The things that did not require remediation are worth noting because they reflect where Semantic Kernel genuinely adds value over building the same things manually. The dependency injection integration was seamless. Plugins registered as services in the ASP.NET Core container were immediately accessible to the kernel without any additional wiring. Business logic that already existed as injectable services could be exposed to the model as callable functions with minimal code. The filter pipeline for governance was straightforward to implement and comprehensive once in place. Input sanitization, PII scrubbing, per-tenant data isolation, and audit logging all lived in one place rather than being scattered across prompt handling code. Streaming responses worked immediately with InvokePromptStreamingAsync and eliminated the client-side timeout problems that had plagued the raw SDK integration. First token latency dropped from the full generation time to under two seconds in most cases. The Managed Identity integration with DefaultAzureCredential meant no API keys in configuration files, no credentials to rotate, and no changes between local development and production deployment. Three things would have saved meaningful time if we had addressed them before rollout. Plugin input validation as a design requirement, not an afterthought. Every function the model can call needs to be defensive about the arguments it receives, because the model's function-calling behavior under real user inputs will not match what it does in controlled testing. Observability infrastructure before the first production user. The filter pipeline and OpenTelemetry configuration should be in place and tested before rollout, not added in response to the first production incident. Per-tenant throttling from day one in multi-tenant products. Azure OpenAI rate limits apply at the deployment level, not the tenant level. A multi-tenant product without application-level per-tenant throttling is one heavy-usage period away from an outage that affects all tenants simultaneously. Most teams building AI features into existing .NET products will encounter the same surprises we did. They are not edge cases. They are the predictable failure modes of Semantic Kernel integration moving from staging to production under real user load. The Azure architecture https://blackthorn-vision.com/technologies/azure-development-services/ that makes these integrations reliable includes Private Endpoints for data isolation, Managed Identity for authentication, Application Insights for end-to-end observability, and Azure AI Search for production RAG pipelines. None of these are optional for enterprise products handling sensitive data at scale. And none of them are visible in a demo environment. According to Microsoft's own Semantic Kernel production guidance https://learn.microsoft.com/en-us/semantic-kernel/concepts/enterprise-readiness/observability , enterprise-grade AI integration requires observability, resilience, and security infrastructure built alongside the AI features themselves. Deloitte's 2025 analysis of digital operating models https://www.deloitte.com/us/en/insights/topics/business-strategy-growth/digital-operating-models.html found that digital ownership and governance structures are among the strongest predictors of program success. The OWASP Top 10 for LLM Applications https://owasp.org/www-project-top-10-for-large-language-model-applications/ identifies prompt injection, insecure plugin design, and excessive agency as the primary production risks, all of which are addressed at the orchestration and filter layer in Semantic Kernel, not at the model level. Production AI success depends less on the model than on the engineering around it. In our architectural practice at Blackthorn Vision, we have made it a standard to treat plugin validation, observability infrastructure, per-tenant throttling, and context window management as production requirements from sprint one, not as items to address after the first incident. The failure modes above are the ones we design around from the start because we have seen each of them surface under real user load. If you are building Azure OpenAI and Semantic Kernel integrations into an existing .NET product and want to compare notes on any of the patterns above, the Blackthorn Vision Clutch profile has context on how these engagements run in practice.