# ML.NET inference pipelines in .NET Aspire with Redis caching > Source: > Published: 2026-11-03 03:37:23+00:00 ## Quick Answer ML.NET inference pipelines in .NET Aspire: By tuning plumbing—static inference sessions, Redis caching, gRPC streaming, batching, and KEDA scaling—ML.NET pipelines in .NET Aspire stay under 50 ms p99 at 10 k RPS. ## Latency Breakdown in Production Pipelines In a production AI service that pushes **ML.NET** models through **.NET Aspire**, the classic 300‑ms stall is usually not the model itself but the plumbing that gets data from an HTTP request to the model and back. 5 ms of request parsing, 10 ms of message queue latency, 20 ms of Redis lookup, 15 ms of model load, and 30 ms of inference already add up. When the target is **under 50 ms at 10 k RPS**, the margin disappears, and the service feels sluggish. ## Real‑World Example Consider a fraud‑detection microservice that receives a payment [Event](https://dev.to/blog/event-driven-ai-orchestration-with-kafka-in-net-a-production-blueprint-20261020), enriches it with user context, and returns a risk score. The service runs in [Azure]()-ai-foundry-vs-azure-openai-service-observability-in-production-20261007" class="internal-link">Azure Container Apps, uses a local *sentiment‑v1.2.0* ONNX model, and publishes a “risk‑completed” event to downstream analytics. In a recent rollout to a 50‑million‑user platform, the service hit a 70 ms p99 latency spike during a flash‑sale, causing a 4 % drop in checkout completion. ## Trade‑Offs - **Local ONNX vs Azure OpenAI fallback** •*Local* gives deterministic*≈0.02 ¢/call* cost but is bound by model size and GPU memory. •*Azure OpenAI* can handle >4 k token context and adaptive prompting but costs*≈0.50 ¢/call* and introduces 50‑ms network hop. Decision: keep a small, frequently‑used model in local memory; route only the tail of the distribution to Azure. - **Redis KV‑cache vs in‑memory cache** • Redis allows cross‑instance state sharing, essential when scaling workers horizontally. • In‑memory cache eliminates network hop but requires sticky routing and complicates failover. Decision: use Redis for KV‑cache; keep a small per‑process in‑memory LRU for hot keys. - **gRPC streaming vs HTTP polling for model blobs** • gRPC reduces round‑trip and allows streaming large ONNX blobs in a single call. • HTTP is easier to debug but adds ~5 ms per request. Decision: gRPC for production; fallback to HTTP only in dev. - **Batch inference vs per‑request** • Batching (up to 64) cuts GPU kernel launch overhead by ~30 % but can increase per‑request latency if the batch window is too wide. • Per‑request guarantees*O(1)* latency but wastes GPU cycles. Decision: use a 5 ms sliding window; if batch size < 8, fall back to single inference. - **Event‑driven scaling (Service Bus + KEDA) vs static scaling** • Event‑driven auto‑scales to traffic spikes with sub‑second queuing time, but introduces message latency and requires careful idempotency. • Static scaling is simpler but cannot handle flash‑sales. Decision: use Service Bus with KEDA on queue depth; keep a minimum of 2 workers to absorb burst traffic. ## Local ONNX vs Azure OpenAI Trade‑offs | Metric | Local ONNX | Azure OpenAI | | Cost per inference | ≈0.02 ¢ | ≈0.50 ¢ | | Latency (p99) | ≤45 ms (with batching) | ≈70 ms (network + compute) | | Context length | ≤4 k tokens | Unlimited (up to 32 k) | | Model update frequency | Daily | On‑demand | | Operational overhead | Low (mcp + Redis) | High (Azure OpenAI billing, token management) | | Scalability | Horizontal via Aspire + Redis | Horizontal via Azure OpenAI endpoint | | Reliability on network partition | High (local inference) | Low (depends on Azure OpenAI availability) | Choose **Local ONNX + Redis KV‑cache** when you need *sub‑50 ms* latency and cost control. Fall back to **Azure OpenAI** only for the *top 5 %* of requests that exceed context limits or require dynamic prompt generation. ## When This Fails in Production - **KV‑cache eviction** : If Redis TTL is too short, a burst of requests for the same user may hit a cache miss, forcing an expensive MCP call for each request. Result: latency spikes and increased MCP load. - **InferenceSession leaks** : Keeping a single`InferenceSession` static and never calling`Dispose` causes native memory growth. In a 24‑hour container, the process can OOM after ~10 k inferences per second. - **Batch window too wide** : A 5 ms window on a burst of 1 k requests can accumulate a 500‑request batch, pushing GPU memory over 80 % and causing OOM or swapping, which turns the latency benefit into a throughput penalty. - **Service Bus partition exhaustion** : If the topic is configured with too few partitions, the KEDA scaler may not trigger additional workers fast enough, leading to queuing delays that exceed the target latency. - **Model hot‑reload race** : When two workers simultaneously detect a new model version and both pull from MCP, they may each allocate a new`InferenceSession` and dispose the old one, causing a brief period of doubled memory usage. If the old session is still in use, inference errors can occur. ## Common Mistakes Engineers Make - Using `static InferenceSession` without`Dispose` . - Hard‑coding Redis connection strings; not using Azure Key Vault or Managed Identity. - Ignoring `ActivitySource` tags; losing the ability to correlate`model_id` with latency spikes. - Batching without a timeout; causing head‑of‑line blocking under heavy load. - Assuming Service Bus latency is constant; not accounting for message size or partition count. ## Better Approach Based on Experience In a production rollout to 100 k concurrent users, we applied the following pattern: - **Model Loader Service** : A singleton`ModelLoader` that watches a`BlobTrigger` on Azure Storage. When a new blob arrives, it streams the ONNX to an`InferenceSession` and swaps it atomically, preserving the old session until no worker is using it. - **Redis KV‑cache with LZ4** : We store the KV‑cache as an`RDB` blob, compress it with`LZ4` to cut size by 60 %. The cache key includes a`userId:sessionId` tuple and TTL of 4 minutes; after that we drop the cache to free memory. - **Batch Collector with Sliding Window** : The collector uses a bounded channel and a 5 ms timer. If the channel fills before the timer, it immediately processes the batch, ensuring that burst traffic is never held longer than necessary. - **KEDA on Service Bus Queue Length** : We configure a`scaleTargetValue` of 50 messages per worker. This keeps the queue depth below 200 during a 10 k RPS spike, guaranteeing*≤10 ms* queuing latency. - **Observability Hooks** : Every stage emits an`Activity` with tags:`model_id` ,`batch_size` ,`kv_cache_hit` . We aggregate these in Azure Monitor and set an alert when`RunML.NET` exceeds 30 ms or`kv_cache_hit` falls below 90 %. ### Performance Considerations - **CPU vs GPU** : On a CPU‑only VM, a single inference takes ~25 ms. Adding a single NVIDIA V100 reduces this to ~5 ms but saturates GPU memory at batch size 32. We monitor`GPUUtilization` and cap batch size to 48 on V100, 32 on A100. - **Memory Footprint** : An ONNX model of 200 MiB + 4 k token KV‑cache (~1 MiB) per worker consumes ~210 MiB. With 10 workers, that is 2.1 GiB. We keep the container image under 1 GiB and rely on Azure Container Apps to allocate 4 GiB of memory per instance. - **Network Latency** : The MCP gRPC call adds ~2 ms on a 10 ms RTT to the storage account. We pin MCP to the same subnet as the inference workers to keep latency < 3 ms. ### Scaling Notes - **Horizontal scaling** : Each worker is stateless except for the local`InferenceSession` . KEDA scales workers based on queue depth; Redis scales via Azure Cache for Redis Cluster. - **Model rollout** : Use feature flags to route a percentage of traffic to the new model. Validate latency before rolling out to 100 %. - **Cost control** : Local inference costs*≈0.02 ¢/call* . With 10 k RPS, that is*$1.44/h* . Azure OpenAI fallback adds*≈0.50 ¢/call* for 5 % of traffic,*$36/h* , but the savings in latency and user churn justify the spend. ### How do I manage InferenceSession lifecycles in .NET Aspire to avoid memory leaks? Create a singleton ModelLoader that holds an InferenceSession per worker and disposes the old session only after no worker references it. Avoid static InferenceSession without Dispose. ### What are the trade‑offs between local ONNX and Azure OpenAI fallback in an Aspire pipeline? Local ONNX gives deterministic ~0.02¢/call and <45 ms p99 with batching, but is limited to ~4 k tokens and GPU memory. Azure OpenAI handles larger context at ~0.50¢/call with a ~70 ms network hop. ### How can I implement an efficient Redis KV‑cache with LZ4 compression in Aspire? Store cache entries as LZ4‑compressed RDB blobs, key by userId:sessionId, set a 4‑minute TTL, and keep a small per‑process LRU for hot keys to reduce network hops. ### What batching strategy works best for GPU inference in .NET Aspire? Use a 5 ms sliding window collector: batch up to 64 requests, but if the channel fills early, process immediately. Cap batch size to 48 on V100 and 32 on A100 to avoid GPU OOM. ### How does KEDA scaling with Service Bus affect latency in an Aspire service? Configure a scaleTargetValue (e.g., 50 messages per worker) so queue depth stays below ~200 during spikes, keeping queuing latency under 10 ms and preventing burst‑induced latency spikes. ## What to Ship - Pre‑load the ONNX model at Aspire startup and keep a singleton instance; confirm the load time is under 100 ms. - Add a latency histogram to Aspire’s metrics that splits time into **ModelInference** and**ExternalCall** buckets; set a 95th‑percentile alert at 150 ms. - When calling Azure OpenAI, wrap the request in a circuit breaker that trips after 5 consecutive failures and falls back to the local ONNX model. - Expose a health‑check endpoint that runs a sample inference and validates the output shape; fail the check if the shape or value is unexpected. - Deploy the service with the `Prefer‑Region` header pointing to the nearest Azure region; run a 10k‑request latency test to ensure the average stays below 200 ms. - Instrument distributed tracing to log the model version on each request; trigger an automatic redeployment if the logged version does not match the expected one. ### Conclusion When you push **ML.NET** inference into a production **.NET Aspire** pipeline, the plumbing matters more than the model. By carefully balancing local inference, Redis KV‑cache, event‑driven scaling, and a hybrid Azure OpenAI fallback, you can keep p99 latency under 50 ms even at 10 k RPS. The key is to treat each layer—model loading, caching, batching, message queuing—as an observable, tunable component, and to guard against the common pitfalls that erode performance in the real world. ### Related Articles