Event-Driven AI Orchestration with Kafka in .NET: A Production Blueprint

A team running a multi-tenant scheduling platform processing roughly 8,000 appointment requests per second rebuilt its AI orchestration on Kafka in .NET after missed appointments, duplicated prompts, and an unbounded topic crashed the broker. The production blueprint combines an EF Core outbox written in the same transaction as the appointment record, transactional producers using sendOffsetsToTransaction, deterministic partition keys, a dead-letter queue with Prometheus alerting, and tuned consumer polling to keep inference off the critical path. The team reports that replaying failed inferences from Kafka's durable log avoids re-invoking the model, saving tokens and cost.

Event-Driven AI Orchestration with Kafka in .NET: Learn how to build a production‑grade AI workflow in .NET using Kafka, covering outbox, transactional producers, deterministic keys, DLQ, and observability for high‑throughput, low‑cost orchestration. In a multi‑tenant scheduling platform, a team glued Azure <https://dev.to/blog/\<a%20href= -ai-foundry-vs-azure-openai-service-observability-in-production-20261007" class="internal-link" Azure OpenAI calls into a handful of background services without a solid event backbone. The result was missed appointments, duplicated prompts, and a Kafka topic that grew until the broker crashed. The LLM was fine; the orchestration was broken. The core issue: an event‑driven architecture that ignored the plumbing that keeps AI pipelines reliable at scale. Our production system processes ~8 k RPS of appointment requests. Each request triggers a language‑model inference, persists a summary, and pushes a notification. The flow looks like: Appointment + AiRequestOutbox in a single EF Core transaction. ai.requests to Kafka. ai.results , updates Queues Service Bus, RabbitMQ offer at‑most‑once delivery and limited replay. Kafka gives you a durable log, partitioned streams, and built‑in replayability. For AI, replaying a failed inference without re‑invoking the model saves tokens and cost. However, Kafka’s higher memory footprint and the need for a cluster add operational overhead. Transactional producers Kafka transactional.id guarantee atomicity across writes and publishes but require idempotent producers and a dedicated transaction manager. The outbox pattern keeps the transaction in the DB, which is simpler and works well with existing relational workloads. The trade‑off is a slight latency increase due to polling. Blocking the consumer on a 200 ms to 2 s inference keeps the code simple but caps throughput. Offloading to a worker pool and emitting a model.completed event frees the consumer but adds complexity in state management. Managed services give you elasticity, compliance, and reduced operational toil. Self‑hosted models provide predictable cost and full control over scaling, but you must manage GPUs, patching, and data residency. The choice hinges on data sensitivity, latency budgets, and cost models. | Scenario | Recommendation | Key Trade‑off | |---|---|---| | Regulated data, strict compliance | Kafka + Azure OpenAI with VNet integration | Higher operational cost vs. compliance | | Low latency, high volume | Kafka with transactional producer and async inference workers | Complexity in idempotency vs. throughput | | Budget‑constrained, predictable usage | Self‑hosted LLM on AKS + Kafka outbox | Operational overhead vs. cost savings | | Rapid prototyping | Azure Service Bus + synchronous inference | Limited replayability vs. simplicity | appointmentId floods one partition, causing a backlog. auto.commit – Commits before successful inference, causing lost work on failure. max.poll.records to throttle; instead, back‑pressure must be handled in the consumer. In production we adopt the following pattern: ProcessedRequests table before calling the model. ai.dlq . We expose dlq message total as a Prometheus counter and trigger alerts on spikes. transactional.id and use producer.sendOffsetsToTransaction to commit offsets only after the result has been persisted. ContextId to avoid re‑fetching the same data for each request. consumer.poll returns quickly and by using max.poll.interval.ms tuned to the longest inference time. consumer.commitAsync to reduce blocking. linger.ms and batch.size settings to batch messages when the outbox worker is idle. kafka.server.KafkaServer totalMemory and scale brokers accordingly. For a 3‑node cluster handling 10 k RPS, 32 GB per broker is a safe baseline. partition reassignment during low traffic windows to avoid data skew. topic.replication.factor to 3 for durability. kafka consumer lag metrics. Use Kafka transactional producers with a unique "transactional.id". In the consumer, call producer.sendOffsetsToTransaction after persisting the result. This guarantees that the offset commit and the result publication happen atomically, eliminating duplicate or lost events. Choose deterministic keys that correlate with the natural partition key, such as appointmentId or a composite of userId+timestamp. Avoid random UUIDs or null keys, which scatter events across partitions and cause uneven load. Maintain a processedRequests lookup e.g., a table or Redis set keyed by the message key. Before invoking the model, check this store; if the request has already been processed, skip the call. This prevents token waste and duplicate notifications. Persist the outbox entity in the same DbContext transaction that creates the domain entity. After commit, a lightweight worker polls the outbox table, publishes the event to Kafka, and marks the row as dispatched. This keeps the write and publish atomic without a dedicated transaction manager. Expose Prometheus counters like dlq message total and kafka consumer lag. Configure alerts for spikes in DLQ volume or lag thresholds, and surface metrics in Grafana dashboards to detect back‑pressure or processing failures early. EnableIdempotence = true and use transactional writes so that an inference request and its response are committed atomically. Event‑driven AI orchestration is not a “plug‑and‑play” feature of .NET or Kafka. It requires a disciplined approach to message design, idempotency, transactional guarantees, and observability. By following the patterns above – deterministic keys, outbox + DLQ, transactional producers, and a robust monitoring stack – you can build a production‑grade AI workflow that scales from a handful of requests to tens of thousands per second while keeping costs under control and ensuring data integrity.