Event-Driven AI Orchestration with Kafka in .NET: Learn how to build a production‑grade AI workflow in .NET using Kafka, covering outbox, transactional producers, deterministic keys, DLQ, and observability for high‑throughput, low‑cost orchestration.
In a multi‑tenant scheduling platform, a team glued Azure-ai-foundry-vs-azure-openai-service-observability-in-production-20261007" class="internal-link">Azure OpenAI calls into a handful of background services without a solid event backbone. The result was missed appointments, duplicated prompts, and a Kafka topic that grew until the broker crashed. The LLM was fine; the orchestration was broken. The core issue: an event‑driven architecture that ignored the plumbing that keeps AI pipelines reliable at scale.
Our production system processes ~8 k RPS of appointment requests. Each request triggers a language‑model inference, persists a summary, and pushes a notification. The flow looks like:
Appointment + AiRequestOutbox in a single EF Core transaction.ai.requests to Kafka.ai.results, updates Queues (Service Bus, RabbitMQ) offer at‑most‑once delivery and limited replay. Kafka gives you a durable log, partitioned streams, and built‑in replayability. For AI, replaying a failed inference without re‑invoking the model saves tokens and cost. However, Kafka’s higher memory footprint and the need for a cluster add operational overhead.
Transactional producers (Kafka transactional.id) guarantee atomicity across writes and publishes but require idempotent producers and a dedicated transaction manager. The outbox pattern keeps the transaction in the DB, which is simpler and works well with existing relational workloads. The trade‑off is a slight latency increase due to polling.
Blocking the consumer on a 200 ms to 2 s inference keeps the code simple but caps throughput. Off to a worker pool and emitting a model.completed event frees the consumer but adds complexity in state management.
Managed services give you elasticity, compliance, and reduced operational toil. Self‑hosted models provide predictable cost and full control over scaling, but you must manage GPUs, patching, and data residency. The choice hinges on data sensitivity, latency budgets, and cost models.
| Scenario | Recommendation | Key Trade‑off |
|---|---|---|
| Regulated data, strict compliance | Kafka + Azure OpenAI with VNet integration | Higher operational cost vs. compliance |
| Low latency, high volume | Kafka with transactional producer and async inference workers | Complexity in idempotency vs. throughput |
| Budget‑constrained, predictable usage | Self‑hosted LLM on AKS + Kafka outbox | Operational overhead vs. cost savings |
| Rapid prototyping | Azure Service Bus + synchronous inference | Limited replayability vs. simplicity |
appointmentId) floods one partition, causing a backlog. auto.commit – Commits before successful inference, causing lost work on failure.max.poll.records to throttle; instead, back‑pressure must be handled in the consumer.
In production we adopt the following pattern:
ProcessedRequests table before calling the model.ai.dlq. We expose dlq_message_total as a Prometheus counter and trigger alerts on spikes.transactional.id and use producer.sendOffsetsToTransaction to commit offsets only after the result has been persisted.ContextId to avoid re‑fetching the same data for each request.consumer.poll returns quickly and by using max.poll.interval.ms tuned to the longest inference time.consumer.commitAsync to reduce blocking.linger.ms and batch.size settings to batch messages when the outbox worker is idle.kafka.server.KafkaServer#totalMemory and scale brokers accordingly. For a 3‑node cluster handling 10 k RPS, 32 GB per broker is a safe baseline.partition reassignment during low traffic windows to avoid data skew.topic.replication.factor to 3 for durability.kafka_consumer_lag metrics.
Use Kafka transactional producers with a unique "transactional.id". In the consumer, call producer.sendOffsetsToTransaction after persisting the result. This guarantees that the offset commit and the result publication happen atomically, eliminating duplicate or lost events. Choose deterministic keys that correlate with the natural partition key, such as appointmentId or a composite of userId+timestamp. Avoid random UUIDs or null keys, which scatter events across partitions and cause uneven load.
Maintain a processedRequests lookup (e.g., a table or Redis set) keyed by the message key. Before invoking the model, check this store; if the request has already been processed, skip the call. This prevents token waste and duplicate notifications.
Persist the outbox entity in the same DbContext transaction that creates the domain entity. After commit, a lightweight worker polls the outbox table, publishes the event to Kafka, and marks the row as dispatched. This keeps the write and publish atomic without a dedicated transaction manager.
Expose Prometheus counters like dlq_message_total and kafka_consumer_lag. Configure alerts for spikes in DLQ volume or lag thresholds, and surface metrics in Grafana dashboards to detect back‑pressure or processing failures early.
EnableIdempotence = true and use transactional writes so that an inference request and its response are committed atomically.
Event‑driven AI orchestration is not a “plug‑and‑play” feature of .NET or Kafka. It requires a disciplined approach to message design, idempotency, transactional guarantees, and observability. By following the patterns above – deterministic keys, outbox + DLQ, transactional producers, and a robust monitoring stack – you can build a production‑grade AI workflow that scales from a handful of requests to tens of thousands per second while keeping costs under control and ensuring data integrity.