# Build vs Buy Streaming for Real-Time RAG: 2026 Guide > Source: > Published: 2026-06-15 21:40:34+00:00 New in Confluent Cloud: Making Data & Pipelines Accessible for AI-Ready Streaming | [Learn More](https://www.confluent.io/blog/2026-q2-confluent-cloud-launch) Moving a retrieval-augmented generation (RAG) prototype from a Python notebook into production isn't an API orchestration challenge. It's a distributed systems problem. For engineering managers and data platform leads, the build-versus-buy decision on streaming infrastructure will dictate your artificial intelligence (AI) feature velocity for the next three to five years. This guide assumes you've already prototyped a RAG pipeline. The question we tackle here is what changes when you put it in front of customers, where the real cost lives, and how to choose a streaming foundation that won't trap your team in maintenance work for the next decade. **The problem.** Production real-time RAG is a streaming-systems problem, not an API-orchestration problem. DIY pipelines accumulate an integration tax that compounds over time, slowing AI feature velocity to a crawl. **The recommendation.** For most enterprises, buying an unified managed streaming platform that delivers stream, connect, process, and govern under a single service-level agreement (SLA) is the correct choice. It should ship with AI-native primitives built in: in-flight embedding generation, Streaming Agents, and context served via the Model Context Protocol (MCP). **The evidence.** A single production change data capture (CDC) connector typically takes three to six engineering months to build and stabilize DIY paths break against the serverless ceiling (e.g., AWS Lambda's 15-minute execution limit) and bleed cross-availability zone (AZ) egress at $0.01 per GB Confluent customers like Henry Schein One, Notion, and Palmerston North City Council credit the platform for moving high-quality data fast enough to power production AI **The build.** A production-grade platform powered by the Kora engine (GBps+ throughput, 99.99% SLA, fully compatible with Apache Kafka® APIs), more than 120 connectors with more than 80 fully managed (PostgreSQL Debezium, Oracle CDC and XStream, Snowflake, S3), Confluent Cloud for Apache Flink® with ML_PREDICT and AI_COMPLETE for in-flight embeddings, Stream Governance (Schema Registry, Data Contracts, Stream Catalog, Stream Lineage), and Confluent Intelligence (Streaming Agents, Real-Time Context Engine, and built-in ML functions) for agentic AI. **Scope.** This guide is for engineering managers and data platform leads weighing build versus buy for a real-time RAG initiative. Build is still the right answer if you're air-gapped, have extreme customization needs, or have a large platform team to staff ongoing operations. Production RAG is never just a stateless app calling a vector database. When you shift from static file uploads to enterprise real-time context, the architecture becomes a persistent, stateful streaming data problem. **Real-time RAG data flow architecture: ** The invisible components in this diagram demand continuous synchronization. CDC ingestion from operational databases translates complex, high-throughput row-level updates into event streams. Those change events need to be normalized, chunked, and routed to embedding APIs (OpenAI, Cohere, Amazon Bedrock, Voyage AI, or self-hosted models). The generated vectors must then be securely upserted into your vector database (Pinecone, Weaviate, Milvus, or PostgreSQL using pgvector) while you continuously monitor end-to-end freshness. Operating this pipeline exposes teams to demanding day two distributed system operations. You need to handle late-arriving data via precise stream watermarking without corrupting the vector index. You need to gracefully process upstream schema changes, like a suddenly dropped column, without breaking downstream [ chunking logic](https://thestackreview.com/practical-guide-to-data-chunking-rag-applications). And when your AI team upgrades their foundation model, you face the challenge of dual-writing to new indexes and re-embedding millions of historical records without triggering application downtime. These aren't problems you can solve with simple Python scripts or basic batch cron jobs. They require handling continuous database updates, maintaining strict idempotency to prevent duplicate embeddings, and executing high-throughput writes. If you don't treat RAG synchronization as a hardened data layer reality, you'll end up with index bloat, stale context, and degraded AI output quality. Faced with these realities, teams pick one of two paths. Build is the natural starting point. Here's why it usually doesn't end there. Engineering teams initially lean toward building their own streaming infrastructure for valid reasons. Extreme customizability, specialized networking protocols, strict air-gapped GovCloud compliance, and a mandate to avoid perceived vendor lock-in often drive the decision to assemble raw open source components. But these architectures rapidly hit the "serverless ceiling." Initial RAG pipelines built on serverless functions or batch jobs buckle under continuous CDC ingestion. Standard serverless limits, such as [ AWS Lambda's strict 15-minute execution limit](https://docs.aws.amazon.com/lambda/latest/dg/gettingstarted-limits.html), break long-running streaming state. Lambda's Kafka Event Source Mapping (ESM) handles polling for free, but you still pay The architectural breaking point arrives when your team stops shipping differentiated AI features and starts maintaining fragile infrastructure. Highly paid engineers spend their sprints tuning Kafka partitions, managing distributed dead letter queues (DLQs), rewriting broken connector scripts, and orchestrating complex re-embedding workflows when a large language model (LLM) is upgraded. This operational drag is the "integration tax." Stitching together best-of-breed raw cloud components comes with an ever-growing maintenance burden that stalls feature velocity. Building and stabilizing a single production-grade CDC connector typically consumes three to six engineering months of labor. That's because building a connector involves navigating single-threaded snapshot bottlenecks, handling complex state management, and overcoming performance barriers. For example, the Debezium PostgreSQL connector is [ architecturally limited to one streaming task](https://debezium.io/documentation/reference/1.9/connectors/postgresql.html), meaning a single thread captures all changes in order. Under high write volumes, this causes lag and requires multiple connectors to scale, adding to the complexity of partitioning and reassembly. The total cost of ownership (TCO) formula has three components: infrastructure (compute, storage, network), operations (labor), and hidden costs (downtime, opportunity cost, cross-AZ traffic). Self-managed deployments also incur a "state tax." Managing Flink requires [ tuning RocksDB block caches](https://nightlies.apache.org/flink/flink-docs-stable/docs/ops/state/large_state_tuning) and remote durable storage for checkpoints. Multi-AZ open source Kafka deployments silently rack up massive AWS cross-AZ data transfer fees at The table below maps each of those three buckets to where DIY teams pay versus what a unified managed platform absorbs. | | | Broker infrastructure | Self-managed VMs, 24/7 on-call, multi-AZ egress at $0.01 per GB | Fully managed, 99.99% SLA, optimized cross-AZ paths | Connectors | Three to six engineering months per source for the first version, plus ongoing schema-drift fixes | More than 80 fully managed connectors out of the box, no source-side maintenance | Stream processing | Self-managed Flink: RocksDB tuning, checkpoint storage, JVM upgrades | Serverless Flink, billed per Confluent Unit for Flink (CFU) consumed, hard spending caps available | Embedding tier | Separate fleet of Python embedding workers, plus queue and retry logic | ML_PREDICT and AI_COMPLETE inside the stream processor, no separate worker tier | Governance and lineage | Build your own schema registry, lineage tracker, and role-based access control (RBAC) layer | Schema Registry, Data Contracts, Stream Catalog, Stream Lineage included | Operational labor | 0.5 to 2 dedicated platform FTEs at small or medium scale, multiple teams at enterprise | Capacity reclaimed for AI feature work | Specific dollar values vary widely by workload, region, and data volume. Anyone who hands you a single annual figure without your topology in hand is selling you a number. Forrester's [ Total Economic Impact study of Confluent Cloud](/resources/report/forrester-economic-impact-confluent-cloud/) is a defensible starting point for benchmarking your own scenario against a self-managed open source build, and Confluent's Generating embeddings natively inside the stream processor eliminates the need to provision, scale, and monitor a separate fleet of Python embedding workers, reducing both your cloud bill and operational headcount. With the cost of building mapped, the next question is what a managed alternative actually needs to deliver to absorb that complexity. Evaluating managed streaming platforms for RAG workloads requires moving beyond basic throughput benchmarks. In 2026, production-grade data streaming infrastructure must natively execute four foundational capabilities: stream, connect, process, and govern. On top of those four, it needs dedicated AI-native primitives (in-flight embedding, MCP-served context, agent runtime) under a single SLA. The four subsections below cover the foundational capabilities. The fifth covers the AI-native layer that sits on top of them. Your foundational messaging layer must support GBps+ throughput, ultra-low tail latency, and a 99.99% uptime SLA, without manual partition rebalancing. Modern cloud-native engines, like the [ Kora engine](/confluent-cloud/kora/), which powers Confluent cloud, decouple compute from storage to deliver 10x faster autoscaling and 10x lower tail latencies than self-managed Kafka while staying fully compatible with Apache Kafka® at the protocol level. Your existing producers and consumers keep working as they are. Cluster Linking creates real-time replicas of existing Kafka data and metadata for zero-downtime migration when you move away from open-source Kafka. The decoupled architecture means a cluster absorbs sudden ingestion spikes (common during a backfill or re-embedding window) without you having to lift a finger. Evaluate platforms strictly on the breadth and depth of their fully managed connector ecosystem. You need out-of-the-box support for complex CDC workloads, software-as-a-service (SaaS) applications, and object storage. A platform offering [ more than 120 connectors](/product/connectors/), where more than 80 are fully managed (including complex integrations like Postgres Debezium, Oracle CDC, and Snowflake), lets your engineers provision reliable data pipelines in minutes rather than dedicating months to custom development. Stream processing must be serverless, support stateful joins, and execute in-flight machine learning (ML) inference. Transforming a text column into a vector embedding directly inside the stream processor simplifies your architecture. Engines like [ Confluent Cloud for Apache Flink](https://docs.confluent.io/cloud/current/flink/reference/functions/model-inference-functions.html) ship SQL functions like ML_PREDICT and AI_COMPLETE that replace a separate embedding worker tier. Your data engineer writes one ANSI SQL statement to turn a text column in a Kafka topic into a continuous stream of vector embeddings, and the platform handles batching, retries, and rate limits against the embedding API. The same engine supports Python and Java for cases where SQL isn't expressive enough, useful for custom chunking strategies or hybrid retrieval logic. What's distinctive about Confluent Cloud for Apache Flink is the combination of three languages, native AI functions, and a managed runtime sharing one SLA with the broker. The closest AWS path pairs Amazon Managed Streaming for Apache Kafka (MSK) with Amazon Managed Service for Apache Flink (MSF), which delivers a real Flink runtime supporting SQL, Python, and Java but ships no ML_PREDICT or AI_COMPLETE equivalent and sits on a separate SLA from MSK. MSK paired with Lambda is simpler for short enrichment, but Lambda's 15-minute execution wall breaks long-running streaming state. Open source Flink demands deep Java fluency and a self-managed cluster, and Redpanda has no native Flink at all (its in-broker WebAssembly transforms are sandboxed and limited, by Redpanda's own admission, to "trivial and stateless" cases). The processing engine must guarantee exactly-once semantics. Without advanced two-phase commit protocols, retry loops will push duplicate embeddings or miss delete commands, permanently corrupting your RAG context. The processor must also offer robust failure handling (configurable backpressure, buffer debloating, exponential retries, and dead letter queues) to safely navigate strict API rate limits from LLM embedding providers. AI outputs are only as trustworthy as their inputs. You need enterprise-grade governance to keep RAG indexes secure, traceable, and accurate. Start with a [ Schema Registry](https://docs.confluent.io/platform/current/schema-registry/index.html) that enforces strict Data Contracts, preventing an upstream database change from silently breaking your downstream embedding pipeline. Pair it with a Stream Catalog that organizes Kafka topics as discoverable data products with metadata tagging, search, and self-service access requests, so AI teams can find and adopt trusted streams without bottlenecking on a central data engineering team. Stream Lineage gives you the audit trail every AI agent's context source needs, answering "where did this RAG document come from, and what schema version produced its embedding?" RBAC, client-side field-level encryption (CSFLE), and masking ensure personally identifiable information (PII) is masked before it ever reaches the vector database. A modern streaming platform must speak the language of agentic AI. The four foundational capabilities above keep your data plane reliable. The AI-native layer on top is what turns it into a substrate for production agents. Confluent Intelligence is the dedicated AI layer of the data streaming platform and ships three components on top of Kafka and Flink: **Streaming Agents.** Agents that run as Flink jobs inside the stream processing pipeline, with always-on state, tool calling via MCP and Agent2Agent (A2A), and replayable, governed event flows. Because they are Flink jobs, the same exactly-once and lineage guarantees apply to agent decisions. **Real-Time Context Engine.** A fully managed service that serves structured context to AI apps and agents over the [ Model Context Protocol](https://modelcontextprotocol.io/specification/2025-03-26), with built-in authentication, RBAC, and audit logging. MCP integrations include LangChain, Amazon Bedrock, Salesforce Agentforce, and Anthropic Claude. **Built-in ML functions.** Native Flink SQL functions for embedding, anomaly detection, fraud prevention, forecasting, and sentiment analysis, with hooks to invoke remote AI/ML models or custom ones. [ Tableflow](/product/tableflow/) extends these same Kafka topics into open table formats (Apache Iceberg™ and Delta Lake), so the streams that feed your real-time RAG pipeline form the bronze and silver layers of an analytics medallion stack. Tableflow eliminates separate ETL pipelines and shifts processing and governance left, an approach Confluent reports Apply those evaluation criteria to the market, and the practical streaming choices for a real-time RAG initiative are narrowed to four. You can roll your own with open source components, lean on a hyperscaler-managed broker like MSK, pick a Kafka-compatible alternative like Redpanda, or buy a complete data streaming platform like Confluent. Each has a defensible use case. Only one was designed end-to-end for production agentic AI. | | | | | | | Self-managed | Self-managed | Self-managed | Self-managed | DIY | | ✓ Managed broker, 99.9% SLA (infrastructure only) | Bring your own connectors, limited managed CDC | Bolt-on via MSF (separate SLA from MSK, no ML_PREDICT/AI_COMPLETE) or Lambda (15-min cap) | Piecemeal (Glue Schema Registry is primarily Java-focused, no unified catalog or lineage) | Bring your own | | ✓ C++ Kafka-compatible broker, 99.99% multi-zone / 99.5% single-zone, bring your own cloud (BYOC) option | More than 10 fully managed connectors | No native Flink (in-broker WebAssembly only) | Basic schema registry, no Stream Catalog or Stream Lineage | Bring your own | | ✓ Kora engine, 99.99% SLA covering infrastructure and Kafka software | ✓ More than 120 connectors, more than 80 fully managed | ✓ Serverless Flink with ML_PREDICT and AI_COMPLETE | ✓ Schema Registry, Data Contracts, Stream Catalog, Stream Lineage, CSFLE, bring your own key (BYOK) | ✓ Confluent Intelligence (Streaming Agents, Real-Time Context Engine, built-in ML functions) | The subsections below give a profile of the best-fit and trade-offs for each option. The decision matrix later in the article maps these options to specific organizational profiles. The traditional self-managed approach involves provisioning open source Kafka, managing KRaft (or legacy ZooKeeper) quorums, deploying Flink clusters, and writing custom Python workers for chunking and vector embeddings. **Best for:** massive enterprises with dedicated, heavily staffed infrastructure teams, extensive legacy on-premises deployments, unique networking constraints, and extreme customization requirements. **Trade-offs:** you assume the maximum possible operational burden and get zero vendor SLAs on integrations, which means your team handles all edge cases, schema evolutions, and scaling events. This path incurs the highest hidden labor costs and delays time-to-market for AI features. MSK provides a managed broker experience. Teams often pair MSK with MSF or Lambda for processing and AWS Glue for schema management. **Best for:** organizations under strict mandates to use only native AWS services for billing consolidation, or teams already deeply entrenched in the AWS ecosystem and willing to absorb significant day 2 operational burden. **Trade-offs:** for production real-time RAG, the gaps add up fast. First, the ZooKeeper-to-KRaft migration. Apache Kafka removed ZooKeeper entirely in Kafka 4.0. For any MSK customer still running on a ZooKeeper-based cluster (which covers most clusters spun up before AWS added KRaft support to MSK), this is a forced cluster rebuild: MSK has no in-place upgrade path from ZooKeeper to KRaft, so those customers must spin up a new cluster and migrate their data and applications. The technical effort to migrate from ZooKeeper-based MSK to KRaft-based MSK is roughly the same as migrating to Confluent Cloud. Second, the SLA gap is structural. MSK provides 99.9% uptime covering infrastructure only, with Kafka and ZooKeeper software failures explicitly excluded. That works out to 7.9 additional hours (or more due to exclusions) of potential downtime per year compared to Confluent Cloud's 99.99%, which covers both infrastructure and Kafka software. For a real-time RAG pipeline feeding production AI, the gap of nearly eight hours is the difference between a minor incident and a stale-context outage. Third, the hidden costs compound. MSK's apparent low price expands once you account for monitoring beyond CloudWatch's basic tier (topic-level metrics cost extra), a Kafka UI (MSK ships none), Cruise Control for partition rebalancing on Standard clusters, schema registry self-management (Glue Schema Registry primarily supports Java clients), proxy infrastructure, and a Private Certificate Authority for mTLS. Layer on a processing tier you assemble yourself: MSF runs on its own SLA separate from MSK and ships no ML_PREDICT or AI_COMPLETE equivalents, and Lambda is bound by a 15-minute execution wall that breaks long-running streaming state. Add a piecemeal governance story across Glue, Identity and Access Management (IAM), and CloudWatch with no unified Stream Catalog or Stream Lineage equivalent, and you're stitching multiple disparate services together with no single SLA, no Kafka-specific support, and AWS-only deployment with no multi-cloud or hybrid path. Companies like Square, Instacart, iFood, SmartThings, and SecurityScorecard switched from MSK to Confluent because the operational burden and feature gaps became intolerable at scale. SecurityScorecard alone reports [ more than $1 million in savings](/compare/confluent-cloud-vs-amazon-msk/#kafka-cost-of-ownership--msk-vs-confluent) after switching from MSK to Confluent. Redpanda is a C++ Kafka clone with high (but not 100%) Kafka API compatibility, packaged across community on-premises, BYOC, dedicated, and serverless tiers. **Best for:** small teams running simple event logging or edge workloads where C++ thread-per-core architecture and broker-level p99 latency are the primary constraints. **Trade-offs:** Redpanda is a broker, not a data streaming platform, and the platform gap matters most for production RAG. First, it isn’t fully compatible with Kafka API. Partial compatibility means edge cases break with tools that the open-source Kafka community treats as standard. Redpanda's "225 connectors" headline counts processors, which are equivalent to Kafka's single-message transforms (SMTs). The genuine production-ready connector count is a fraction of that figure, none of which are offered as a managed service, compared with Confluent's more than 120 connectors, with more than 80 fully managed. Second, performance claims deserve scrutiny. Redpanda's "10x faster than Kafka" headline holds in synthetic, single-producer benchmarks. It degrades in real production workloads with larger producer groups, record keys, and long-running tests. Confluent's Kora engine, on production-shaped workloads, has been measured up to 10x faster than self-managed Kafka and delivers GBps+ throughput with elastic scaling rather than tier-based manual sizing. Third, compliance and reliability are uneven. Redpanda lists two production-grade certifications (SOC 2 and GDPR readiness, plus a recent HIPAA self-attestation) against Confluent's 10 (SOC 1/2/3, ISO 27001/27701, PCI DSS, CSA Star, TISAX, HITRUST, HIPAA). The single-zone Redpanda BYOC and Dedicated SLA is 99.5%, equivalent to approximately 43 more hours of potential downtime per year than Confluent Cloud. Redpanda BYOC additionally requires installing an agent inside your virtual private cloud (VPC) with break-glass support access for Redpanda engineers, a model that enterprise security teams with strict data sovereignty requirements may find concerning. Stream processing is bolt-on. Redpanda's in-broker WebAssembly transforms are sandboxed and, by Redpanda's own admission, limited to "[ trivial and stateless](https://www.redpanda.com/blog/comparing-flink-vs-redpanda-data-transforms#:~:text=using%20Apache%20Flink-,Your%20operations%20are%20complex%20and%20stateful,your%20transformation%20is%20data%2Dintensive.)" cases. There is no native Flink, no ML_PREDICT or AI_COMPLETE equivalent, no Stream Lineage, no Stream Catalog, no client-side field level encryption, and no BYOK. Customers building real-time RAG end up assembling external processing and governance, which puts them back at the integration tax we already mapped. Real customer migrations underscore the gap. Elemental Cognition, an AI digital native, switched from Redpanda to Confluent Cloud for [ mission-critical real-time workloads](/blog/data-streaming-powers-trustworthy-AI/). Confluent delivers a complete data streaming platform that encompasses the Kora engine, Confluent Cloud for Apache Flink, more than 120 managed connectors, Stream Governance, Tableflow, and Confluent Intelligence under one SLA. **Best for:** enterprises that need to stream, connect, process, and govern data under a single 99.99% SLA covering both infrastructure and Kafka software, and especially for teams building production-grade agentic AI applications who want first-class AI primitives natively integrated into the data plane. **Trade-offs:** Confluent's list price can feel premium for basic, low-volume logging use cases. For complex, multi-source RAG architectures, the consolidated ecosystem typically yields the lowest TCO once connector development time, embedding worker tier consolidation, and avoided governance build-out are included. Forrester's[ Total Economic Impact study](/resources/report/forrester-economic-impact-confluent-cloud/) reports 257% ROI and $2.58M in savings over self-managed Apache Kafka, and Confluent's The Confluent advantage stack is concrete. Kora delivers GBps+ throughput with full Kafka protocol compatibility, so your existing producers and consumers don't change. Cluster Linking gives you a zero-downtime migration path from MSK or self-managed Kafka. Stream Governance bundles Schema Registry, Data Contracts, Stream Catalog, and Stream Lineage into a single suite, and CSFLE and BYOK lock down PII before it reaches the vector index. The people and the AI layer round it out. Confluent was founded by the original co-creators of Apache Kafka. It’s one of the largest contributors to the Apache Kafka open source project, and offers committer-led support with a 60-minute contractual P1 response. On top of that foundation, Confluent Intelligence ships Streaming Agents, the Real-Time Context Engine, and built-in ML functions as native primitives, which is exactly the surface area a production RAG pipeline needs. Customer evidence backs the position. [ Henry Schein One](https://www.youtube.com/watch?v=nc2JaR4czRc&t=230s) frames it directly: "Everyone wants AI, but the hard part is getting high-quality data moving in real time. The Confluent data streaming platform makes that possible for us." Choosing the right streaming infrastructure requires an assessment of your organizational constraints, existing engineering headcount, and strategic AI goals. | | | | | | | | | | Real-time RAG is a streaming systems problem before it is an AI problem. That single reframe is what separates teams who ship production AI from teams who stall in pilot purgatory. The case for building is narrow and well-defined. If you operate in an air-gapped or sovereign environment, have unique networking constraints, or already staff a team of more than 20 engineers dedicated to Kafka and Flink operations, the upfront flexibility of open source components can justify the integration tax. For most enterprises, that case doesn't apply. The cost math in this article is not subtle: three to six engineering months per CDC connector, a serverless ceiling that breaks long-running streaming state, and cross-AZ egress fees that compound silently. None of those costs show up in a vendor proposal. They show up two years in, when your AI roadmap is being held hostage by day two operations on infrastructure your team didn't set out to own. A unified managed streaming platform shifts that math. Stream, connect, process, and govern collapse into one SLA. The embedding worker tier disappears into Confluent Cloud for Apache Flink. Schema Registry, Data Contracts, and Stream Lineage replace governance you would otherwise build yourself. And on top of those four foundational capabilities, AI-native primitives (Streaming Agents, Real-Time Context Engine, and built-in ML functions) give your agent teams a substrate they can actually ship against. If your organization is building agentic AI and needs continuous, trusted context, Confluent is the streaming foundation that absorbs the integration tax instead of charging you for it. To go deeper, explore [ Confluent's ML_PREDICT and AI_COMPLETE model-inference functions](https://docs.confluent.io/cloud/current/flink/reference/functions/model-inference-functions.html) inside Confluent Cloud for Apache Flink, or Real-time RAG continuously syncs changes from operational systems into a vector index so LLM responses use fresh context. That requires CDC ingestion, stateful processing, and reliable delivery, not periodic batch jobs. Use CDC connectors to capture inserts, updates, and deletes, process events to chunk text and generate embeddings, then apply upserts and deletes to the vectors database to prevent drift. The integration tax is the ongoing engineering cost of stitching together and operating connectors, stream processing, retries and dead letter queues (DLQs), schema evolution handling, and re-embedding workflows. It often dwarfs the initial build effort. [ Real-time analytics databases](https://www.linkedin.com/pulse/what-best-real-time-analytics-database-2026-buyers-guide-chawla-gt83c/?trackingId=4mmssiWbRFqnxsuriT1CcA%3D%3D) serve a different role from streaming platforms. The streaming platform handles ingestion, processing, governance, and delivery. A real-time analytics database sits downstream as a query engine, powering sub-second dashboards, operational monitoring, and ad-hoc investigation over the same governed event streams. In architectures that use Tableflow, the analytics engine can query Kafka topics directly as Iceberg tables without a separate ETL pipeline. Commonly, three to six engineering months per connector, once you include snapshots, backfills, failure handling, schema changes, and operational runbooks. Without exactly-once semantics, retries can create duplicate embeddings or miss deletes, corrupting the vector index and leading to stale or incorrect retrieval results. Pipelines can break or silently produce wrong embeddings unless schemas are governed with contracts and a registry, and downstream processors are compatible with additive and breaking changes. You typically dual-write to a new index, backfill historical records, and cut over once parity is verified. This requires orchestration, lineage, and careful rollback planning. When you must run in air-gapped or sovereign environments, need extreme customization, or already have a large platform team to own Kafka, Flink, connectors, and 24/7 operations. MSK can cover the broker layer, but teams often still need to assemble connectors, processing, governance, and reliability patterns across multiple services. That raises operational complexity. Native support for stream, connect, process, and govern, plus AI-ready capabilities like in-flight embedding generation, strong SLAs, schema governance, lineage, and secure PII handling. If embeddings are generated within the stream processor, you can eliminate the need for a separate fleet of Python workers and the associated scaling, monitoring, retries, and queue management overhead. Apply governance controls (RBAC, masking, data minimization) and enforce policies in-stream before embedding or upserting, so sensitive fields never reach the index. EU AI Act obligations for high-risk systems hit in August 2026. Stateless agent frameworks can't satisfy them. This guide covers seven types of state compliant agents must maintain, four streaming patterns for auditability, and a reference architecture using Kafka and Flink as the control plane.