# Your Kafka Producer Might be Lying to You Right Now > Source: > Published: 2026-07-07 07:43:19+00:00 Kafka’s biggest promise is a lie under the right conditions. I found out the hard way, building something that had no business breaking. I built the ingestion pipeline for a RAG system the way most people do once they’ve read enough architecture blogs: documents come in, get chunked, get embedded, land in pgvector. Somewhere in that chain I wanted a queue instead of calling functions directly, and Kafka was the obvious answer — fan-out to multiple consumers, replay on failure, and the line every Kafka pitch leads with: exactly-once delivery. Send a message once, it gets processed once. For an ingestion pipeline, that’s not a nice-to-have. Process the same document twice and you don’t get a harmless retry — you get the same chunk embedded twice, sitting in your vector store as two entries competing for the same retrieval slot. So I trusted it. That was the whole point of picking Kafka over pushing tasks into a Python queue and hoping for the best. Then my vector store had more entries than documents I’d ingested. Not a lot more. Not the kind of gap you catch by eyeballing a Grafana dashboard. It surfaced mid debugging session, on something unrelated, when a retrieval query returned the same chunk twice — worded identically, embedded identically — like the pipeline ran the same document through the same path twice. Except every log said it hadn’t. One ingestion event. One producer send. One success callback. Kafka’s own guarantee said this failure mode wasn’t supposed to exist. I spent a while assuming I’d misconfigured something. I hadn’t. There was a bug sitting underneath the guarantee itself — quietly breaking a promise Kafka’s been making since it introduced the idea in the first place. Here’s the thing nobody says out loud about “exactly-once delivery”: it’s not really about the delivery. It’s about deduplication with really good PR. Kafka can’t stop your producer from sending the same message twice — a network blip, a timeout, a retry, any of that can cause a duplicate send. What idempotence actually does is make sure the *broker* recognizes a resend for what it is and throws it away instead of processing it twice. Every producer gets a Producer ID, and every message on a partition gets a sequence number. If the broker’s already seen sequence number 47 from that producer on that partition, and 47 shows up again, it gets discarded, not reprocessed. That’s the entire trick. Not magic, not some distributed-consensus miracle — a counter and a broker that checks it. Transactions are the version of this that stretches across multiple partitions or topics at once, tying a set of writes together with a transaction coordinator so either all of them commit or none do. That’s what lets Kafka Streams claim end-to-end exactly-once — read a message, do some processing, write the result, commit the offset, all as one atomic unit. My pipeline used both. Chunking service produces to a topic, embedding service consumes it, and the moment an embedding was generated, that same service produced to another topic that pgvector’s writer consumed from. Idempotence on every hop, a transactional.id set per service instance, offsets committed only after the vector write succeeded. On paper, a document could not get embedded twice. The sequence numbers wouldn't allow it. Here’s roughly what that pipeline looked like: And on paper, none of that should have let a document through twice. My first instinct wasn’t “Kafka has a bug.” Nobody’s first instinct is that. Kafka is one of the most battle-tested pieces of infrastructure in existence — assuming you broke it yourself is the statistically correct prior. So I went through the checklist of “how exactly-once actually breaks,” because it does break, in well-documented ways that have nothing to do with a CVE. Rebalances were the first suspect — if a consumer restarts mid-processing and a partition gets reassigned before an offset commits, you can get reprocessing. I checked. My embedding service wasn’t rebalancing; it had been stable for hours before the duplicate showed up. Next was transactional.id fencing — reuse the same transactional ID across two producer instances and Kafka's coordinator will fence one of them off, sometimes in ways that look like silent failure if you're not watching for ProducerFencedException. I checked my producer initialization. One instance, one ID, no restarts, no fencing errors in the logs. Then I checked the boring stuff. max.in.flight.requests.per.connection — if you push this above 5 with idempotence on, you can lose ordering guarantees, and I've seen enough Stack Overflow threads to know it's a classic "I turned a knob without reading what it does" mistake. Mine was set correctly. I checked isolation.level on the consumer side, because if you're not reading read_committed, you'll see uncommitted transactional writes that later get aborted — which looks exactly like a duplicate if you're not paying attention. Also fine. ``` Properties producerProps = new Properties();producerProps.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, true);producerProps.put(ProducerConfig.MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION, 5);producerProps.put(ProducerConfig.TRANSACTIONAL_ID_CONFIG, "embedding-service-1");// one transactional.id per instance, no reuse across restartsProperties consumerProps = new Properties();consumerProps.put(ConsumerConfig.ISOLATION_LEVEL_CONFIG, "read_committed");// ignore uncommitted transactional writes from aborted producer transactions ``` Every documented failure mode for exactly-once semantics, and my setup didn’t match any of them. That’s the specific kind of frustrating that makes you start doubting your own logs, not the system you’re debugging. So I did what you do when the config is clean and the bug is still there: I stopped assuming the problem was in my code, and started reading Kafka’s own producer internals instead of my usage of them. Here’s what I found, and it’s uglier than a config mistake. Kafka’s producer keeps a shared pool of ByteBuffer objects to avoid allocating memory for every single batch of messages — a reasonable optimization, the kind you'd write yourself if you were building a high-throughput client. A batch gets a buffer, gets filled with records, gets sent over the network. Once the broker acknowledges it, the buffer goes back to the pool for the next batch to reuse. Normal, boring, correct. Except there’s a specific edge case — CVE-2026-35554 — where a batch can expire, via delivery.timeout.ms, or the broker connection can drop, while the network request for that batch is still in flight. When that happens, failBatch returns the buffer to the pool immediately. Not after the in-flight request finishes. Immediately. The next batch — for a completely different topic, going to a completely different partition — grabs that "free" buffer and starts writing its own records into it, while the original request is still out on the wire, still reading from that same memory. ``` private void failBatch(ProducerBatch batch, RuntimeException exception) { batch.done(-1L, RecordBatch.NO_TIMESTAMP, exception); free.deallocate(batch.buffer()); // buffer released back to the pool here — // but the in-flight NetworkClient request tied to this batch // hasn't actually completed. It still holds a reference // to this same ByteBuffer, and it's about to be overwritten.} ``` The original request eventually completes. But now it’s carrying whatever the second batch wrote into the buffer in the meantime. Wrong data, sent under the first batch’s identity, to the first batch’s topic. And here’s the part that made my stomach drop when I read the advisory: Kafka’s per-record CRC doesn’t cover the topic name. A corrupted payload that happens to still look like a valid record passes its checksum fine. It’s not garbled bytes that trip an error. It’s clean-looking data in the wrong place. That’s my duplicate. Not a duplicate at all, technically — a batch from my chunking service reused a buffer that a batch from an entirely different producer instance was still writing into, and the embedding pipeline received a payload that matched a chunk it had already processed, because the corrupted write happened to reconstruct something close enough to valid. No error. No failed checksum. No retry. Just a message that looked exactly like one I’d already seen. Here’s the buffer lifecycle that causes it: Every guarantee I’d checked — idempotence, transactional IDs, isolation levels — sits on top of this buffer pool. None of them know it exists. None of them are supposed to have to. Most CVEs you skim past. Version bump, changelog entry, move on. This one earned an actual re-read, for three reasons. First: the CRC. Kafka checksums the record payload, not the topic it’s headed to. So a corrupted buffer that happens to still parse as a valid record — which is exactly what you get when Batch B’s well-formed data overwrites Batch A’s buffer — sails through checksum validation without complaint. The corruption isn’t garbled bytes. It’s *coherent* data, just addressed to the wrong place. That’s a fundamentally harder failure to detect than “the bytes got mangled,” because nothing downstream has a reason to flag it. Second: this bug has existed since Kafka 2.8.0, not because someone introduced sloppy code recently, but because that’s the version where topic/partition routing metadata got split into a separate serialization buffer from the actual record payload — a refactor that almost certainly made the code cleaner and the client faster. Nobody shipped this race condition on purpose. It’s the kind of bug that hides inside a good decision for years. Third, and the one that actually made me sit up: the CVSS vector includes S:C — scope changed. That's not boilerplate severity scoring. It means the impact isn't contained to the producer that has the bug. It's every consumer with read access to whatever topic the corrupted data lands on. In my pipeline, that was just an embedding service reading its own duplicate. In a multi-tenant system, that's data from one tenant's topic potentially readable by a consumer that was never supposed to see it. The bug doesn't care about your access control model, because it happens beneath the layer where access control exists. A guarantee that holds “as long as nothing below it misbehaves” isn’t a guarantee. It’s a hope with good documentation. That’s the actual takeaway from this CVE, and it has nothing to do with my vector store: a guarantee that holds “as long as nothing below it misbehaves” isn’t a guarantee. It’s a hope with good documentation. The honest fix is boring: upgrade. kafka-clients 3.9.2+, 4.0.2+, or 4.1.2+ closes the race condition at the source. If you're pinned below that on any transitive dependency — spring-kafka, Kafka Streams, Flink's Kafka connector — you're exposed whether or not you ever typed the word "Kafka" into your own build file. That's worth actually checking, not assuming. But the upgrade isn’t the interesting part. The interesting part is what this bug did to how I think about the guarantee itself. I’d built the vector writer’s offset commit to depend entirely on Kafka’s exactly-once machinery being correct all the way down. That’s the mistake — not a config mistake, an architectural one. I was treating “exactly-once” as a property of the system instead of a property of the layers I could actually see. The buffer pool was never part of my mental model, because it isn’t supposed to be. It’s an implementation detail three layers below anything the client API exposes. And that’s exactly why a bug there is scarier than a bug in code I’m actually reading. So the pipeline looks a little different now. The vector writer still consumes idempotently, still commits offsets after a successful write — that part was never wrong. What’s new is a content hash on each embedding, computed independently of anything Kafka guarantees, checked against existing entries before an insert happens. Not because I stopped trusting Kafka’s idempotence. Because I stopped trusting *any single mechanism* to be the only thing standing between me and a duplicate, no matter how good its documentation sounds. This is the outbox-pattern instinct applied one layer earlier than usual — instead of only guarding the boundary where Kafka hands off to Postgres, I’m guarding the boundary where Kafka hands data to itself, hop to hop. It’s a small amount of extra work. A hash column, a lookup before an insert, maybe twenty lines of code. ``` // Before: trusting Kafka's idempotence alone@KafkaListener(topics = "embeddings.ready")public void onEmbeddingReady(EmbeddingEvent event, Acknowledgment ack) { vectorStore.insert(event.getChunkId(), event.getEmbedding()); ack.acknowledge();}// After: an independent guard, one layer earlier@KafkaListener(topics = "embeddings.ready")public void onEmbeddingReady(EmbeddingEvent event, Acknowledgment ack) { String contentHash = hash(event.getChunkText()); if (vectorStore.existsByContentHash(contentHash)) { ack.acknowledge(); // already have it — skip the insert, still commit return; } vectorStore.insert(event.getChunkId(), event.getEmbedding(), contentHash); ack.acknowledge();} ``` It’s also the kind of twenty lines you only write after something’s already gotten through once. I don’t think Kafka lied to me, exactly. I think I misread the promise. “Exactly-once” was never a claim about reality holding still. It’s a claim about a specific, narrow contract — producer to broker, broker to consumer, inside the boundaries Kafka actually controls. The moment you extend that trust one layer further than the contract covers, into a buffer pool implementation detail three abstractions below anything the client API exposes, you’re not relying on a guarantee anymore. You’re relying on hope that dresses well. That’s not a Kafka problem, really. It’s every distributed system’s problem, wearing a Kafka costume this time. Every guarantee anyone ever sells you — exactly-once, strong consistency, five nines — is a guarantee about the layer it was designed to cover, not the layers underneath it that you never had to think about until one of them broke. My pipeline doesn’t trust Kafka less now. It just doesn’t trust it *alone*. The content hash sitting between the embedding service and pgvector isn’t a vote of no confidence in idempotence — idempotence is still doing its job, most of the time, for most of the failure modes it was built for. It’s an admission that “most of the time” and “for the failure modes I know about” were never the same sentence as “exactly-once,” and I was the one who blurred them together. If there’s a version of this lesson that generalizes past Kafka, it’s this: the scariest bugs aren’t the ones in the code you wrote. They’re the ones in the code you trusted specifically *because* you didn’t have to think about it. Trust the guarantee. Just make sure you know exactly where it ends. [Your Kafka Producer Might be Lying to You Right Now](https://blog.stackademic.com/your-kafka-producer-might-be-lying-to-you-right-now-a3bfa031e8e1) was originally published in [Stackademic](https://blog.stackademic.com) on Medium, where people are continuing the conversation by highlighting and responding to this story.