Kafgres: Embedding a Kafka Broker into Postgres

Developer kafgres released an extension that embeds a singleton Kafka broker directly into PostgreSQL 16.15, implementing dozens of Kafka APIs so standard Kafka clients and admin tools treat the database as a regular broker. The extension supports producing events via SQL functions like kafgres_produce inside transactions and mapping table changes to topics through kafgres_add_mapping, aiming to preserve the Kafka ecosystem while avoiding a separate cluster or Debezium deployment. The author argues Postgres-based queues such as SELECT FOR UPDATE SKIP LOCKED or pgmq are sufficient at lower scales but discard Kafka's library ecosystem.

TL;DR “Just use Postgres” - Mostly compelling - But why throw away the Kafka ecosystem Dozens of blogs, listicles and wikis center around the concept of “just use Postgres’. Many have contrasted the operational burden and cost of managing a Kafka cluster against just using various techniques to get message queues or event logs out of your existing Postgres clusters. Proposed options range from using Postgres locks to handle competitive consumption from tables that can be treated as logs . SELECT FOR UPDATE SKIP LOCKED To using purpose built extensions extending psql for message queues js SELECT from pgmq.send queue name = 'my queue', msg = '{"foo": "bar1"}' ; These blog posts are broadly correct, in that Postgres especially when extended is sufficient for the vast majority of use-cases, especially at lower scales, and that relying on your existing database alleviates a large operational burden. One caveat not usually addressed is that relying on bespoke table-row-based queues or tailor-made Postgres extensions throws away the Kafka ecosystem, and leaves you with a smaller corpus of libraries to draw on during development. In many cases, that is a reasonable trade-off, but “writing wrappers around Postgres” is not most people’s end goal. I have been working on kafgres , an extension embedding a performant singleton Kafka broker into Postgres. Not only does doing this let you reuse your database as your log broker and store, it allows tight coupling between your event log and database activity, without needing to spin up yet another deployment i.e. debezium . bash $ psql -h Postgres -p 5432 -U Postgres -c "SELECT version " PostgreSQL 16.15 Debian 16.15-1.pgdg12+2 on aarch64-unknown-linux-gnu, compiled by gcc Debian 12.2.0-14+deb12u1 12.2.0, 64-bit $ kcat -b Postgres:9092 -L Metadata for all topics from broker 1: Postgres:9092/1 : 1 brokers: broker 1 at Postgres:9092 controller BEGIN; INSERT INTO orders id, customer id, total, status VALUES 1001, 1, 249.99, 'placed' ; SELECT kafgres produce 'order-events', 'order-1001', jsonb build object 'event', 'OrderPlaced', 'order id', 1001, 'customer', 'Acme Corp', 'total', 249.99 ::text AS offset assigned; COMMIT; Of course, Postgres already provides its own WAL, which is ultimately what debezium usually reads. If you’re more used to mapping events from the CDC than emitting via Postgres, you can do that as well. SELECT kafgres add mapping 'orders-cdc', -- mapping name 'public.orders', -- source table 'orders-cdc', -- destination topic value expr = $$ jsonb build object 'op', op, 'order id', new.id, 'total', new.total, 'status', new.status, 'customer', SELECT jsonb build object 'name', c.name, 'tier', c.tier FROM customers c WHERE c.id = new.customer id $$, key expr = $$ new.id::text $$, filter expr = $$ new.status < 'draft' $$ ; Kafgres isn’t just a server pointed at your tables doing this would be prone to causing WAL churn , it is built on a pluggable storage engine living in the Postgres engine, and it implements dozens of Kafka APIs, so that both Kafka clients and admin tools sees it as a regular, singleton Kafka broker. bash 1. an ordinary Kafka client produces $ echo 'order-3001:{"event":"OrderPlaced","order id":3001,"source":"Kafka-client"}' \ | kcat -b Postgres:9092 -t order-events -K: -P 2. a SQL transaction produces to the same topic $ psql -h Postgres -c "BEGIN; INSERT INTO orders VALUES 3002, 4, 1250.00, 'placed' ; SELECT kafgres produce 'order-events', 'order-3002', jsonb build object 'event','OrderPlaced','order id',3002,'source','sql-transaction' ::text ; COMMIT;" 3. one consumer reads both, and cannot tell them apart $ kcat -b Postgres:9092 -t order-events -C -e -q -o beginning -f '%p:%o %k - %s\n' 0:0 order-1001 - {"event": "OrderPlaced", "total": 249.99, "customer": "Acme Corp", "order id": 1001} 1:0 order-3001 - {"event":"OrderPlaced","order id":3001,"source":"Kafka-client"} 1:1 order-3002 - {"event": "OrderPlaced", "source": "sql-transaction", "order id": 3002} On a machine with an NVME and an i7 from 2017, a Postgres instance with kafgres installed is stable and fluid at a throughput of 30k-70k event/s. The DB remains responsive during this time, and with the default storage engine, we only see a few percent-point hit in pgbench performance. Kafgres versus: | Kafgres | Kafka | |---|---| | 0 new nodes | Often 3+ nodes, or SAAS subscription | | 30k-70k msg/s | 100k+ msg/s | | CDC out of the box | CDC via Debezium/etc | | Singleton, same HA posture as Postgres | Distributed | | "I want to use Kafka ecosystem, and don't need to worry about uber scale yet" | "We need a distributed log and/or to scale through 100k msg/s" | In an order pipeline demo https://github.com/RayElg/kafgres/tree/main/demo , we are able to reduce a 3-deployment event-driven architecture to one Postgres deployment, while still using standard Kafka libraries to build our microservices . | Normally | With Kafgres | |---|---| | Postgres: the business database | the same Postgres | | Debezium: change capture off the WAL | kafgres's CDC mappings kafgres add mapping | | Kafka: the broker services talk over | the same Postgres | | An outbox table plus a relay, to make a write and a publish atomic | kafgres produce inside the business transaction | I have recently released Kafgres 0.1.0 https://github.com/RayElg/kafgres , and will be continuing to iterate on the broker to chase down performance, safety, and utility gains. Misc. on the writing of Kafgres Kafgres is written using pgrx , a framework that I am a big fan of, allowing you to write Postgres extensions in Rust. While performance is important for the extension, considering we want a performant broker, the main appeal of Rust for this project is reducing the footprint of code that could crash the broker. If Kafgres were super prone to crashing, the appeal of collapsing two deployments to one would be defeated by the risk of bringing down your one Postgres deploy. I transported a lot of learnings from my previous broker-in-Postgres project, pgmqtt https://github.com/RayElg/pgmqtt . Where we diverge from the current version of pgmqtt is leaving a lot more internals off of the WAL. Although this means we need to write a lot more code, it means we can achieve a much higher throughput without crowding the other Postgres workers. Q : If we keep our topic off the WAL, how do we get transactional kafgres produce ? A : kafgres produce uses a commit marker row that does land on the WAL, and is inserted in the caller's transaction. The produced message doesn't land in that transaction, but the row that tells the kafgres engine whether the message commits does. This allows us to replicate Kafka's own aborted-transaction protocol. Q : If Postgres is a singleton, how do we handle replication? A : Take cues from Postgres. We stream replication data to a standby another kafgres, in another Postgres . Since we avoid using the WAL for topic data, the kafgres follower needs to pull the log using the Kafka fetch protocol as if it were reading from another, regular Kafka broker . We replicate Kafka's epoch reconciliation to handle divergent data.