{"slug": "show-hn-puffgres-logically-replicates-postgres-entities-in-turbopuffer", "title": "Show HN: Puffgres logically replicates Postgres entities in turbopuffer", "summary": "A24 Labs released puffgres, a beta logical replication service that mirrors Postgres entities in turbopuffer, automatically syncing vector embeddings without extra database calls. The tool handles at-least-once delivery, uses immutable configs and transforms, and aims to prevent data drift between primary and vector databases.", "body_md": "If you have feedback about puffgres, or are interested in building tools for the future of film and TV production, A24 Labs is hiring software engineers — email [lgelfond@a24films.com](mailto:lgelfond@a24films.com) for more.\n\npuffgres (beta) is a logical replication service that keeps Postgres entities mirrored in turbopuffer. Rather than duplicating application code every time you modify a vector (and risking partial successes that keep data out of sync), your Postgres changes automatically update.\n\nA bit of puffgres' design philosophy:\n\n**You should not need extra database calls to keep vectors up to date**. Upserting rows in your primary database and a secondary vector database is bound to produce drift (forgetting to add parallel / compensating calls) and hard-to-detect failures (i.e. just one of the two calls succeeds). puffgres lets us \"derive\" state, making Postgres the source of truth and keeping Turbopuffer in sync.**The service handles at-least once delivery**. Developers should not need to consider batching, retry logic, backfills, or change data capture in any of the code that they write. The service maintains its own state in a dedicated`puffgres`\n\nschema inside your source Postgres, and can stop/start/resume at any time without losing changes (even if they are slightly out of date). Co-locating state with the source means PITR restores naturally roll the two together.**Sync is maintained through \"configs\" which link Postgres tables to turbopuffer namespaces.** Each defines a mapping, and a TypeScript-based \"transform,\" which lets us easily do operations like tokenization, embedding, and other manipulation.**Configs and transforms are immutable**. We avoid an abundance of thorny cases that come from letting us change a mapping (i.e. rows produced with two different set of transforms.). If we want to make a change, we should \"tombstone\" the old one and create a new one.\n\nRead our [docs](https://a24films.github.io/puffgres/) to get started.\n\nInstall the CLI as a native binary — under the hood verifies you have the Rust toolchain and builds from source + adds `puffgres`\n\nto your PATH.\n\n```\ncurl -fsSL https://raw.githubusercontent.com/a24films/puffgres/main/install.sh | sh\n```\n\nThe docs are published as one file at [https://a24films.github.io/puffgres/AGENTS.md](https://a24films.github.io/puffgres/AGENTS.md). Install it as a skill — paste one of these:\n\n```\n# Claude Code\nmkdir -p ~/.claude/skills/puffgres && curl -fsSL https://a24films.github.io/puffgres/AGENTS.md -o ~/.claude/skills/puffgres/SKILL.md\n# Codex\nmkdir -p ~/.codex/skills/puffgres && curl -fsSL https://a24films.github.io/puffgres/AGENTS.md -o ~/.codex/skills/puffgres/SKILL.md\n```\n\nWe built this because we needed vector embeddings internally and had read compelling evidence [pgvector was a bad solution](https://alex-jacobs.com/posts/the-case-against-pgvector/) because of performance hits to maintain indexes, poor filtered queries, etc. Our naive / base solution was very hacky; we kept a separate table everytime we kept something in turbopuffer that kept an `id`\n\n, `turbopuffer_updated_at`\n\n, and `updated_at`\n\nand would simply embed / upsert whenever `updated_at`\n\nwas more recent. This meant a full table scan whenever our pipeline ran (very inefficient) and effectively polling for changes in turbopuffer. It didn't handle deletes, required tons of duplicative code, and meant all updates only happened when the pipeline ran.\n\nThis was inspired by two tech talks: Martin Kleppmann's [Turning the database inside out with Apache Samza](https://martin.kleppmann.com/2015/03/04/turning-the-database-inside-out.html), that argues strongly for making changes in one place and having derived data act like a materialized view, and Bryan Cantrill's [Sharpening the Axe: The Primacy of Toolmaking](https://www.youtube.com/watch?v=_GpBkplsGus), which suggests companies are well-suited investing in (and releasing) generic tools when they find themselves doing repeated work.\n\nI built a [very hacky](https://github.com/lucasgelfond/puffgres) version of this over a weekend, starting with a detailed spec and working through it with Claude. When we decided to use it in-house, I broke it up into PRs, and, especially at the beginning, ran each through code review + traditional testing, CI, etc. We've been running puffgres internally for a bit now without issue, and after talking with the turbopuffer team figured others might find use in it as well.\n\nThese benchmarks are a bit artificial, in isolation on a GitHub action (`ubuntu-latest`\n\n, the 4-core x86 / 16GB RAM runner). In practice, we've used `puffgres`\n\non tables of a few hundred thousand rows, although it should scale much beyond this. If you implement this at large scale or hit bumps, feel free to shoot me an [email](mailto:lgelfond@a24films.com). Initial benchmarking on GitHub Actions runners shows:\n\n**Throughput**: >600K events/sec sustained over 100M events** Batch latency**: p50 <10µs, p99 <100µs across 100K transactions** Recovery**: <60ms to resume from checkpoint after crash** Memory**: <160 bytes/event at scale, and total memory usage grows sub-linearly with event volume** Router fanout**: >1M source events/sec across 1000 configs\n\npuffgres is a Rust workspace divided into several crates under `crates/`\n\n:\n\n— Postgres setup, creates / manages publication and slot, generates schema files from table definitions, and runs backfill queries to bring in data`pg`\n\n— the change data capture stream. Decodes Postgres logical replication protocol, manages caching relations, schema changes + in-transaction batching.`replication`\n\n— routes change events to their respective configs, runs transforms via a TypeScript subprocess, manages retry logic / dead letter queue, and calls to the puff client`core`\n\n— turbopuffer API client, light wrapper around`puff`\n\n`rs-puff`\n\n— stores persistent information (i.e. streaming replication checkpoints, backfill progress, failed entry queue) in a dedicated Postgres schema (by default:`state`\n\n`puffgres`\n\n).— the`cli`\n\n`puffgres`\n\nbinary, handles subcommands, environment setup, orchestration, and default / template files.— definitions, parsing, validation, and hashing of config files.`config`\n\n— light server / web UI to inspect turbopuffer / WAL contents, just easier than the turbopuffer dashboard / making a bunch of cURLs`debug`\n\nDocumentation lives in `docs/`\n\nand is built with [mdbook](https://rust-lang.github.io/mdBook/).\n\nBuild the binary from source:\n\n- Install\n[Just](https://github.com/casey/just#installation). - Install the\n[Rust toolchain](https://rust-lang.org/tools/install/). - Build and install\n`puffgres`\n\ninto`~/.cargo/bin`\n\n:\n\n```\njust install\n\n# To overwrite an existing install\njust reinstall\n# Unit tests\ncargo test --workspace --lib\n\n# Integration tests (requires Postgres)\ncargo test --workspace\ncargo bench --package replication --bench decoder_bench\n```\n\nFuzz targets live in `fuzz/`\n\nand use [cargo-fuzz](https://github.com/rust-fuzz/cargo-fuzz).\n\n```\ncargo install cargo-fuzz\ncargo +nightly fuzz run fuzz_decoder\n\n# Regenerate seed corpus\ncd fuzz && cargo run --bin generate_seeds\n```\n\n", "url": "https://wpnews.pro/news/show-hn-puffgres-logically-replicates-postgres-entities-in-turbopuffer", "canonical_source": "https://github.com/a24films/puffgres", "published_at": "2026-07-16 19:14:24+00:00", "updated_at": "2026-07-16 19:25:10.318301+00:00", "lang": "en", "topics": ["developer-tools", "ai-infrastructure"], "entities": ["A24 Labs", "turbopuffer", "Postgres", "puffgres", "Martin Kleppmann", "Bryan Cantrill"], "alternates": {"html": "https://wpnews.pro/news/show-hn-puffgres-logically-replicates-postgres-entities-in-turbopuffer", "markdown": "https://wpnews.pro/news/show-hn-puffgres-logically-replicates-postgres-entities-in-turbopuffer.md", "text": "https://wpnews.pro/news/show-hn-puffgres-logically-replicates-postgres-entities-in-turbopuffer.txt", "jsonld": "https://wpnews.pro/news/show-hn-puffgres-logically-replicates-postgres-entities-in-turbopuffer.jsonld"}}