{"slug": "the-startup-s-postgres-survival-guide", "title": "The startup's Postgres survival guide", "summary": "Hatchet has published an internal guide to preventing PostgreSQL from failing in production, distilled from two years of operational experience. The guide covers schema design, query optimization, the query planner, bulk updates, and autovacuum, assuming readers are familiar with SQL basics. It recommends using identity columns or built-in UUIDs for primary keys, always using timestamptz, and using foreign keys with cascading deletes for low-volume tables.", "body_md": "## The startup's Postgres survival guide\n\nA guide to preventing Postgres from toppling over.\n\nOver the past half year or so, I’ve been writing an internal doc for our engineers trying to distill two years of Postgres battles into a somewhat cohesive document. While I love [the Postgres manual](https://www.postgresql.org/docs/18/index.html), I find it’s hard to turn to when shit hits the fan because it’s just so darn comprehensive. I thought this might be useful for others and would appreciate feedback (or other tidbits that you’ve learned running Postgres in production).\n\nBefore starting Hatchet, while I was familiar with SQL, the extent of my knowledge was basically: if a query is slow, you need an index. That’s the starting point for this doc; I’m going to assume you’re familiar with SQL basics, rows, tables, and know roughly what an index is.\n\n*And if Claude is writing all of your queries, this might be a waste of time! I recommend supabase/agent-skills*\n\n[A quick note on ORMs](#a-quick-note-on-orms)\n\nThis guide should still be useful, but you might need to translate some of these tips into your ORM of choice. Lots of optimizations as you scale just aren’t possible with ORMs unless you can break past the abstraction layer and write SQL. You can do this gracefully or non-gracefully; [Prisma TypedSQL](https://www.prisma.io/docs/orm/prisma-client/using-raw-sql/typedsql) or equivalents look interesting for this. We use `sqlc`\n\nat Hatchet which gets us very similar behavior; highly recommend if you’re a Go stack.\n\n[Table of contents](#table-of-contents)\n\n[The simple stuff: good reads, writes and schemas](#the-simple-stuff-good-reads-writes-and-schemas)[Intermediate: the query planner, bulk updates, and autovacuum](#intermediate-the-query-planner-bulk-updates-and-autovacuum)[Some advanced stuff](#some-advanced-stuff)\n\n[The simple stuff: good reads, writes and schemas](#the-simple-stuff-good-reads-writes-and-schemas)\n\nLet’s start with the basics: queries and schemas at low volume.\n\n[Writing a good schema](#writing-a-good-schema)\n\nAfter you’re deployed, schemas are by far the hardest to change moving forward, so it’s worth spending some time on them. I’d recommend building your schema iteratively: start with a rough approximation for your tables and primary keys, then write some queries on those tables based on your application needs. You can approximate this with some questions: *Is this a high-read and/or high-write table? What are the most common filters on reads? Which columns am I updating the most?*\n\nIf you want to be more formal about it, you can look into [database normalization](https://www.digitalocean.com/community/tutorials/database-normalization) into 1NF/2NF/3NF, but I’ve found normal forms to sometimes be at odds with query efficiency and ease of use, which is critical when you’re moving fast—sometimes it’s just easier to dump data into a `jsonb`\n\ncolumn.\n\nMy rules of thumb for schemas are:\n\n- Use identity columns (auto-incrementing integers, slightly more performant than\n`bigserial`\n\n) or built-in UUIDs for primary keys - Always use\n`timestamptz`\n\n- Always use primary keys\n- Use foreign keys with cascading deletes for low-volume tables, particularly where database consistency and correctness are important. Careful at higher volume.\n\n[Writing good read queries](#writing-good-read-queries)\n\nLet’s start with `SELECT`\n\nqueries. A useful—albeit slightly inaccurate—mental model for fast selects is: under the hood, Postgres is either going to find a single row in a table very quickly, or it’s going to read every single row in your table using something called a *sequential scan* 😞.\n\nIt’s going to find a single row very quickly when you filter by:\n\n- An explicit index\n- A unique constraint (just a special case of index)\n- A primary key (these are automatically indexed in Postgres)\n\nIndexes by default use a `btree`\n\nimplementation. It’s most helpful to think of indexes as just another table in Postgres, with data stored in a specific format which is optimized for lookups (more on this later). These trees are great because finding a single row happens in approximately `log(n)`\n\ntime, where `n`\n\nis the number of rows in the table—in other words, really fast.\n\nWhen Postgres can’t use an index, it’ll use something called a sequential scan, or `seq scan`\n\n. Seq scans are much slower than index lookups, but modern databases are so fast at loading rows into memory that you probably won’t even notice at first: seq scans on tables with less than 20k rows are pretty much instant.\n\n[Writing performant joins](#writing-performant-joins)\n\nFor inner joins, there’s rarely an argument for not using primary keys as the inner join; it usually speaks to a schema design or normalization problem. Treat `ON`\n\nclauses with the same respect as a `WHERE`\n\nclause—the same principles apply. Use an index.\n\n[Compound indexes and aligning ORDER BY to your indexes](#compound-indexes-and-aligning-order-by-to-your-indexes)\n\nOften the first slow query in your application will be a list query across a large table. Something like:\n\nIn this case, you can use a compound index—a sensible one might be:\n\nIn more complex cases, a good rule of thumb is: the `ORDER BY`\n\ncolumns should be the last columns in the index, and you should align columns to the ordering in the `ORDER BY`\n\n. Note that Postgres can scan btrees in both directions, so sometimes the `DESC`\n\nis irrelevant—but for compound indexes it’s good practice. More information [here](https://www.cybertec-postgresql.com/en/benefits-of-a-descending-index).\n\n[Writing good write queries](#writing-good-write-queries)\n\nThe premise of successful writes is:\n\n**Keep transactions short.** Don’t go querying an external service in the middle of a transaction unless you have a really good reason to.**Be careful of the rows you’re locking** for writing; in other words, only lock what you need. Every time you update a row, you’re taking out a lock on that row for a short period of time until the transaction commits.\n\nAs your system gets busier, you’re going to start noticing the impact of locks more. In particular, you might try to create an index at some point in the future with a simple `CREATE INDEX`\n\ncommand: turns out this locks your table and prevents inserts and updates! When creating an index on an existing large table, always use `CREATE INDEX CONCURRENTLY`\n\n.\n\n[Migrations](#migrations)\n\nGetting really good at writing migrations is an important technical advantage: it helps you iterate much faster and increases your uptime. As a starting point, try to keep migrations additive (in other words, don’t delete or remove columns) and run them in a transaction wherever possible; this will make rollbacks and partial migrations much easier to deal with. As you get more advanced, you can start looking into [expand and contract](https://martinfowler.com/bliki/ParallelChange.html) migrations.\n\nThe simplest mental model for good migrations is: does this block all of my writes, or does it not? Creating an index without `CONCURRENTLY`\n\nblocks all your writes, so you might see downtime. Generally, operations which call `ALTER TABLE`\n\nshould be worth a second look; for example, adding a new check constraint to a very large table can block your writes as well (unless you add it with the `NOT VALID`\n\nkeyword).\n\n[Connection management](#connection-management)\n\nEvery time you execute a transaction or query against your database, you’re utilizing a connection. Connections are expensive in a number of dimensions (cpu and memory), and high connection churn can lead to a lot of unnecessary resource waste, so connections should be long-lived. Connection storms (when you start using up a ton of new connections at the same time) can also lead to very hard to debug edge cases related to internal Postgres locks.\n\nBecause of all these connection footguns, external connection poolers like pgbouncer are great! If you can’t add this for whatever reason, in-memory connection poolers are a great second option. For example, because Hatchet is open-source, we don’t assume that all user databases use connection poolers, so we use [pgxpool](https://pkg.go.dev/github.com/jackc/pgx/v5/pgxpool) (an in-memory connection pool for Go) for this purpose.\n\n[Intermediate: the query planner, bulk updates, and autovacuum](#intermediate-the-query-planner-bulk-updates-and-autovacuum)\n\n[Introducing the leakiest of abstractions, the query planner](#introducing-the-leakiest-of-abstractions-the-query-planner)\n\nAt a certain point, your queries might become complex enough that a simple index won’t cut it (and you shouldn’t endlessly add indexes to your tables—they come with overhead). The queries might involve many `JOIN`\n\nstatements or different types of joins where the correct path for querying the data isn’t clear.\n\nAt this point, you will need to concern yourself with the *query planner*. At best, the query planner is a [leaky abstraction](https://www.joelonsoftware.com/2002/11/11/the-law-of-leaky-abstractions/). It’s an internal implementation, and you have virtually no control over it, but you have to know its spontaneous and sometimes irrational behavior. It’s like working with an LLM!\n\nThe query planner looks at the query you pass in, and it figures out how it should translate your query into a set of internal operations in the database. For example, it might look at your query, and realize that it needs to use an index. In an ideal world, the query planner would know, for every query and set of parameters, the perfect plan to use. But the query planner is operating on limited information, and sometimes it doesn’t pick the best option.\n\nThis limited information is the table statistics. You can actually query it directly in Postgres:\n\nThese statistics are collected for every `ANALYZE`\n\n. This also happens when autovacuum is run (see below), so more frequent autovacuums also mean that your query statistics will be more up to date. A common reason why your query is behaving improperly is not analyzing frequently enough.\n\nThe reason I think it’s useful to view queries as binary—they either seq scan or they don’t seq scan—is: the more you micro-optimize a query, the more of a risk you take that the query planner goes rogue. If you stick to querying by primary keys and indexes, the query planner will have a much easier time.\n\nLet’s say that there’s nothing obviously wrong in your query, but it’s still slow—how do you go about debugging this? Some Postgres database providers (like Google CloudSQL) will sample your queries and save slow ones—but many don’t. This is where `EXPLAIN ANALYZE`\n\nis your friend. This outputs the query plan for the query and executes the query (careful running this in production—you can use `EXPLAIN`\n\nwithout `ANALYZE`\n\nto get a query plan), and then compares its estimates based on the table statistics to the actual number of rows scanned. I usually place my sql query in a file, prefix it with `EXPLAIN (ANALYZE, COSTS, VERBOSE, BUFFERS, FORMAT JSON)`\n\nand run:\n\nAnd then use [explain.dalibo.com](https://explain.dalibo.com) to visualize the execution plan.\n\n[Sometimes it just makes sense to seq scan](#sometimes-it-just-makes-sense-to-seq-scan)\n\nThere are cases where you think an index should be used, but the query planner is still seq scanning anyway, despite table statistics being up to date and the index being valid. In these cases, Postgres is usually estimating that the cost of the seq scan will be smaller than the cost of the index scan. Index scans do come with some overhead; indexes are stored separately from the actual data in the table (called the heap)—finding all of the rows in the heap can be expensive!\n\nUnless you can dramatically restructure your query, you might have to accept that it’s going to seq scan, or think about something like partitioning (more on that below).\n\n[Writing lots of data](#writing-lots-of-data)\n\nLet’s say your application is scaling and you need to write a lot of data fast. Each query has some overhead associated with it (separate from the connection overhead we talked about before): this includes the round-trip time to the database, the time it takes the internal application connection pool to acquire a connection, and the time it takes Postgres to process the query (including a set of [internal Postgres locks](https://github.com/postgres/postgres/blob/master/src/backend/storage/lmgr/README) which can be bottlenecks in high-throughput scenarios).\n\nTo reduce this overhead, we can pack a batch of rows into each query. The simplest way to do this is to send all queries to the Postgres server at once in an implicit transaction (in Go, we can use `pgx`\n\nto execute a `SendBatch`\n\n). Batching is very powerful: we found that it can ~10× your throughput. I wrote more about this plus some other tips for writing data quickly [here](https://hatchet.run/blog/fastest-postgres-inserts).\n\n[Default autovacuum settings can kill your database](#default-autovacuum-settings-can-kill-your-database)\n\nAutovacuum is a critical operation in Postgres databases that sometimes needs to be tuned, especially in high-write scenarios. The autovacuum daemon is responsible for a number of things, including cleaning up dead tuples and managing transaction ids.\n\nWhat’s a *dead tuple*? A tuple is an instance of a row on the filesystem. Every time you update or delete a row, a version of that row is left in Postgres until all transactions which started before that row was updated or deleted have committed or rolled back. These rows which can no longer be read by any transactions are *dead tuples*.\n\nIf you’re writing data quickly enough, sometimes autovacuum can’t keep up, which will get you into a very unhealthy state, very quickly. You’ll see this when you query for active processes on the database:\n\nIf you see an autovacuum query running for more than ~1 hour, you might want to consider changing your autovacuum settings! See [this article](https://www.cybertec-postgresql.com/en/tuning-autovacuum-postgresql/) for more information.\n\nIt’s worth monitoring this: if you use up all transaction ids in the system before they can be reclaimed by autovacuum, you’ll reach a dreaded state called **transaction id wraparound**. This will mean a big chunk of downtime.\n\n[Other types of bloat](#other-types-of-bloat)\n\nBesides dead tuples, there are two other kinds of bloat you’ll often encounter in a busy Postgres system:\n\n- Table bloat caused by partially filled pages. Postgres stores rows on pages on disk, each of which are 8kb in size. When Postgres can’t fit new rows onto an existing page, it creates a new one. But when dead tuples are reclaimed, this can lead to pages not being entirely filled, which can increase the disk usage of Postgres, sometimes significantly. The best way to avoid table bloat is by tuning autovacuum before you’re bloated. But there are some extensions to help with bloated tables, like\n`pg_repack`\n\n, because the built-in Postgres`VACUUM FULL`\n\nis rarely a good idea. Note that Postgres 19 is getting`REPACK...CONCURRENTLY`\n\n, which I haven’t tested, but seems like potentially a good solution for concurrent table repacking. - Index bloat is a special case of table bloat, and is similarly solved by good autovacuum settings. But Postgres has a built-in command for dealing with this, which is\n`REINDEX INDEX CONCURRENTLY`\n\n.\n\n[Some advanced stuff](#some-advanced-stuff)\n\nI wanted to end with a set of advanced Postgres features which have been particularly useful for us at Hatchet.\n\n[FOR UPDATE SKIP LOCKED](#for-update-skip-locked)\n\nThe best way to think about this Postgres feature is that it reserves the rows that you’re selecting for use in your transaction without interfering with other queries. We use it primarily for implementing our [job queue](https://hatchet.run/blog/multi-tenant-queues); a single-query queue in Postgres can be implemented like this:\n\nIt’s also very useful in cases where you’re doing many independent updates of rows, or you’re managing leases on objects in your system across many instances of your application (for example, we use this to distribute tenant leases across Hatchet engines).\n\n[Partitioning](#partitioning)\n\nPostgres has built-in partitioning which allows you to subdivide your tables based on row values, like timestamps or hashes. This can be incredibly useful for time-series data (in our case, historical task data) because:\n\n- Each partition can be autovacuumed independently, allowing you to scale up autovacs on your table\n- Deleting old data is near-instant—you simply drop the table partition, rather than iterating through rows\n\nPartitioning does come with drawbacks: there can be overhead on read queries if Postgres doesn’t prune partitions during the planning phase (something that Postgres has gotten much better at in recent releases).\n\nI’ve written more about our experience with partitioning [here](https://hatchet.run/blog/postgres-partitioning).\n\n[Tricks for large table migrations](#tricks-for-large-table-migrations)\n\n(note that this isn’t referring to a database migration, but rather moving large amounts of data from one table to another, which we find ourselves doing a few times a year)\n\nIf you try to migrate really large tables, it can take many hours to copy data in a single transaction. This isn’t good—long-running transactions prevent autovacuum from doing its job properly, and therefore make the entire system bloated with dead tuples. Also if you want to keep writing to the old tables, the new table won’t see the new data.\n\nSo we need to figure out how to migrate data safely without a transaction, with new writes (after the migration has started) being copied to the new table. One of the tricks we’ve learned is to use Postgres triggers and run a large batched backfill outside of a transaction, using unique constraints on the primary key to prevent duplicate writes.\n\nThat’s it! If you have any other tidbits about Postgres scaling—or questions about Postgres scaling in general—feel free to reach out.", "url": "https://wpnews.pro/news/the-startup-s-postgres-survival-guide", "canonical_source": "https://hatchet.run/blog/postgres-survival-guide", "published_at": "2026-07-22 12:36:08+00:00", "updated_at": "2026-07-22 12:52:39.889220+00:00", "lang": "en", "topics": ["developer-tools"], "entities": ["Hatchet", "PostgreSQL", "Prisma", "sqlc"], "alternates": {"html": "https://wpnews.pro/news/the-startup-s-postgres-survival-guide", "markdown": "https://wpnews.pro/news/the-startup-s-postgres-survival-guide.md", "text": "https://wpnews.pro/news/the-startup-s-postgres-survival-guide.txt", "jsonld": "https://wpnews.pro/news/the-startup-s-postgres-survival-guide.jsonld"}}