Before You Add Kafka, Redis, and Elasticsearch: Try One Postgres First

A developer in Dhaka built an AI agent infrastructure using a single Postgres database for job queuing, caching, and full-text search instead of adopting Redis, RabbitMQ, and Elasticsearch. The approach, which uses SKIP LOCKED for queues, unlogged tables for cache, and tsvector with GIN indexes for search, is gaining traction as highlighted by the 'PostgreSQL for Everything' essay and examples from Contentful, Instacart, and The Guardian.

Last month I was building out my own AI agent infrastructure, the side project where I run a few agents that research, draft, and publish content. The agents needed three things beyond the main database: a job queue, a cache for repeated lookups, and search over article text. My instinct, six years of Spring Boot muscle memory, was to reach for the usual stack. RabbitMQ or Redis Streams for the queue. Redis for the cache. Elasticsearch or Meilisearch for search. Then I counted the services. One Spring Boot app plus three more moving parts, each with its own Docker image, its own failure modes, its own 3 AM pager behavior. For a system serving a handful of agents, that felt like hiring an orchestra to play a ringtone. So I tried something that old-school database people keep telling us and we keep ignoring: I did all three jobs in Postgres. A table with SKIP LOCKED for the queue. An unlogged table for the cache. A tsvector column with a GIN index for search. The whole thing is one schema, one backup strategy, one connection pool. This idea is having a moment again. An essay titled "PostgreSQL for Everything" by Raphael Bauer has been on the Hacker News front page https://www.raphaelbauer.com/posts/postgresql-everything/ this week, and it makes the case I stumbled into: before you adopt a new specialized system, ask whether Postgres already does that job well enough. Contentful rebuilt their full-text search on Postgres https://www.contentful.com/blog/contentful-faster-full-text-search/ instead of a separate search cluster. Instacart built a modern search infrastructure on Postgres https://tech.instacart.com/how-instacart-built-a-modern-search-infrastructure-on-postgres-c528fa601d54 . The Guardian famously migrated off MongoDB onto Postgres https://www.theguardian.com/info/2018/nov/30/bye-bye-mongo-hello-postgres for parts of their platform. Those are big companies. I am one developer in Dhaka with a VPS. But that is exactly the point. The smaller your team, the more a single database that does five jobs is worth. Here is the Spring Boot version of "Postgres for everything", the patterns I actually used, with code. Full disclosure up front: I have run these patterns in my own projects, not at giant scale. I will point out where each one breaks down as you grow, because some of them do. The workhorse. You have a jobs table, workers poll it, and each job must be claimed by exactly one worker. The naive approach, SELECT then UPDATE WHERE status = 'pending' , races between workers. Postgres has had the fix since 9.5: SELECT ... FOR UPDATE SKIP LOCKED . Each worker locks the rows it grabs, and rows already locked by another worker are simply skipped, so two workers never claim the same job. No advisory locks, no leader election, no broker. The Postgres docs cover this under row-level locking https://www.postgresql.org/docs/current/explicit-locking.html LOCKING-ROWS , and Crunchy Data has a good deep dive on queuing with native Postgres https://www.crunchydata.com/blog/message-queuing-using-native-postgresql . The entity: @Entity @Table name = "jobs", indexes = @Index name = "idx jobs pending", columnList = "status, runAt" public class Job { @Id @GeneratedValue strategy = GenerationType.UUID private UUID id; private String type; // "draft-article", "fetch-stats", ... private String status; // pending, running, done, failed @Column columnDefinition = "jsonb" private String payload; private Instant runAt; private Instant lockedAt; private int attempts; } The claim query, as a native query in your Spring Data repository: public interface JobRepository extends JpaRepository<Job, UUID { @Query value = """ UPDATE jobs SET status = 'running', locked at = now WHERE id IN SELECT id FROM jobs WHERE status = 'pending' AND run at <= now ORDER BY run at FOR UPDATE SKIP LOCKED LIMIT :batch RETURNING """, nativeQuery = true List<Job claimJobs @Param "batch" int batch ; } One statement. FOR UPDATE SKIP LOCKED inside the subquery means concurrent workers each get a disjoint batch, even if they run at the same instant. RETURNING hands you the claimed rows without a second round trip. Then a poller, which is just a scheduled method: @Scheduled fixedDelay = 2000 public void poll { List<Job jobs = repository.claimJobs 5 ; jobs.forEach processor::handle ; // handles retry/backoff on failure } That is a durable job queue in about forty lines. Your jobs survive restarts because they live in a table you already back up. Compare that to bootstrapping RabbitMQ, writing consumers, configuring dead-letter exchanges, and explaining to your future self how prefetch works. Where it breaks: this gives you at-least-once delivery with polling latency, not push, and not Kafka-style replayable ordered logs. If you need millions of jobs per minute, streaming semantics, or long retention for event replay, use a real broker. Bauer's advice in his essay is the right heuristic: start with Postgres, and only swap in Kafka or RabbitMQ when it demonstrably stops performing. One more production detail from experience: add a reaper. If a worker dies mid-job, rows stay running forever. A scheduled job that flips running rows older than some timeout back to pending and bumps attempts completes the story. Everyone reaches for Redis here. But a cache has a defining property: it can be lost. Postgres has a table type for exactly that, UNLOGGED . Writes skip the write-ahead log, which is most of the write cost, so it gets you dramatically closer to cache-like latency while keeping the query language and tooling you already have. The CREATE TABLE docs https://www.postgresql.org/docs/current/sql-createtable.html spell out the trade: an unlogged table is not crash-safe, it gets truncated after a crash. For a cache, that is not a bug, it is a cold start. CREATE UNLOGGED TABLE cache entry key text PRIMARY KEY, value jsonb NOT NULL, expires at timestamptz NOT NULL ; The Spring side is trivial, one repository method: @Query value = """ SELECT value FROM cache entry WHERE key = :key AND expires at now """, nativeQuery = true String get @Param "key" String key ; TTL handling can be a trigger on write, as Martin Heinz describes in his Postgres-as-cache write-up https://martinheinz.dev/blog/105 , or the simpler version I use: a @Scheduled job deleting expired rows every minute. Lazy and unglamorous, and in a week of running my agent lookups it never once mattered. In my usage, agent lookups that used to cost an API call land in this table and repeated reads come back in low single-digit milliseconds, which for my workload is indistinguishable from a network hop to Redis. Your workload will differ; measure yours before believing mine. Where it breaks: no pub/sub invalidation across app instances, no eviction policies as sophisticated as Redis LRU, and it is one machine's cache, not a shared cluster. The moment you have many app nodes hammering one Postgres for cache reads, you are spending your database's capacity on cache traffic. That is the point to graduate to Redis, not before. This is the one with the strongest big-company evidence. Contentful replaced their search infrastructure with Postgres full-text search and wrote about the results https://www.contentful.com/blog/contentful-faster-full-text-search/ . Instacart did the same for their search stack https://tech.instacart.com/how-instacart-built-a-modern-search-infrastructure-on-postgres-c528fa601d54 . The core idea is a generated column of type tsvector plus a GIN index, described in the Postgres text search docs https://www.postgresql.org/docs/current/textsearch.html . First the schema: ALTER TABLE article ADD COLUMN search vector tsvector GENERATED ALWAYS AS setweight to tsvector 'english', coalesce title, '' , 'A' || to tsvector 'english', coalesce body, '' STORED; CREATE INDEX idx article search ON article USING GIN search vector ; Title matches rank higher because of the A weight. Then query it from Spring: @Query value = """ SELECT id, title, ts rank search vector, query AS score FROM article, websearch to tsquery 'english', :q AS query WHERE search vector @@ query ORDER BY score DESC LIMIT 20 """, nativeQuery = true List<ArticleSearchHit search @Param "q" String q ; websearch to tsquery accepts search terms the way users type them, quoted phrases included, instead of the arcane to tsquery syntax. No data sync problem, because the index lives next to the data. When I searched my own article corpus, a few thousand rows, results came back faster than anything I could perceive, and relevance was entirely acceptable for a site search box. Where it breaks: no typo tolerance or semantic matching out of the box, and relevance tuning is nothing like Elasticsearch's. At tens of millions of documents or heavy faceting needs, a dedicated engine earns its keep. A few thousand to a few million rows with plain keyword search: Postgres wins on operations alone, one less system to keep in sync with your primary data. The Guardian's migration story is instructive because they were running MongoDB at real scale and moved to Postgres https://www.theguardian.com/info/2018/nov/30/bye-bye-mongo-hello-postgres partly to reduce operational surface. The jsonb type https://www.postgresql.org/docs/current/datatype-json.html stores documents, indexes inside them with GIN, and queries them with path operators. In Spring Boot, annotate a field with @Column columnDefinition = "jsonb" , map it with a Hibernate AttributeConverter , and you have schema-flexible documents inside your relational database. I use this for agent payloads and tool outputs, shapes that drift as I experiment. But I keep the columns I actually query on as real typed columns. JSONB is a sharp knife: wonderful for "store this blob of shape X", risky as a foundation for a heavily queried data model where you lose constraints, foreign keys, and clear types. The pragmatic pattern is typed columns for what you filter and join on, JSONB for the parts that legitimately vary. The pattern most relevant to what I actually do in 2026. The pgvector extension https://github.com/pgvector/pgvector turns Postgres into a vector database with HNSW indexes, and Spring AI ships a PgVector vector store implementation, so your RAG embeddings can live in the same database as everything else. For a personal agent setup like mine, that means similarity search without running a separate Pinecone or Qdrant instance. Same rule applies: at serious scale or with heavy multi-tenant filtering, dedicated vector databases justify themselves. For thousands to low millions of embeddings, one Postgres is fewer systems to babysit. Here is the decision checklist I now run through before adopting any new data infrastructure. Copy it into your next design doc. SKIP LOCKED is enough when you have up to a few thousand jobs per minute and can tolerate poll latency. Move to a broker when you need streaming, fan-out to external systems, or replayable event logs.The common thread: each swap should be triggered by a measured limit, not by an architecture diagram that looks more impressive with more boxes. If I were starting my agent infrastructure today, I would begin with Postgres doing every job from day one, and I would treat each specialized system as an extraction to be earned by evidence. That is roughly the opposite of how I actually did it, which was to sketch a Kafka-shaped diagram first because it looked "production-grade". What makes a system production-grade is that you can operate it at 3 AM, and one database with good backups beats four systems you half-understand. The honest caveat: "Postgres for everything" is a starting position, not a religion. The companies I cited above did not stop at Postgres because it was trendy; they measured, and Postgres met the bar for their workload at the time. Do the same measurement for yours. Have you shipped a Postgres-only queue or replaced a search cluster with tsvector ? What scale did it hold up to before you had to move on? I would genuinely like to know where the walls are, so drop your experience in the comments. I write about Java, Spring Boot, and AI every week, mostly things I broke myself so you do not have to. Subscribe, it is free.