{"slug": "before-you-add-kafka-redis-and-elasticsearch-try-one-postgres-first", "title": "Before You Add Kafka, Redis, and Elasticsearch: Try One Postgres First", "summary": "A developer in Dhaka built an AI agent infrastructure using a single Postgres database for job queuing, caching, and full-text search instead of adopting Redis, RabbitMQ, and Elasticsearch. The approach, which uses SKIP LOCKED for queues, unlogged tables for cache, and tsvector with GIN indexes for search, is gaining traction as highlighted by the 'PostgreSQL for Everything' essay and examples from Contentful, Instacart, and The Guardian.", "body_md": "Last month I was building out my own AI agent infrastructure, the side project where I run a few agents that research, draft, and publish content. The agents needed three things beyond the main database: a job queue, a cache for repeated lookups, and search over article text. My instinct, six years of Spring Boot muscle memory, was to reach for the usual stack. RabbitMQ or Redis Streams for the queue. Redis for the cache. Elasticsearch or Meilisearch for search.\n\nThen I counted the services. One Spring Boot app plus three more moving parts, each with its own Docker image, its own failure modes, its own 3 AM pager behavior. For a system serving a handful of agents, that felt like hiring an orchestra to play a ringtone.\n\nSo I tried something that old-school database people keep telling us and we keep ignoring: I did all three jobs in Postgres. A table with `SKIP LOCKED`\n\nfor the queue. An unlogged table for the cache. A `tsvector`\n\ncolumn with a GIN index for search. The whole thing is one schema, one backup strategy, one connection pool.\n\nThis idea is having a moment again. An essay titled \"PostgreSQL for Everything\" by Raphael Bauer has been on the [Hacker News front page](https://www.raphaelbauer.com/posts/postgresql-everything/) this week, and it makes the case I stumbled into: before you adopt a new specialized system, ask whether Postgres already does that job well enough. Contentful [rebuilt their full-text search on Postgres](https://www.contentful.com/blog/contentful-faster-full-text-search/) instead of a separate search cluster. Instacart [built a modern search infrastructure on Postgres](https://tech.instacart.com/how-instacart-built-a-modern-search-infrastructure-on-postgres-c528fa601d54). The Guardian famously [migrated off MongoDB onto Postgres](https://www.theguardian.com/info/2018/nov/30/bye-bye-mongo-hello-postgres) for parts of their platform.\n\nThose are big companies. I am one developer in Dhaka with a VPS. But that is exactly the point. The smaller your team, the more a single database that does five jobs is worth. Here is the Spring Boot version of \"Postgres for everything\", the patterns I actually used, with code.\n\nFull disclosure up front: I have run these patterns in my own projects, not at giant scale. I will point out where each one breaks down as you grow, because some of them do.\n\nThe workhorse. You have a `jobs`\n\ntable, workers poll it, and each job must be claimed by exactly one worker. The naive approach, `SELECT`\n\nthen `UPDATE WHERE status = 'pending'`\n\n, races between workers. Postgres has had the fix since 9.5: `SELECT ... FOR UPDATE SKIP LOCKED`\n\n. Each worker locks the rows it grabs, and rows already locked by another worker are simply skipped, so two workers never claim the same job. No advisory locks, no leader election, no broker. The Postgres docs cover this under [row-level locking](https://www.postgresql.org/docs/current/explicit-locking.html#LOCKING-ROWS), and Crunchy Data has a good [deep dive on queuing with native Postgres](https://www.crunchydata.com/blog/message-queuing-using-native-postgresql).\n\nThe entity:\n\n```\n@Entity\n@Table(name = \"jobs\", indexes =\n    @Index(name = \"idx_jobs_pending\", columnList = \"status, runAt\"))\npublic class Job {\n    @Id @GeneratedValue(strategy = GenerationType.UUID)\n    private UUID id;\n\n    private String type;          // \"draft-article\", \"fetch-stats\", ...\n    private String status;        // pending, running, done, failed\n\n    @Column(columnDefinition = \"jsonb\")\n    private String payload;\n\n    private Instant runAt;\n    private Instant lockedAt;\n    private int attempts;\n}\n```\n\nThe claim query, as a native query in your Spring Data repository:\n\n```\npublic interface JobRepository extends JpaRepository<Job, UUID> {\n\n    @Query(value = \"\"\"\n        UPDATE jobs SET status = 'running', locked_at = now()\n        WHERE id IN (\n            SELECT id FROM jobs\n            WHERE status = 'pending' AND run_at <= now()\n            ORDER BY run_at\n            FOR UPDATE SKIP LOCKED\n            LIMIT :batch\n        )\n        RETURNING *\n        \"\"\", nativeQuery = true)\n    List<Job> claimJobs(@Param(\"batch\") int batch);\n}\n```\n\nOne statement. `FOR UPDATE SKIP LOCKED`\n\ninside the subquery means concurrent workers each get a disjoint batch, even if they run at the same instant. `RETURNING *`\n\nhands you the claimed rows without a second round trip.\n\nThen a poller, which is just a scheduled method:\n\n```\n@Scheduled(fixedDelay = 2000)\npublic void poll() {\n    List<Job> jobs = repository.claimJobs(5);\n    jobs.forEach(processor::handle);   // handles retry/backoff on failure\n}\n```\n\nThat is a durable job queue in about forty lines. Your jobs survive restarts because they live in a table you already back up. Compare that to bootstrapping RabbitMQ, writing consumers, configuring dead-letter exchanges, and explaining to your future self how prefetch works.\n\n**Where it breaks:** this gives you at-least-once delivery with polling latency, not push, and not Kafka-style replayable ordered logs. If you need millions of jobs per minute, streaming semantics, or long retention for event replay, use a real broker. Bauer's advice in his essay is the right heuristic: start with Postgres, and only swap in Kafka or RabbitMQ when it demonstrably stops performing.\n\nOne more production detail from experience: add a reaper. If a worker dies mid-job, rows stay `running`\n\nforever. A scheduled job that flips `running`\n\nrows older than some timeout back to `pending`\n\n(and bumps `attempts`\n\n) completes the story.\n\nEveryone reaches for Redis here. But a cache has a defining property: it can be lost. Postgres has a table type for exactly that, `UNLOGGED`\n\n. Writes skip the write-ahead log, which is most of the write cost, so it gets you dramatically closer to cache-like latency while keeping the query language and tooling you already have. The [CREATE TABLE docs](https://www.postgresql.org/docs/current/sql-createtable.html) spell out the trade: an unlogged table is not crash-safe, it gets truncated after a crash. For a cache, that is not a bug, it is a cold start.\n\n```\nCREATE UNLOGGED TABLE cache_entry (\n    key        text PRIMARY KEY,\n    value      jsonb NOT NULL,\n    expires_at timestamptz NOT NULL\n);\n```\n\nThe Spring side is trivial, one repository method:\n\n```\n@Query(value = \"\"\"\n    SELECT value FROM cache_entry\n    WHERE key = :key AND expires_at > now()\n    \"\"\", nativeQuery = true)\nString get(@Param(\"key\") String key);\n```\n\nTTL handling can be a trigger on write, as [Martin Heinz describes in his Postgres-as-cache write-up](https://martinheinz.dev/blog/105), or the simpler version I use: a `@Scheduled`\n\njob deleting expired rows every minute. Lazy and unglamorous, and in a week of running my agent lookups it never once mattered.\n\nIn my usage, agent lookups that used to cost an API call land in this table and repeated reads come back in low single-digit milliseconds, which for my workload is indistinguishable from a network hop to Redis. Your workload will differ; measure yours before believing mine.\n\n**Where it breaks:** no pub/sub invalidation across app instances, no eviction policies as sophisticated as Redis LRU, and it is one machine's cache, not a shared cluster. The moment you have many app nodes hammering one Postgres for cache reads, you are spending your database's capacity on cache traffic. That is the point to graduate to Redis, not before.\n\nThis is the one with the strongest big-company evidence. Contentful replaced their search infrastructure with Postgres full-text search and [wrote about the results](https://www.contentful.com/blog/contentful-faster-full-text-search/). Instacart did the same for [their search stack](https://tech.instacart.com/how-instacart-built-a-modern-search-infrastructure-on-postgres-c528fa601d54). The core idea is a generated column of type `tsvector`\n\nplus a GIN index, described in the [Postgres text search docs](https://www.postgresql.org/docs/current/textsearch.html).\n\nFirst the schema:\n\n```\nALTER TABLE article ADD COLUMN search_vector tsvector\n    GENERATED ALWAYS AS (\n        setweight(to_tsvector('english', coalesce(title, '')), 'A') ||\n        to_tsvector('english', coalesce(body, ''))\n    ) STORED;\n\nCREATE INDEX idx_article_search ON article USING GIN (search_vector);\n```\n\nTitle matches rank higher because of the `A`\n\nweight. Then query it from Spring:\n\n```\n@Query(value = \"\"\"\n    SELECT id, title, ts_rank(search_vector, query) AS score\n    FROM article, websearch_to_tsquery('english', :q) AS query\n    WHERE search_vector @@ query\n    ORDER BY score DESC\n    LIMIT 20\n    \"\"\", nativeQuery = true)\n    List<ArticleSearchHit> search(@Param(\"q\") String q);\n```\n\n`websearch_to_tsquery`\n\naccepts search terms the way users type them, quoted phrases included, instead of the arcane `to_tsquery`\n\nsyntax. No data sync problem, because the index lives next to the data. When I searched my own article corpus, a few thousand rows, results came back faster than anything I could perceive, and relevance was entirely acceptable for a site search box.\n\n**Where it breaks:** no typo tolerance or semantic matching out of the box, and relevance tuning is nothing like Elasticsearch's. At tens of millions of documents or heavy faceting needs, a dedicated engine earns its keep. A few thousand to a few million rows with plain keyword search: Postgres wins on operations alone, one less system to keep in sync with your primary data.\n\nThe Guardian's migration story is instructive because they were running MongoDB at real scale and [moved to Postgres](https://www.theguardian.com/info/2018/nov/30/bye-bye-mongo-hello-postgres) partly to reduce operational surface. The [jsonb type](https://www.postgresql.org/docs/current/datatype-json.html) stores documents, indexes inside them with GIN, and queries them with path operators. In Spring Boot, annotate a field with `@Column(columnDefinition = \"jsonb\")`\n\n, map it with a Hibernate `AttributeConverter`\n\n, and you have schema-flexible documents inside your relational database.\n\nI use this for agent payloads and tool outputs, shapes that drift as I experiment. But I keep the columns I actually query on as real typed columns. JSONB is a sharp knife: wonderful for \"store this blob of shape X\", risky as a foundation for a heavily queried data model where you lose constraints, foreign keys, and clear types. The pragmatic pattern is typed columns for what you filter and join on, JSONB for the parts that legitimately vary.\n\nThe pattern most relevant to what I actually do in 2026. The [pgvector extension](https://github.com/pgvector/pgvector) turns Postgres into a vector database with HNSW indexes, and Spring AI ships a PgVector vector store implementation, so your RAG embeddings can live in the same database as everything else. For a personal agent setup like mine, that means similarity search without running a separate Pinecone or Qdrant instance. Same rule applies: at serious scale or with heavy multi-tenant filtering, dedicated vector databases justify themselves. For thousands to low millions of embeddings, one Postgres is fewer systems to babysit.\n\nHere is the decision checklist I now run through before adopting any new data infrastructure. Copy it into your next design doc.\n\n`SKIP LOCKED`\n\nis enough when you have up to a few thousand jobs per minute and can tolerate poll latency. Move to a broker when you need streaming, fan-out to external systems, or replayable event logs.The common thread: each swap should be triggered by a measured limit, not by an architecture diagram that looks more impressive with more boxes.\n\nIf I were starting my agent infrastructure today, I would begin with Postgres doing every job from day one, and I would treat each specialized system as an extraction to be earned by evidence. That is roughly the opposite of how I actually did it, which was to sketch a Kafka-shaped diagram first because it looked \"production-grade\". What makes a system production-grade is that you can operate it at 3 AM, and one database with good backups beats four systems you half-understand.\n\nThe honest caveat: \"Postgres for everything\" is a starting position, not a religion. The companies I cited above did not stop at Postgres because it was trendy; they measured, and Postgres met the bar for their workload at the time. Do the same measurement for yours.\n\nHave you shipped a Postgres-only queue or replaced a search cluster with `tsvector`\n\n? What scale did it hold up to before you had to move on? I would genuinely like to know where the walls are, so drop your experience in the comments.\n\nI write about Java, Spring Boot, and AI every week, mostly things I broke myself so you do not have to. Subscribe, it is free.", "url": "https://wpnews.pro/news/before-you-add-kafka-redis-and-elasticsearch-try-one-postgres-first", "canonical_source": "https://dev.to/jamilxt/before-you-add-kafka-redis-and-elasticsearch-try-one-postgres-first-20g1", "published_at": "2026-08-20 12:03:00+00:00", "updated_at": "2026-08-20 12:15:35.089995+00:00", "lang": "en", "topics": ["developer-tools", "artificial-intelligence", "ai-infrastructure"], "entities": ["Postgres", "Spring Boot", "Redis", "RabbitMQ", "Elasticsearch", "Contentful", "Instacart", "The Guardian"], "alternates": {"html": "https://wpnews.pro/news/before-you-add-kafka-redis-and-elasticsearch-try-one-postgres-first", "markdown": "https://wpnews.pro/news/before-you-add-kafka-redis-and-elasticsearch-try-one-postgres-first.md", "text": "https://wpnews.pro/news/before-you-add-kafka-redis-and-elasticsearch-try-one-postgres-first.txt", "jsonld": "https://wpnews.pro/news/before-you-add-kafka-redis-and-elasticsearch-try-one-postgres-first.jsonld"}}