Last month I was building out my own AI agent infrastructure, the side project where I run a few agents that research, draft, and publish content. The agents needed three things beyond the main database: a job queue, a cache for repeated lookups, and search over article text. My instinct, six years of Spring Boot muscle memory, was to reach for the usual stack. RabbitMQ or Redis Streams for the queue. Redis for the cache. Elasticsearch or Meilisearch for search.
Then I counted the services. One Spring Boot app plus three more moving parts, each with its own Docker image, its own failure modes, its own 3 AM pager behavior. For a system serving a handful of agents, that felt like hiring an orchestra to play a ringtone.
So I tried something that old-school database people keep telling us and we keep ignoring: I did all three jobs in Postgres. A table with SKIP LOCKED
for the queue. An unlogged table for the cache. A tsvector
column with a GIN index for search. The whole thing is one schema, one backup strategy, one connection pool.
This idea is having a moment again. An essay titled "PostgreSQL for Everything" by Raphael Bauer has been on the Hacker News front page this week, and it makes the case I stumbled into: before you adopt a new specialized system, ask whether Postgres already does that job well enough. Contentful rebuilt their full-text search on Postgres instead of a separate search cluster. Instacart built a modern search infrastructure on Postgres. The Guardian famously migrated off MongoDB onto Postgres for parts of their platform.
Those are big companies. I am one developer in Dhaka with a VPS. But that is exactly the point. The smaller your team, the more a single database that does five jobs is worth. Here is the Spring Boot version of "Postgres for everything", the patterns I actually used, with code.
Full disclosure up front: I have run these patterns in my own projects, not at giant scale. I will point out where each one breaks down as you grow, because some of them do.
The workhorse. You have a jobs
table, workers poll it, and each job must be claimed by exactly one worker. The naive approach, SELECT
then UPDATE WHERE status = 'pending'
, races between workers. Postgres has had the fix since 9.5: SELECT ... FOR UPDATE SKIP LOCKED
. Each worker locks the rows it grabs, and rows already locked by another worker are simply skipped, so two workers never claim the same job. No advisory locks, no leader election, no broker. The Postgres docs cover this under row-level locking, and Crunchy Data has a good deep dive on queuing with native Postgres.
The entity:
@Entity
@Table(name = "jobs", indexes =
@Index(name = "idx_jobs_pending", columnList = "status, runAt"))
public class Job {
@Id @GeneratedValue(strategy = GenerationType.UUID)
private UUID id;
private String type; // "draft-article", "fetch-stats", ...
private String status; // pending, running, done, failed
@Column(columnDefinition = "jsonb")
private String payload;
private Instant runAt;
private Instant lockedAt;
private int attempts;
}
The claim query, as a native query in your Spring Data repository:
public interface JobRepository extends JpaRepository<Job, UUID> {
@Query(value = """
UPDATE jobs SET status = 'running', locked_at = now()
WHERE id IN (
SELECT id FROM jobs
WHERE status = 'pending' AND run_at <= now()
ORDER BY run_at
FOR UPDATE SKIP LOCKED
LIMIT :batch
)
RETURNING *
""", nativeQuery = true)
List<Job> claimJobs(@Param("batch") int batch);
}
One statement. FOR UPDATE SKIP LOCKED
inside the subquery means concurrent workers each get a disjoint batch, even if they run at the same instant. RETURNING *
hands you the claimed rows without a second round trip.
Then a poller, which is just a scheduled method:
@Scheduled(fixedDelay = 2000)
public void poll() {
List<Job> jobs = repository.claimJobs(5);
jobs.forEach(processor::handle); // handles retry/backoff on failure
}
That is a durable job queue in about forty lines. Your jobs survive restarts because they live in a table you already back up. Compare that to bootstrapping RabbitMQ, writing consumers, configuring dead-letter exchanges, and explaining to your future self how prefetch works.
Where it breaks: this gives you at-least-once delivery with polling latency, not push, and not Kafka-style replayable ordered logs. If you need millions of jobs per minute, streaming semantics, or long retention for event replay, use a real broker. Bauer's advice in his essay is the right heuristic: start with Postgres, and only swap in Kafka or RabbitMQ when it demonstrably stops performing.
One more production detail from experience: add a reaper. If a worker dies mid-job, rows stay running
forever. A scheduled job that flips running
rows older than some timeout back to pending
(and bumps attempts
) completes the story.
Everyone reaches for Redis here. But a cache has a defining property: it can be lost. Postgres has a table type for exactly that, UNLOGGED
. Writes skip the write-ahead log, which is most of the write cost, so it gets you dramatically closer to cache-like latency while keeping the query language and tooling you already have. The CREATE TABLE docs spell out the trade: an unlogged table is not crash-safe, it gets truncated after a crash. For a cache, that is not a bug, it is a cold start.
CREATE UNLOGGED TABLE cache_entry (
key text PRIMARY KEY,
value jsonb NOT NULL,
expires_at timestamptz NOT NULL
);
The Spring side is trivial, one repository method:
@Query(value = """
SELECT value FROM cache_entry
WHERE key = :key AND expires_at > now()
""", nativeQuery = true)
String get(@Param("key") String key);
TTL handling can be a trigger on write, as Martin Heinz describes in his Postgres-as-cache write-up, or the simpler version I use: a @Scheduled
job deleting expired rows every minute. Lazy and unglamorous, and in a week of running my agent lookups it never once mattered.
In my usage, agent lookups that used to cost an API call land in this table and repeated reads come back in low single-digit milliseconds, which for my workload is indistinguishable from a network hop to Redis. Your workload will differ; measure yours before believing mine.
Where it breaks: no pub/sub invalidation across app instances, no eviction policies as sophisticated as Redis LRU, and it is one machine's cache, not a shared cluster. The moment you have many app nodes hammering one Postgres for cache reads, you are spending your database's capacity on cache traffic. That is the point to graduate to Redis, not before.
This is the one with the strongest big-company evidence. Contentful replaced their search infrastructure with Postgres full-text search and wrote about the results. Instacart did the same for their search stack. The core idea is a generated column of type tsvector
plus a GIN index, described in the Postgres text search docs.
First the schema:
ALTER TABLE article ADD COLUMN search_vector tsvector
GENERATED ALWAYS AS (
setweight(to_tsvector('english', coalesce(title, '')), 'A') ||
to_tsvector('english', coalesce(body, ''))
) STORED;
CREATE INDEX idx_article_search ON article USING GIN (search_vector);
Title matches rank higher because of the A
weight. Then query it from Spring:
@Query(value = """
SELECT id, title, ts_rank(search_vector, query) AS score
FROM article, websearch_to_tsquery('english', :q) AS query
WHERE search_vector @@ query
ORDER BY score DESC
LIMIT 20
""", nativeQuery = true)
List<ArticleSearchHit> search(@Param("q") String q);
websearch_to_tsquery
accepts search terms the way users type them, quoted phrases included, instead of the arcane to_tsquery
syntax. No data sync problem, because the index lives next to the data. When I searched my own article corpus, a few thousand rows, results came back faster than anything I could perceive, and relevance was entirely acceptable for a site search box.
Where it breaks: no typo tolerance or semantic matching out of the box, and relevance tuning is nothing like Elasticsearch's. At tens of millions of documents or heavy faceting needs, a dedicated engine earns its keep. A few thousand to a few million rows with plain keyword search: Postgres wins on operations alone, one less system to keep in sync with your primary data.
The Guardian's migration story is instructive because they were running MongoDB at real scale and moved to Postgres partly to reduce operational surface. The jsonb type stores documents, indexes inside them with GIN, and queries them with path operators. In Spring Boot, annotate a field with @Column(columnDefinition = "jsonb")
, map it with a Hibernate AttributeConverter
, and you have schema-flexible documents inside your relational database.
I use this for agent payloads and tool outputs, shapes that drift as I experiment. But I keep the columns I actually query on as real typed columns. JSONB is a sharp knife: wonderful for "store this blob of shape X", risky as a foundation for a heavily queried data model where you lose constraints, foreign keys, and clear types. The pragmatic pattern is typed columns for what you filter and join on, JSONB for the parts that legitimately vary.
The pattern most relevant to what I actually do in 2026. The pgvector extension turns Postgres into a vector database with HNSW indexes, and Spring AI ships a PgVector vector store implementation, so your RAG embeddings can live in the same database as everything else. For a personal agent setup like mine, that means similarity search without running a separate Pinecone or Qdrant instance. Same rule applies: at serious scale or with heavy multi-tenant filtering, dedicated vector databases justify themselves. For thousands to low millions of embeddings, one Postgres is fewer systems to babysit.
Here is the decision checklist I now run through before adopting any new data infrastructure. Copy it into your next design doc.
SKIP LOCKED
is enough when you have up to a few thousand jobs per minute and can tolerate poll latency. Move to a broker when you need streaming, fan-out to external systems, or replayable event logs.The common thread: each swap should be triggered by a measured limit, not by an architecture diagram that looks more impressive with more boxes.
If I were starting my agent infrastructure today, I would begin with Postgres doing every job from day one, and I would treat each specialized system as an extraction to be earned by evidence. That is roughly the opposite of how I actually did it, which was to sketch a Kafka-shaped diagram first because it looked "production-grade". What makes a system production-grade is that you can operate it at 3 AM, and one database with good backups beats four systems you half-understand.
The honest caveat: "Postgres for everything" is a starting position, not a religion. The companies I cited above did not stop at Postgres because it was trendy; they measured, and Postgres met the bar for their workload at the time. Do the same measurement for yours.
Have you shipped a Postgres-only queue or replaced a search cluster with tsvector
? What scale did it hold up to before you had to move on? I would genuinely like to know where the walls are, so drop your experience in the comments.
I write about Java, Spring Boot, and AI every week, mostly things I broke myself so you do not have to. Subscribe, it is free.