{"slug": "show-hn-notifyd-notification-queue-on-postgres-alone-rust-mcp-no-ui", "title": "Show HN: Notifyd – notification queue on Postgres alone (Rust, MCP, no UI)", "summary": "Ramzi Laieb released notifyd 0.2.1, a Rust-based notification queue that uses PostgreSQL alone, achieving 3,537 jobs/s on an 8-vCPU host with 23 MB of resident memory, up from 516 jobs/s in an unoptimized implementation. The service handles email, SMS, WhatsApp, push, and in-app inbox, with a built-in MCP server, and addresses six concerns left open by the SKIP LOCKED primitive, including provider rate limits and retry handling.", "body_md": "Agent-first notification service. One Rust binary, Postgres only. Email, SMS, WhatsApp, push, in-app inbox, MCP server built in.\n\n`SKIP LOCKED` gives you, and what it does not\n*Ramzi Laieb, 2026-09-06. Applies to notifyd 0.2.1. Parts of the code and of\nthis text were drafted with an AI coding assistant and reviewed by the author.\nEvery citation was fetched from its primary source on 2026-09-06; two that\ncould not be fetched are marked as such.*\n\n`SELECT … FOR UPDATE SKIP LOCKED` has made PostgreSQL a credible job queue\nsince 9.5 (2016) and a default one in several ecosystems (Rails 8’s Solid\nQueue, Oban, River, Graphile Worker, pg-boss). Notification delivery, however,\nis not a generic job: the bottleneck is not the database but a third-party\nprovider with an account-wide quota, whose `429` answers must be interpreted,\nwhose failures must be classified, and whose limits must be shared between a\npassword reset and a 5 000-recipient campaign. This article surveys what the\nPostgres-queue literature and the existing queues provide, identifies six\nconcerns that the claim primitive leaves open, describes how notifyd (a\nsingle Rust binary, Postgres as its only dependency) addresses them, and\nreports measurements on commodity hardware: an unoptimised implementation\ndrained 516 jobs/s; batching the claim, the context lookups and the\nfinalisation, and removing a per-job task, brought it to 3 537 jobs/s on the\nsame 8-vCPU host with 23 MB of resident memory. We then list what these\nnumbers do not show, and what remains open.\n\nA product sends two kinds of notifications through the same providers:\ntransactional messages (order shipped, password reset) whose value decays in\nseconds, and campaigns whose value is indifferent to a delay of an hour. Both\ngo out through providers that meter by account: Resend’s default limit is\n10 requests per second per team with a 100-message batch endpoint [R1, R2];\nAmazon SES exposes a per-second *sending rate* that “you can exceed for short\nbursts, but not for sustained periods” [R3]; Postmark caps a batch at 500\nmessages [R4]. A provider answers excess with `429 Too Many Requests`, which\nRFC 6585 allows to carry a `Retry-After` header [L3]. Since February 2024,\nbulk senders to Gmail and Yahoo must also keep the reported spam rate under\n0.3 % and honour one-click unsubscribe within two days [L5, L6, L7].\n\nThe engineering question is therefore not “can Postgres hold a queue” (it\ncan) but “what must sit between the queue and the provider so that a\ncampaign never starves a password reset, a `429` never burns a retry, and an\noperator, human or agent, can see why something did not leave”.\n\n`SKIP LOCKED` landed in PostgreSQL 9.5.0 on 2016-01-07, in the same release as\n`INSERT … ON CONFLICT` [P1]. The manual is explicit about its intended use and\nits cost: “Skipping locked rows provides an inconsistent view of the data, so\nthis is not suitable for general purpose work, but can be used to avoid lock\ncontention with multiple consumers accessing a queue-like table” [P2]. Two\ncaveats in the same page matter for a queue: the table-level `ROW SHARE`\nlock is still taken, and with `LIMIT`, “locking stops once enough rows have\nbeen returned to satisfy the limit” [P2]. The alternative, advisory locks, is\n“faster, avoid[s] table bloat” but carries its own `LIMIT` hazard, which the\nmanual illustrates with a query it labels `-- danger!` [P3].\n\nThe operational caveat is dead tuples. A queue row is inserted, updated to\n`processing`, then to `sent` or `retry`: three versions per job. Brandur\nLeach documented in 2015 how a long-running transaction elsewhere left\n“247311 dead row versions [that] cannot be removed yet” and pushed the lock\nquery “from < 0.01 seconds to 0.1 s and above” [P5]. Crunchy Data’s 2021\nwrite-up of the `SKIP LOCKED` + `DELETE … RETURNING` pattern ends on the same\nadvice: monitor bloat, tune autovacuum, possibly rotate the table [P6].\nHatchet’s 2026 survival guide is blunter: “If autovacuum can’t keep up …\nyou’ll get into a very unhealthy state, very quickly” [P7]. The manual itself\nnotes that “some installations with extremely high update rates vacuum their\nbusiest tables as often as once every few minutes” [P4]. PostgreSQL 17 made\nvacuum’s dead-tuple storage up to 20× smaller [P8]; PostgreSQL 18 added\n`autovacuum_vacuum_max_threshold`, a fixed dead-tuple trigger that no longer\nscales with table size, plus asynchronous I/O covering vacuum and B-tree skip\nscans [P9]. River’s 2026 “concurrent repack” describes the remaining gap:\nvacuum “marks space as reusable … but never fully reclaims it” [P10].\n\nTable 1 summarises the queues whose source or documentation we read. All but\ntwo use `SKIP LOCKED`; most add `LISTEN/NOTIFY` to wake workers.\n\n| Queue | Language | Claim | Published throughput | Priority | Outbound rate limiting | \n|---|---|---|---|---|---|\n| pg-boss 12 [Q1] | Node | `SKIP LOCKED` + NOTIFY | none | yes | queue storage policies | \n| Graphile Worker 0.17 [Q2] | Node | `SKIP LOCKED` + NOTIFY | ~183 000 jobs/s batched, ~15 600 unbatched; 200 000 trivial jobs, 4 processes × 24, i9-14900K, DB on the same machine | yes | no | \n| Oban 2.24 [Q3] | Elixir | `SKIP LOCKED` ,`ORDER BY priority, scheduled_at, id` | formula only: `(1000 / cooldown) × limit` per queue | 10 levels | Pro only (“Smart Engine”) | \n| River 0.47 [Q4] | Go | `SKIP LOCKED` ,`ORDER BY priority, scheduled_at, id` | ~46 000 jobs/s, 1 M no-op jobs, 2 000 goroutines, M2 MacBook Air | 1–4 | Pro concurrency limits; global rate limiting “groundwork” (2025) | \n| Solid Queue 1.7 [Q5, Q6] | Ruby | `SKIP LOCKED` “if available” | HEY: ~20 M jobs/day; ~1 300 polling queries/s at 110 µs | integer | concurrency controls, no time-based limit | \n| Que [Q7] | Ruby | advisory locks, NOTIFY | none | integer | no | \n| PGMQ [Q8] | SQL extension | `SKIP LOCKED` + visibility timeout | none | no (FIFO) | no | \n| Procrastinate [Q9] | Python | `SKIP LOCKED` ,`LIMIT 1` | none published | yes | no | \n| apalis-postgres 1.0-rc [Q10] | Rust | `SKIP LOCKED` ,`ORDER BY priority DESC, run_at` | none | yes | tower layers, not queue-native | \n| sqlxmq [Q11] | Rust | `UPDATE … FROM (SELECT … LIMIT)` re-checked, NOTIFY | none | no | concurrency only | \n| pgqueuer [Q12] | Python | `SKIP LOCKED` + NOTIFY | none | yes | per-entrypoint limits | \n\n*Table 1. Only Graphile Worker and River publish a jobs/s figure with the\nhardware. None of the surveyed queues ships a per-channel token bucket or a\nnotion of “the provider said 429” inside the queue: rate limiting, where it\nexists, bounds the consumer’s own concurrency, not a third party’s quota.*\n\nTwo figures from Table 1 frame our results. Graphile Worker’s ~183 000 jobs/s\nand River’s ~46 000 jobs/s are for *no-op* jobs; the authors say so, and River\nadds that “benchmarking is a highly imperfect science” [Q4]. A notification\njob is not a no-op: it resolves a sender identity, checks suppressions and\npreferences, renders a template, builds an RFC 5322 message with\nunsubscribe headers, and calls a provider. Section 5 measures that path with\nthe provider call stubbed, which is the right comparison point for the\nengine, and states what it leaves out.\n\nThe notification platforms we could read handle the provider limit\ndifferently from what a queue reader might expect. Novu’s documentation\nstates that a channel step “makes a single call to the provider. There is no\nautomatic retry, no backoff, and no ceiling, because there is no second\nattempt”, and that “a `429 Too Many Requests` … from a provider is handled the\nsame way as a `400 Bad Request`” [N1]; the worker source confirms that the\nsend is wrapped in a `try/catch` that records `PROVIDER_ERROR` without\ninspecting the status code [N2]. Knock, Courier and SuprSend each document a\n“throttle” step, but all three define it as a per-recipient anti-flood\ncontrol (“limit the number of times a workflow is executed for a recipient\nwithin a given window” [N3]; “how many … messages a user or group receives\nwithin a set timeframe” [N4]; “rate limit workflow executions per user” [N5]),\nnot as pacing against the provider. Their API rate limits are documented on\nthe inbound side [N3]. We found no public documentation of provider-side\n`429` pacing at any of the four.\n\nThe mechanisms we use are old and documented. Token buckets bound rate and\nburst [L1]; RFC 2697 formalises a two-rate variant [L2]. Exponential backoff\nwith jitter is analysed by Brooker (2015), whose “full jitter” and\n“decorrelated jitter” formulas are the usual references [L4]. `Retry-After`\nis specified in RFC 9110 §10.2.3 and attached to `429` by RFC 6585 §4 [L3].\nOne-click unsubscribe is RFC 8058 [L5]; Gmail’s and Yahoo’s 2024 requirements\nmake it mandatory above 5 000 messages a day and set the 0.3 % spam-rate\nceiling [L6, L7]. Staging jobs in the same transaction as the business write,\nso that a crash between commit and enqueue cannot lose them, is Leach’s\n“transactionally staged job drain” [D2]; the broader “use Postgres, spend\nyour innovation tokens elsewhere” argument is McKinley’s [D1] and Hunt’s [D3].\n\nReading Sections 2.1 to 2.3 together, six concerns are not addressed by the claim statement, and are only partly addressed by the queues built on it.\n\n`429`.` 422 unverified sender`, a\n`503`, a network timeout and a `Err` forces the caller to encode this, and most callers do not.`SELECT count(*) … GROUP BY status` at 3 a.m. The queues in\nTable 1 expose metrics; the platforms in 2.3 expose dashboards. Neither\nsays what to do.\nnotifyd is one Rust binary (axum, sqlx, tokio) with PostgreSQL as its only\ndependency. This section describes the parts of it that answer Section 3.\nAll SQL below is quoted from `src/worker.rs` and `src/api/send.rs` at 0.2.1.\n\nJobs are rows. A worker claims a batch in a transaction, orders by priority then by schedule, skips channels that a provider has asked to pause, and marks the batch in a second statement:\n\n```\nSELECT … FROM jobs\nWHERE status IN ('pending', 'retry')\n  AND scheduled_at <= $1\n  AND (next_retry_at IS NULL OR next_retry_at <= $1)\n  AND NOT (channel = ANY($3))          -- channels paused after a 429\nORDER BY priority ASC, scheduled_at ASC\nLIMIT $2\nFOR UPDATE SKIP LOCKED;\n\nUPDATE jobs SET status = 'processing', attempts = attempts + 1, claimed_at = now()\nWHERE id = ANY($1);\n```\n\nA partial index ```\nON jobs (priority, scheduled_at) WHERE status IN ('pending',\n'retry')\n```\n keeps the claim cheap as the table fills with `sent` rows. Priority\nis a 0–100 integer; the API accepts `critical` (10), `high` (30), `normal`\n(50), `low` (70), `bulk` (80) or a number, and `POST /v1/batch` defaults to\n`bulk`, as does any send tagged `campaign`, `marketing` or `newsletter`.\n\nA token bucket per channel (`EMAIL_RATE_PER_SEC`, `SMS_RATE_PER_SEC`) bounds\noutbound calls per replica [L1]. When a provider answers `429`, the worker\n(a) tries the fallback provider if one is configured, then (b) pauses the\n*channel* for `Retry-After` when the provider sent one, or a configured\ndefault otherwise, and (c) re-queues the job **without consuming an\nattempt**:\n\n```\nUPDATE jobs SET status = 'retry', attempts = GREATEST(attempts - 1, 0),\n                error = $2, next_retry_at = $3 WHERE id = $1;\n```\n\nPriorities do not bypass the pause. The provider’s limit is per account, so a\n`critical` message sent during the pause would be refused as well. What the\ndesign guarantees is order on resume: the claim statement puts `critical`\nahead of `bulk`, so the password reset is in the first batch after\n`Retry-After` elapses, whatever the campaign backlog. Other channels are not\naffected by an email pause.\n\nConnectors return a typed error: `RateLimited { retry_after }`, `Transient`,\n`Permanent`, `Suppressed`. Only `Transient` and `RateLimited` trigger the\nfailover breaker; `Permanent` (4xx other than 429, SQLSTATE 23xxx on\nin-app inserts) fails immediately; `Suppressed` records the outcome without\ncalling the provider. Transient failures follow a fixed schedule of 30 s,\n2 min, 10 min, 30 min, 2 h with ±20 % jitter [L4], five attempts by default.\nA reaper re-queues jobs left in `processing` for more than ten minutes, so a\nworker that dies mid-batch loses at most that.\n\nEach connector declares `batch_max()`: 100 for Resend (its API’s ceiling\n[R2]), 1 for SMTP, Twilio and Telnyx. The worker chunks the claimed batch\naccordingly. A batch refused with a 4xx is retried item by item, so one bad\naddress does not fail 99 good ones. Successful items are finalised in one\nstatement:\n\n```\nUPDATE jobs SET status = 'sent', sent_at = now(), error = NULL,\n                provider = r.provider, provider_message_id = r.mid\nFROM unnest($1::uuid[], $2::text[], $3::text[]) AS r(id, provider, mid)\nWHERE jobs.id = r.id;\n```\n\nBefore the change measured in Section 5, each job performed its own sender\nlookup, suppression check, preference check and spawned a webhook task that\ncreated an HTTP client and queried the project’s webhooks. After it, the\nworker loads senders, suppressions, preferences and the set of projects that\nhave webhooks once per claimed batch; if that prefetch fails, it falls back\nto the per-job query rather than skipping the check (a suppression must\nnever be bypassed because a cache failed). `POST /v1/batch` inserts its N\njobs with one `INSERT … SELECT FROM unnest(…)`, with the idempotency\nconflict handled by a partial unique index on ```\n(project_id, idempotency_key)\nWHERE status NOT IN ('failed', 'cancelled')\n```\n.\n\nMarketing email carries RFC 8058 headers pointing at an HMAC-signed\nunsubscribe URL [L5]; suppressions have a scope (`all` or `marketing`) so a\ncustomer who leaves the newsletter still receives their invoice. Send windows\nare evaluated in the recipient’s timezone. `GET /v1/admin/digest` ranks\nfindings (paused channel, bounce rate above 2 % or 5 %, oldest waiting job,\nfailed jobs with their top cause, missing fallback provider) and attaches to\neach one the action an operator would take. The same operations are exposed\nas MCP tools with `readOnlyHint` / `destructiveHint` annotations, and a\nread-only key restricts an agent to the reporting subset. This is the answer\nto concern 6: not a dashboard, but a ranked list with actions that a human\nor an agent can execute.\n\n| Host | 8 vCPU Arm Neoverse-V2, 30 GB RAM, Ubuntu, Docker | \n| PostgreSQL | 16, default `postgresql.conf` , one container on the same host | \n| notifyd | 0.2.x release build; `WORKER_BATCH_SIZE=500` ,`WORKER_POLL_INTERVAL_MS=100` ,`DATABASE_MAX_CONNECTIONS=20` ,`EMAIL_RATE_PER_SEC=0` (pacer disabled) | \n| Provider | `EMAIL_PROVIDER=log` , a no-op connector with`batch_max = 100` , i.e. the Resend code path minus the network | \n| Load | one project, unlimited inbound rate; 100 000 email jobs enqueued through `POST /v1/batch` in calls of 5 000, all`scheduled_at` in the future, released by one`UPDATE` , drained by one worker | \n| Measurement | wall-clock from release to `count(*) WHERE status IN ('pending','processing','retry') = 0` ; RSS sampled every 250 ms with`ps` ;`sent_at - scheduled_at` percentiles from the table | \n\nThe commands are in `docs/BENCHMARKS.md`. Everything runs on one machine,\nwhich flatters latency and penalises nothing else; it is the setup a small\ncompany actually deploys.\n\n| Path | Before (0.2.0-pre) | After (3423bc7) | \n|---|---|---|\n| `POST /v1/batch` , 5 000 recipients per call | 719 jobs/s | 44 546 jobs/s | \n| Drain, provider batching 100 | 516 jobs/s | 3 537 jobs/s | \n| Drain, provider batching 1 (SMTP, SMS) | — | 640 jobs/s | \n| RSS at idle / peak while draining 100 000 jobs | 13 MB / 22 MB | 13 MB / 23 MB | \n| `jobs` table + indexes after 100 000 sent jobs |  | 84 MB (0.85 kB/job, body included) | \n| `GET /v1/admin/digest` over 100 000 jobs |  | 70 ms | \n| `POST /v1/send` , 8 concurrent clients |  | 1 867 req/s, p50 3.9 ms | \n\n*Table 2. Same host, same Postgres, same load. “Before” is the per-job\nimplementation; “after” is Section 4.4–4.5.*\n\nThe 6.9× drain improvement did not come from `SKIP LOCKED`, which was already\nthere; it came from removing per-job round trips (concern 3) and batching\nfinalisation (concern 4). With a non-batching provider the same engine caps\nnear 640 jobs/s: each message becomes its own provider call and its own\nfinalisation `UPDATE`, which is the shape SMTP and SMS impose in production\nanyway, where the provider’s quota, not the engine, is the ceiling.\n\nAgainst Table 1, 3 537 jobs/s is an order of magnitude below Graphile Worker\nand River, and it should be: a no-op job has none of the fixed costs listed\nin Section 3, and our poll interval (100 ms, no `LISTEN/NOTIFY`) bounds\nminimum latency. For the target workload the relevant comparison is the\nprovider: at Resend’s 10 requests/s × 100 recipients, 100 000 emails leave in\nroughly 100 s if the provider allows the burst and around 8–9 minutes at a\npaced 2 requests/s; the engine spends most of that time waiting on the token\nbucket, at 23 MB of memory.\n\n`LISTEN/NOTIFY`.` NOTIFY`, the worker does not.` finalize_sent_batch` commits results in a resend after the reaper\nfires; the provider’s idempotency, where it exists, is not used.`autovacuum_vacuum_max_threshold` [P9] is the\nright default for a queue table that sees 3 × N tuple versions per N\njobs is an empirical question we have not answered.`UPDATE … RETURNING`) costs a round trip per batch and would be\nexact; River’s Pro roadmap names the same problem [Q4].`(status, priority, scheduled_at)` index serve both the claim and the\noperator queries; we have not measured it.\nCode: `github.com/rmzlb/notifyd`, tag `v0.2.1`, MIT. Benchmark protocol and\ncommands: `docs/BENCHMARKS.md`. Unit tests: `cargo test` (65 tests, no\ndatabase required). The numbers in Table 2 were produced on 2026-09-05 and\n2026-09-06 on the host described in 5.1; we will link any independent\nmeasurement, on any hardware, that follows the protocol.\n\n**PostgreSQL**\n\n`SELECT`, “The Locking Clause”. https://www.postgresql.org/docs/current/sql-select.html\n**Queues**\n\n`lib/oban/engines/basic.ex`.\n**Notification platforms and providers**\n\n`apps/worker/src/app/workflow/usecases/send-message/send-message-email.usecase.ts`, branch `next`, read 2026-09-06.\n**Mechanisms and requirements**\n\n**Discourse**", "url": "https://wpnews.pro/news/show-hn-notifyd-notification-queue-on-postgres-alone-rust-mcp-no-ui", "canonical_source": "https://rmzlb.github.io/notifyd/articles/postgres-queue-what-skip-locked-does-not-give-you.html", "published_at": "2026-09-08 12:45:05+00:00", "updated_at": "2026-09-08 12:58:56.608645+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-tools", "ai-infrastructure", "developer-tools"], "entities": ["Ramzi Laieb", "notifyd", "PostgreSQL", "Rust", "Resend", "Amazon SES", "Postmark", "Gmail"], "alternates": {"html": "https://wpnews.pro/news/show-hn-notifyd-notification-queue-on-postgres-alone-rust-mcp-no-ui", "markdown": "https://wpnews.pro/news/show-hn-notifyd-notification-queue-on-postgres-alone-rust-mcp-no-ui.md", "text": "https://wpnews.pro/news/show-hn-notifyd-notification-queue-on-postgres-alone-rust-mcp-no-ui.txt", "jsonld": "https://wpnews.pro/news/show-hn-notifyd-notification-queue-on-postgres-alone-rust-mcp-no-ui.jsonld"}}