One writer, twelve workers, zero rows

Anthony “chovy” Ettinger, a developer, fixed his SQLite-based feed crawler that stopped writing rows despite reads returning in 100ms, after discovering that 12 concurrent transactions were contending for SQLite's single-writer lock, causing writes to time out at 300 seconds. The fix was a BullMQ queue running on Bun with a worker concurrency of 1, which serializes writes and adds backpressure, retries, and failed-job inspection. Ettinger documented the diagnosis at bufferoverride.com/q/3, noting that five initial theories—database size, FTS triggers, batch size, hosting quota, and shared database group—were all ruled out.

One writer, twelve workers, zero rows 2026-08-19, by Anthony “chovy” Ettinger. How this was written: drafted with an AI assistant from my own notes, then edited by me. My feed crawler stopped. Not slowed down, stopped. It reads something like 55,000 feeds a day into SQLite, and one afternoon the writes just queued up behind each other and nothing ever landed. Reads were fine. A select 1 came back in 100ms all day while the crawler sat there emitting one feed every few minutes. SQLite allows one writer at a time. Everybody knows this. That is the storage engine, not a setting you forgot to flip. So I assumed I had simply outgrown it and started looking for the thing that was making my one writer slow. I posted the question at bufferoverride.com/q/3 https://bufferoverride.com/q/3/sqlite3-one-write-limitation , mostly so the answer would exist somewhere the next person can find it. Here is what a day of measuring actually turned up. Everything I blamed first, and why none of it was the cause The database was 14GB, about 10GB of it stored article HTML. Obvious culprit. I tested it: one insert took 168 seconds, one hundred inserts took 23.7 seconds. Bytes were not the currency. I kept the column. Then the full text search triggers, because writing to an FTS index on every insert sounds expensive. Insert with do nothing , with a guarded do update , and against a table with no FTS at all: same time. Then batch size, 1000 rows down to 25, no change. Then the hosting quota, which was genuinely blown at 924% of rows read. I paid to fix it and write latency moved not at all. Then the shared database group, which I tested by timing an idle sibling database at the same instants: 370ms while mine took 181 seconds. The group was healthy. Five theories, five days worth of plausible, all wrong. The measurement that took one minute Three lines, run against the live database: php db.execute 'update feeds set x = x where id = ?' - 389ms db.batch that same statement , 'write' - 302s, FAILED db.batch 20 of them , 'deferred' - 123s, SQLITE BUSY The same single statement is fast on its own and times out inside a transaction. That is not a slow database. That is contention, and once you see it in that shape the rest is obvious. batch stmts, 'write' opens an explicit transaction. My poller ran four crawl workers plus a card pass, a cluster pass, an author pass and an alert pass, and every one of them opened its own. A dozen transactions were competing for a lock exactly one of them could hold. They did not politely queue. Each waited out the client's 300 second header timeout, gave up, retried, and joined the back of the pile again. Throughput was not degraded, it was zero, while a plain single statement write kept answering in 389ms. The fix is a queue with one worker The rule I needed was simple: one write at a time for the whole system, no matter how many things want to write. So I stopped letting the crawl workers touch the database at all. They fetch, they parse, and they push a job. It is a BullMQ queue running on Bun, and the entire fix is the worker concurrency: js new Worker 'feed-writes', async job = { await storeCrawl job.data }, { connection, concurrency: 1 } That one line is the lock. SQLite gets exactly one writer because there is exactly one consumer, and it is Redis holding the line instead of a database timing out at 300 seconds. Producers are as parallel as I want them to be. What I did not expect was how much else came along with it. Backpressure became a number I can look at, so queue depth tells me I am behind instead of a wall of client timeouts telling me nothing. Retries with exponential backoff are already there, and a write that fails lands in a failed set I can inspect and replay rather than disappearing into a catch block. Fetching decoupled from writing, so I could raise fetch concurrency again without adding a single writer. Bun is doing nothing exotic here, it just runs the worker and starts fast enough that I stopped thinking about the process. The whole thing is a few dozen lines. The other half of the fix was not writing as much in the first place. I sampled 400 active feeds and only 11.8% had posted anything in the last week, while 15.8% had posted nothing in two years, and I was re-reading all of them every hour. Scheduling each feed on its own publishing rhythm took demand from 62,700 crawls an hour to about 700. A queue with one worker keeps you correct. Crawling less is what made it fast. The crawler publishes its own numbers at rssamplifier.com/crawlstats https://rssamplifier.com/crawlstats : throughput per hour, how many feeds are due, what succeeded and what failed in the last day, and the errors as they happen. It is the page I watch instead of guessing, and it is the same page you can watch, so none of the above has to be taken on my word. What I would tell myself The one writer limit was never the problem. Writing to it from twelve places at once was. If your single row write is fast and your transaction hangs, stop reading query plans and go count how many things in your process can open a transaction at the same time.