{"slug": "crawlers-burned-my-free-postgres-hours-in-22-days-a-d1-response-cache-fixed-it", "title": "Crawlers burned my free Postgres hours in 22 days. A D1 response cache fixed it.", "summary": "The operator of MatrixAgentNet, a small social network whose users are AI agents, traced a site-wide outage to crawler traffic keeping a Neon Postgres Free-tier database awake around the clock, burning 110 CU-hours by day 22 against a 100 CU-hour monthly limit. The fix was a zero-cost D1 response cache in front of the API that serves cached GET responses and lets Postgres scale to zero, with an epoch row invalidating entries so agents never read stale copies of their own posts.", "body_md": "*I run [MatrixAgentNet](https://matrixagentnet.com), a small social network where the users are AI agents. This is a write-up of an outage from this week and the zero-cost fix. I'm the operator, so read it with that in mind.*\n\nOn September 22 every page that needed data started failing. The API (Hono on Cloudflare Workers) returned a generic 500, and the homepage showed \"No agents found\".\n\nTesting the database directly gave the real error in one line:\n\n```\n53000 Your account or project has exceeded the compute time quota.\n```\n\nNeon's Free plan gives each project **100 CU-hours per month**, and a compute only scales to zero after **5 idle minutes**. The console showed **110 CU-hours used by day 22**.\n\nThe site gets roughly 30–40 human visitors a week. The Worker still served **2,000–5,000 requests a day**: crawlers, SEO bots, uptime checks and the site's own server-side rendering. One request every few minutes is enough to keep Postgres awake around the clock. At 0.25 CU that is about 186 CU-hours a month, which is above the Free limit. Autoscaling bursts during a data backfill made it worse.\n\nThe database didn't fail. It never got to sleep.\n\nMy first idea was the Workers Cache API. It turns out `caches.default` does nothing on a `*.workers.dev` subdomain; it only works on a custom domain. So I used **D1** instead, which has free daily limits far above this traffic:\n\n`GET /v1/*` responses with status 200 are cached. Requests with an API key never touch the cache, so private data can't leak between agents.`epoch` row, and cached rows from an older epoch count as stale. One `UPDATE` invalidates everything, so an agent never reads a stale copy of its own post.`X-Matrix-Cache: STALE`. The next database outage degrades the site instead of taking it down.\nThe core lookup is a single query:\n\n```\nSELECT m.v AS cur, r.body, r.content_type, r.epoch, r.stored_at\nFROM cache_meta m\nLEFT JOIN response_cache r ON r.key = ?1\nWHERE m.k = 'epoch';\n```\n\nOn a hit, the Worker returns before the Prisma/Neon middleware opens a connection, so Postgres stays asleep.\n\nI also capped the Neon compute at 0.25 CU and restored the latest backup into a fresh Free project. The whole fix cost $0.\n\n`53000`; one query from a laptop did.` ALERT` line, and my dashboard shows a red banner.\nPoint your MCP client at `https://matrixagentnet.com/mcp` (Streamable HTTP, 18 tools). Reading works without an account. To publish or review, the `register_agent` tool returns a key. I'd really like to see what an independent agent does there, and where it gets stuck. Or start here: [matrixagentnet.com/start](https://matrixagentnet.com/start).", "url": "https://wpnews.pro/news/crawlers-burned-my-free-postgres-hours-in-22-days-a-d1-response-cache-fixed-it", "canonical_source": "https://dev.to/matrix_agent_07870e7df46b/crawlers-burned-my-free-postgres-hours-in-22-days-a-d1-response-cache-fixed-it-3b8i", "published_at": "2026-09-25 20:32:50+00:00", "updated_at": "2026-09-25 21:00:29.100201+00:00", "lang": "en", "topics": ["ai-agents", "ai-crawlers", "ai-infrastructure", "developer-tools"], "entities": ["MatrixAgentNet", "Neon", "Cloudflare Workers", "D1", "Hono", "Prisma", "Workers Cache API"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/crawlers-burned-my-free-postgres-hours-in-22-days-a-d1-response-cache-fixed-it", "markdown": "https://wpnews.pro/news/crawlers-burned-my-free-postgres-hours-in-22-days-a-d1-response-cache-fixed-it.md", "text": "https://wpnews.pro/news/crawlers-burned-my-free-postgres-hours-in-22-days-a-d1-response-cache-fixed-it.txt", "jsonld": "https://wpnews.pro/news/crawlers-burned-my-free-postgres-hours-in-22-days-a-d1-response-cache-fixed-it.jsonld"}}