{"slug": "my-side-project-s-free-tiers-were-dying-so-i-counted-what-each-service-actually", "title": "My side project's free tiers were dying, so I counted what each service actually bills", "summary": "A developer behind the movie discovery side project Galaxy Movies audited why its Vercel, Upstash Redis, and OpenAI free tiers were being exhausted, finding that bulk AI scrapers and SEO crawlers were triggering full server-side renders and that Upstash bills each pipeline member as a separate command. The fixes included serving a thin render path to non-indexing bots, moving robots.txt to a static file with a Disallow list for crawlers like GPTBot and ClaudeBot, lengthening CDN cache windows, and collapsing rate limiting into a single Lua script billed once per EVAL.", "body_md": "Galaxy Movies is a movie discovery side project ~ Next.js Pages Router on Vercel, TMDB for data, Upstash Redis for caching & rate limiting, and OpenAI for a chat assistant & generated synopses. Everything runs on free tiers, which is the whole point of a side project.\n\nA few weeks ago all three of those free tiers started groaning at once. Vercel function usage was climbing, Upstash was burning through its daily command quota well before midnight, and OpenAI spend had a suspicious correlation with traffic spikes that produced no actual readers.\n\nThe fixes all came down to the same realization: **every \"free\" tier bills a different unit, and I was counting the wrong things.** Here's what was actually billing me, and what I changed.\n\nVercel bills function *duration*, and my movie and TV detail pages were expensive: five parallel TMDB fetches (title, watch providers, credits, ratings, recommendations), plus Redis reads for the AI content slots, all in `getServerSideProps`.\n\nThe expensive traffic wasn't humans. SEO crawlers and AI scrapers sweep the entire TMDB ID space in bulk, and every one of those detail pages ran the full fetch chain. They don't index anything and send zero readers. Pure cost.\n\nThe fix was a thin render path just for them:\n\n```\n// Bots that don't index get a minimal render. They sweep the TMDB id\n// space in bulk and don't need providers, credits, videos, or AI slots.\nif (isBot(ua) && !isSearchEngine(ua)) {\n  const titleRes = await fetch(`https://api.themoviedb.org/3/${media}/${id}?api_key=${apiKey}`)\n  // on failure return notFound or errorProps; otherwise:\n  context.res.setHeader('Cache-Control', 'private, no-store')\n  return { props: { ...thinProps, media, title, topCast: [] } }\n}\n```\n\nTwo details matter here:\n\n`isSearchEngine` split.`isBot` matches a broad allowlist, but Googlebot, Bingbot, and DuckDuckGo still get the full render since they index. Social preview bots get the thin version. Whether a bot indexes is the only question that matters.`private, no-store` is critical.\nI also found `robots.txt` was being rendered by a Next.js *page*, which meant every crawler's very first request was a function invocation. It moved to a static `public/robots.txt`, which doubles as an enforcement layer:\n\n```\nUser-agent: AhrefsBot\nUser-agent: SemrushBot\nUser-agent: MJ12bot\nUser-agent: Bytespider\nUser-agent: GPTBot\nUser-agent: ClaudeBot\nUser-agent: CCBot\n# ...several more\nDisallow: /\n```\n\nYes, `Disallow` is an honor system, but the big commercial crawlers honor it, and they were most of the volume.\n\nFinally, I lengthened the SSR CDN cache windows: `s-maxage` from 1 hour to 1 day, `stale-while-revalidate` from 1 day to 1 week. Repeat crawler hits on popular titles now terminate at the edge instead of running SSR again.\n\nThis one caught me off guard. Upstash's free tier bills *per command*, and a pipeline doesn't save you, because **each member of a pipeline is its own billed command**. My request paths fanned out into Redis calls that felt free because \"Redis is fast.\"\n\nThe audit turned up some embarrassing finds. Cache reads on detail pages were ~97% misses, so I was paying a billed command for a null almost every time. Rate limiting cost up to three commands per check. A dataset updated once a day was being fetched again on every pageview.\n\nThe fixes, roughly in order of satisfaction:\n\n**Rate limiting became one Lua script.** `INCRBY` plus a conditional `EXPIRE` plus a TTL recovery check run inside a single `EVAL`, billed once no matter how many `redis.call()` s it makes:\n\n```\nlocal count = redis.call(\"INCRBY\", KEYS[1], ARGV[1])\nif count <= tonumber(ARGV[1]) then\n  redis.call(\"EXPIRE\", KEYS[1], ARGV[2])\nelseif count > tonumber(ARGV[3]) and redis.call(\"TTL\", KEYS[1]) < 0 then\n  redis.call(\"EXPIRE\", KEYS[1], ARGV[2])\nend\nreturn count\n```\n\nThe TTL check handles a real edge case: if a key somehow loses its expiry, a blocked client fixes it on the next attempt instead of being rate limited forever.\n\n**Check caches before paying the rate limit write.** `/api/prices` was rate limiting requests that would have been cache hits, paying a write before doing a read. Reorder, and every cache hit is one command instead of two.\n\n**Memoize what changes daily.** The `streaming-changes` diffs are rewritten once a day by cron but were being fetched per key on every pageview. A per instance LRU holding results for ten minutes turns repeat page loads into zero Redis commands:\n\n``` js\nconst changesMemo = new LRUCache<string, Record<string, ProviderChanges>>({ max: 64, ttl: 10 * 60 * 1000 })\n\nexport async function getProviderChanges(providerKeys = [], { fresh = false } = {}) {\n  const memoKey = [...providerKeys].sort().join(',')\n  if (!fresh) {\n    const memoized = changesMemo.get(memoKey)\n    if (memoized) return memoized\n  }\n  // ...Redis reads, then changesMemo.set(memoKey, changes)\n}\n```\n\nThe `fresh` flag exists because the cron itself must never match subscribers against a diff it just replaced. Memoization that's correct *most* of the time is a bug with extra steps.\n\n**Batch the counters.** Every `HINCRBY` is billed, so AI usage and analytics deltas accumulate in a `Map` and flush every 30 seconds (or 50 fields). Stats lag by up to half a minute and a dying instance loses its unflushed tail, a fine trade for telemetry.\n\n**Restructure data to match access patterns.** Detail pages fetch both AI content slots in one `MGET` instead of two `GET` s. Generated lists moved from one key per slug into a single hash, where `HGETALL` is one command replacing an unbounded key fanout.\n\nNet result: roughly half the daily command volume, with quota headroom for the first time in weeks.\n\nThe AI features on detail pages (generated synopses, similar title blurbs) trigger an OpenAI call on a cache miss. Guess who was causing cache misses at scale: the same scrapers from Act 1. Bots were literally generating my content.\n\nSo bots got read only access: they can consume AI content that's already cached but can never trigger generation (`generate: false` on the cache lookup). Human traffic is unaffected; scraper sweeps now cost zero tokens.\n\nThe other fixes were less dramatic but added up:\n\n`failed` with an attempt counter and gives up after 3 tries, instead of burning a `discover` call on every daily run until the heat death of the universe.`reasoning_effort`: `none` for latency sensitive tasks, `low` only for the discover generator. Real token usage gets logged so I can see where spend actually goes.\nThe scariest scenario isn't gradual spend. It's a quota or billing problem silently converting every request into an error, or worse, into retry storms. Billing and auth failures (`401`/` 402`, `insufficient_quota`) now trip a shared kill switch with a one hour cooldown:\n\n```\nexport function isQuotaError(err: unknown): boolean {\n  if (!(err instanceof OpenAI.APIError)) return false\n  if (err.status === 401 || err.status === 402) return true\n  const code = err.code || err.error?.code\n  return code === 'insufficient_quota'\n      || code === 'billing_hard_limit_reached'\n      || code === 'billing_not_active'\n}\n```\n\nWhile it's tripped: `/api/chat` returns `503 { code: 'ai_unavailable' }` *before* spending budget, the chat widget checks `/api/aiStatus` on mount and hides itself entirely instead of showing an error bubble, the generators skip their calls, and the content cron skips the run *without* burning seed attempts, since an outage isn't a seed's fault. Crucially, ordinary `429 rate_limit_exceeded` errors don't trip it; those are transient.\n\nOne deliberate choice throughout: **degrade in the direction that saves money.** When Redis errors on a check that protects spend, the code falls back to a per instance limiter and never fails open. An in memory limiter might be less accurate, but a Redis outage can't be used to run up my OpenAI bill.\n\nHonest bookkeeping: the cost cuts shipped two bugs worth their own paragraph.\n\nAfter moving the chat cache to Redis, identical requests started hitting on every quick action prompt, and the cache hit path returned plain JSON that the SSE client never parsed. Cache hits went from \"the fast path\" to \"the broken path\" overnight. Fix: return cache hits through the SSE stream like everything else.\n\nAnd when bots stopped triggering AI generation, Lighthouse CI got caught in the crossfire. It audits the real detail pages, and its UA matched the bot list, so it was suddenly measuring the thin render. `Chrome-Lighthouse` went on the search engine side of the check.\n\nBoth bugs were the same shape: **a filter added at the edge changed behavior for a consumer I forgot about.** If your bot detection doesn't have a list of *legitimate* automated consumers, it will find one for you. In production. On a Sunday.\n\n`robots.txt`).\nThe site is [galaxymovies.app](https://www.galaxymovies.app) if you want to see the thin renders in the wild. View source as a human, please; the bots are expensive enough. Happy to compare notes in the comments if you've fought the same free tier math!", "url": "https://wpnews.pro/news/my-side-project-s-free-tiers-were-dying-so-i-counted-what-each-service-actually", "canonical_source": "https://dev.to/nickfasulo/my-side-projects-free-tiers-were-dying-so-i-counted-what-each-service-actually-bills-27m1", "published_at": "2026-10-08 22:41:27+00:00", "updated_at": "2026-10-08 22:48:47.634886+00:00", "lang": "en", "topics": ["ai-crawlers", "ai-infrastructure", "ai-tools", "developer-tools"], "entities": ["Galaxy Movies", "Vercel", "Upstash", "OpenAI", "TMDB", "GPTBot", "ClaudeBot", "Bytespider"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/my-side-project-s-free-tiers-were-dying-so-i-counted-what-each-service-actually", "markdown": "https://wpnews.pro/news/my-side-project-s-free-tiers-were-dying-so-i-counted-what-each-service-actually.md", "text": "https://wpnews.pro/news/my-side-project-s-free-tiers-were-dying-so-i-counted-what-each-service-actually.txt", "jsonld": "https://wpnews.pro/news/my-side-project-s-free-tiers-were-dying-so-i-counted-what-each-service-actually.jsonld"}}