My side project's free tiers were dying, so I counted what each service actually bills

A developer behind the movie discovery side project Galaxy Movies audited why its Vercel, Upstash Redis, and OpenAI free tiers were being exhausted, finding that bulk AI scrapers and SEO crawlers were triggering full server-side renders and that Upstash bills each pipeline member as a separate command. The fixes included serving a thin render path to non-indexing bots, moving robots.txt to a static file with a Disallow list for crawlers like GPTBot and ClaudeBot, lengthening CDN cache windows, and collapsing rate limiting into a single Lua script billed once per EVAL.

Galaxy Movies is a movie discovery side project ~ Next.js Pages Router on Vercel, TMDB for data, Upstash Redis for caching & rate limiting, and OpenAI for a chat assistant & generated synopses. Everything runs on free tiers, which is the whole point of a side project. A few weeks ago all three of those free tiers started groaning at once. Vercel function usage was climbing, Upstash was burning through its daily command quota well before midnight, and OpenAI spend had a suspicious correlation with traffic spikes that produced no actual readers. The fixes all came down to the same realization: every "free" tier bills a different unit, and I was counting the wrong things. Here's what was actually billing me, and what I changed. Vercel bills function duration , and my movie and TV detail pages were expensive: five parallel TMDB fetches title, watch providers, credits, ratings, recommendations , plus Redis reads for the AI content slots, all in getServerSideProps . The expensive traffic wasn't humans. SEO crawlers and AI scrapers sweep the entire TMDB ID space in bulk, and every one of those detail pages ran the full fetch chain. They don't index anything and send zero readers. Pure cost. The fix was a thin render path just for them: // Bots that don't index get a minimal render. They sweep the TMDB id // space in bulk and don't need providers, credits, videos, or AI slots. if isBot ua && isSearchEngine ua { const titleRes = await fetch https://api.themoviedb.org/3/${media}/${id}?api key=${apiKey} // on failure return notFound or errorProps; otherwise: context.res.setHeader 'Cache-Control', 'private, no-store' return { props: { ...thinProps, media, title, topCast: } } } Two details matter here: isSearchEngine split. isBot matches a broad allowlist, but Googlebot, Bingbot, and DuckDuckGo still get the full render since they index. Social preview bots get the thin version. Whether a bot indexes is the only question that matters. private, no-store is critical. I also found robots.txt was being rendered by a Next.js page , which meant every crawler's very first request was a function invocation. It moved to a static public/robots.txt , which doubles as an enforcement layer: User-agent: AhrefsBot User-agent: SemrushBot User-agent: MJ12bot User-agent: Bytespider User-agent: GPTBot User-agent: ClaudeBot User-agent: CCBot ...several more Disallow: / Yes, Disallow is an honor system, but the big commercial crawlers honor it, and they were most of the volume. Finally, I lengthened the SSR CDN cache windows: s-maxage from 1 hour to 1 day, stale-while-revalidate from 1 day to 1 week. Repeat crawler hits on popular titles now terminate at the edge instead of running SSR again. This one caught me off guard. Upstash's free tier bills per command , and a pipeline doesn't save you, because each member of a pipeline is its own billed command . My request paths fanned out into Redis calls that felt free because "Redis is fast." The audit turned up some embarrassing finds. Cache reads on detail pages were ~97% misses, so I was paying a billed command for a null almost every time. Rate limiting cost up to three commands per check. A dataset updated once a day was being fetched again on every pageview. The fixes, roughly in order of satisfaction: Rate limiting became one Lua script. INCRBY plus a conditional EXPIRE plus a TTL recovery check run inside a single EVAL , billed once no matter how many redis.call s it makes: local count = redis.call "INCRBY", KEYS 1 , ARGV 1 if count <= tonumber ARGV 1 then redis.call "EXPIRE", KEYS 1 , ARGV 2 elseif count tonumber ARGV 3 and redis.call "TTL", KEYS 1 < 0 then redis.call "EXPIRE", KEYS 1 , ARGV 2 end return count The TTL check handles a real edge case: if a key somehow loses its expiry, a blocked client fixes it on the next attempt instead of being rate limited forever. Check caches before paying the rate limit write. /api/prices was rate limiting requests that would have been cache hits, paying a write before doing a read. Reorder, and every cache hit is one command instead of two. Memoize what changes daily. The streaming-changes diffs are rewritten once a day by cron but were being fetched per key on every pageview. A per instance LRU holding results for ten minutes turns repeat page loads into zero Redis commands: js const changesMemo = new LRUCache<string, Record<string, ProviderChanges { max: 64, ttl: 10 60 1000 } export async function getProviderChanges providerKeys = , { fresh = false } = {} { const memoKey = ...providerKeys .sort .join ',' if fresh { const memoized = changesMemo.get memoKey if memoized return memoized } // ...Redis reads, then changesMemo.set memoKey, changes } The fresh flag exists because the cron itself must never match subscribers against a diff it just replaced. Memoization that's correct most of the time is a bug with extra steps. Batch the counters. Every HINCRBY is billed, so AI usage and analytics deltas accumulate in a Map and flush every 30 seconds or 50 fields . Stats lag by up to half a minute and a dying instance loses its unflushed tail, a fine trade for telemetry. Restructure data to match access patterns. Detail pages fetch both AI content slots in one MGET instead of two GET s. Generated lists moved from one key per slug into a single hash, where HGETALL is one command replacing an unbounded key fanout. Net result: roughly half the daily command volume, with quota headroom for the first time in weeks. The AI features on detail pages generated synopses, similar title blurbs trigger an OpenAI call on a cache miss. Guess who was causing cache misses at scale: the same scrapers from Act 1. Bots were literally generating my content. So bots got read only access: they can consume AI content that's already cached but can never trigger generation generate: false on the cache lookup . Human traffic is unaffected; scraper sweeps now cost zero tokens. The other fixes were less dramatic but added up: failed with an attempt counter and gives up after 3 tries, instead of burning a discover call on every daily run until the heat death of the universe. reasoning effort : none for latency sensitive tasks, low only for the discover generator. Real token usage gets logged so I can see where spend actually goes. The scariest scenario isn't gradual spend. It's a quota or billing problem silently converting every request into an error, or worse, into retry storms. Billing and auth failures 401 / 402 , insufficient quota now trip a shared kill switch with a one hour cooldown: export function isQuotaError err: unknown : boolean { if err instanceof OpenAI.APIError return false if err.status === 401 || err.status === 402 return true const code = err.code || err.error?.code return code === 'insufficient quota' || code === 'billing hard limit reached' || code === 'billing not active' } While it's tripped: /api/chat returns 503 { code: 'ai unavailable' } before spending budget, the chat widget checks /api/aiStatus on mount and hides itself entirely instead of showing an error bubble, the generators skip their calls, and the content cron skips the run without burning seed attempts, since an outage isn't a seed's fault. Crucially, ordinary 429 rate limit exceeded errors don't trip it; those are transient. One deliberate choice throughout: degrade in the direction that saves money. When Redis errors on a check that protects spend, the code falls back to a per instance limiter and never fails open. An in memory limiter might be less accurate, but a Redis outage can't be used to run up my OpenAI bill. Honest bookkeeping: the cost cuts shipped two bugs worth their own paragraph. After moving the chat cache to Redis, identical requests started hitting on every quick action prompt, and the cache hit path returned plain JSON that the SSE client never parsed. Cache hits went from "the fast path" to "the broken path" overnight. Fix: return cache hits through the SSE stream like everything else. And when bots stopped triggering AI generation, Lighthouse CI got caught in the crossfire. It audits the real detail pages, and its UA matched the bot list, so it was suddenly measuring the thin render. Chrome-Lighthouse went on the search engine side of the check. Both bugs were the same shape: a filter added at the edge changed behavior for a consumer I forgot about. If your bot detection doesn't have a list of legitimate automated consumers, it will find one for you. In production. On a Sunday. robots.txt . The site is galaxymovies.app https://www.galaxymovies.app if you want to see the thin renders in the wild. View source as a human, please; the bots are expensive enough. Happy to compare notes in the comments if you've fought the same free tier math