{"slug": "my-live-job-pages-are-all-empty-and-my-code-cannot-tell-me-why", "title": "My Live Job Pages Are All Empty, and My Code Cannot Tell Me Why", "summary": "A developer building Upwork Scout, a job-alert SaaS that scrapes Upwork per category and scores matches with an LLM, found that all eleven of the site's public job category pages rendered empty with noindex tags, and traced the identical output to four distinct causes: no subscribers for a category, an exhausted daily scraping budget, failed scrapes skipped under Promise.allSettled, and a catch block that silently returns empty buckets when the datastore is unreachable. The root problem, the developer writes, is that the data function never throws and discards the information that would distinguish these failure modes.", "body_md": "I opened the public job pages on [Upwork Scout](https://upwork-scout.com) while writing this, expecting to screenshot a nice feed. Every one of the eleven category pages said the same thing: \"No public sample for this category right now.\" Every one of them also carried a `noindex` tag.\n\nThe design did exactly what I told it to. Nothing thin got indexed, nothing crashed, the pages returned 200 from the CDN. And that is the problem I want to write about, because from the outside I genuinely cannot tell you why they are empty. At least four different causes produce that exact page, and the code that renders it throws away the information that would separate them.\n\nUpwork Scout is my job-alert SaaS for freelancers. A cron job scrapes Upwork per category, filters jobs against each user's rules, scores the survivors with an LLM, and emails them. I wrote about the cost side in [One Scrape Per Category, Not Per User](https://dev.to/nabeelbaghoor/one-scrape-per-category-not-per-user-how-i-kept-a-job-alert-saas-from-going-broke-3e19) and the matcher in [Cheap Filters First, LLM Last](https://dev.to/nabeelbaghoor/cheap-filters-first-llm-last-running-an-ai-matcher-inside-a-cron-job-702).\n\nEvery job the scanner stores lands in a Firestore `jobs` collection. That firehose was private, so I added public pages on top of it: `/jobs` as an index and `/jobs/[category]` for each of the eleven categories. The idea is the standard aggregator play. Show a recent sample with snippet-length excerpts and outbound links, let search engines find \"AI voice agent jobs on Upwork\", and convert visitors to alerts.\n\nThe data layer is one function:\n\n``` js\nexport const PUBLIC_WINDOW_DAYS = 7;\nexport const MIN_JOBS_TO_INDEX = 3; // below this a category page is noindex\n\nexport async function recentJobsByCategory() {\n  const buckets = new Map(CATEGORIES.map((c) => [c.id, []]));\n  try {\n    const { jobsCol } = await import(\"./db\");\n    const cutoff = Date.now() - PUBLIC_WINDOW_DAYS * 24 * 3600_000;\n    const snap = await jobsCol()\n      .where(\"fetchedAt\", \">\", cutoff)\n      .orderBy(\"fetchedAt\", \"desc\")\n      .limit(400)\n      .get();\n    // bucket each job into its categories, max 25 per bucket\n  } catch {\n    // unreachable datastore (e.g. env-less build) -> empty buckets\n  }\n  return buckets;\n}\n```\n\nThe pages use ISR with `revalidate = 1800`, and `generateMetadata` sets `robots: { index: false, follow: true }` when a category has fewer than three jobs.\n\nThere were a few decisions in here I still like. The public card shows a spend band (\"$1k+ spent\", \"has hired before\") instead of the client's exact spend, because the page is a teaser and exact figures belong to logged-in users. Excerpts are cut at 160 characters. Outbound links are `nofollow`. And the function never throws, because a public SEO page that 500s at build time is worse than one that renders empty.\n\nThat last decision is the one that came back around.\n\nWhen I traced backwards from \"every bucket is empty\", I found four distinct paths, and they all converge on the same HTML.\n\n**1. Nobody subscribes to that category.** This is by design. The scanner only scrapes categories that at least one active user has picked, most-subscribed first:\n\n``` js\nlet activeCats = CATEGORIES.filter((c) => catCounts.has(c.id));\n```\n\nA category with zero subscribers never gets scraped, so its public page has nothing to show. The empty-state copy was written for exactly this case: \"Scanning activates when subscribers pick a category.\"\n\n**2. The daily spend budget ran out.** Scraping is billed per item, so the cron route checks a global daily budget before fetching. If the budget is spent, `activeCats` becomes an empty array and that run fetches nothing. Run it enough times with the budget exhausted and the seven-day window drains.\n\n**3. The scrape itself failed.** Fetches run under `Promise.allSettled`, and a rejected category gets a `console.error` and is skipped. That is correct for alerts (one bad category should not block the other ten), but it means a scraper that fails every run for a week produces no jobs and no visible error anywhere a visitor or I would look.\n\n**4. The read failed.** If the Firestore query on the public page throws, for credentials, quota, a missing index, anything, the bare `catch {}` returns eleven empty buckets. The comment says \"env-less build\", but the catch does not know it is at build time. It catches everything, forever.\n\nFour causes. One of them is healthy (nobody picked that category), one is a business limit working as intended, and two are outages. The page renders all four identically, and the copy confidently asserts the first one.\n\nThat is the part that bothers me. An empty state is a claim. Mine claims \"no subscribers yet\" without checking.\n\n**The 400-row window is shared.** The query takes the 400 newest jobs across all categories and then buckets them. The page header says \"the last 7 days\", but if a few busy categories produce 400 jobs in two days, the effective window for everyone shrinks to two days. A quiet category can fall under the three-job index threshold not because it is quiet, but because its neighbours are loud. I did this deliberately to avoid composite indexes (one range scan, filter in memory), and I would still make that trade at this size, but the \"7 days\" on the page is a ceiling, not a promise.\n\n**Every render reads the datastore twice.** `generateMetadata` needs the count to decide `noindex`, and the page needs the jobs to render. Both call `recentJobsByCategory()`. Next only dedupes `fetch` automatically; a Firestore SDK call is not a `fetch`, so each regeneration of each category page runs the 400-document query twice. The Next docs call this out directly and recommend wrapping the function in React's `cache`:\n\n``` js\nimport { cache } from \"react\";\nexport const recentJobsByCategory = cache(async () => { /* same body */ });\n```\n\nThat is a two-line fix I should have written on day one.\n\nNone of this needs a rewrite. It needs the function to stop flattening information.\n\n**Return a status, not just data.** Instead of `Map<string, PublicJob[]>`, return the buckets plus a reason: `ok`, `read_failed`, or `stale`. A failed read can still render an empty page for visitors, but it should log loudly and it should not pretend to be the healthy case.\n\n**Read what the scanner already knows.** Here is the embarrassing bit. The cron route already writes `meta/scan` at the end of every run with `lastRunAt` and stats, including a `categoriesScanned` string listing exactly which categories it fetched, or `(none)`. The public page never reads it. One extra document read per render would let the empty state say the true thing: \"not scanned because no subscribers\", \"scanned, nothing new in 7 days\", or \"scanner has not run since Tuesday\".\n\n**Separate the visitor message from the operator signal.** Visitors do not need to know my spend budget ran out. I do. The page can keep a friendly empty state and still emit a log line, or a tiny `/api/health/public-jobs` endpoint I can point an uptime check at, that fails when no category has been scanned in 24 hours.\n\n**Memoize with `cache`.** Halves the reads per regeneration.\n\n**Keep the noindex gate.** This was the one part that worked perfectly. I originally built the sitemap from live counts too, listing only categories above the threshold. I changed it to list all eleven unconditionally and let the per-page `noindex` meta be the real signal, because a sitemap is a hint and was being generated from the same swallowable query. That turned out to be the right instinct: the sitemap is stable, and the page itself is the only place that decides whether it deserves indexing.\n\n\"Never throw\" is a reasonable rule for a public page. But a function that never throws has to put the failure somewhere, and if the answer is \"nowhere\", you have built something that is indistinguishable from success with zero results. I made the same mistake in a different shape in [You Can Retry a Write. You Cannot Retry an Email.](https://dev.to/nabeelbaghoor/you-can-retry-a-write-you-cannot-retry-an-email-lfo), where the question was which side effects are allowed to fail silently. The answer there was \"decide per side effect, on purpose\". The answer here is the same: decide per failure, on purpose, and write the decision down where you will see it.\n\nThe empty page is not the bug. The empty page that cannot explain itself is.\n\nIf you have built public pages on top of a pipeline that only runs on demand, how do you tell \"nothing happened\" from \"nothing could happen\"? I would like to steal your approach.", "url": "https://wpnews.pro/news/my-live-job-pages-are-all-empty-and-my-code-cannot-tell-me-why", "canonical_source": "https://dev.to/nabeelbaghoor/my-live-job-pages-are-all-empty-and-my-code-cannot-tell-me-why-508b", "published_at": "2026-11-01 21:36:59+00:00", "updated_at": "2026-11-01 21:52:27.178507+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "generative-engine-optimization", "ai-search", "developer-tools"], "entities": ["Upwork Scout", "Upwork", "Firestore", "Next.js"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/my-live-job-pages-are-all-empty-and-my-code-cannot-tell-me-why", "markdown": "https://wpnews.pro/news/my-live-job-pages-are-all-empty-and-my-code-cannot-tell-me-why.md", "text": "https://wpnews.pro/news/my-live-job-pages-are-all-empty-and-my-code-cannot-tell-me-why.txt", "jsonld": "https://wpnews.pro/news/my-live-job-pages-are-all-empty-and-my-code-cannot-tell-me-why.jsonld"}}