I opened the public job pages on Upwork Scout while writing this, expecting to screenshot a nice feed. Every one of the eleven category pages said the same thing: "No public sample for this category right now." Every one of them also carried a noindex tag.
The design did exactly what I told it to. Nothing thin got indexed, nothing crashed, the pages returned 200 from the CDN. And that is the problem I want to write about, because from the outside I genuinely cannot tell you why they are empty. At least four different causes produce that exact page, and the code that renders it throws away the information that would separate them.
Upwork Scout is my job-alert SaaS for freelancers. A cron job scrapes Upwork per category, filters jobs against each user's rules, scores the survivors with an LLM, and emails them. I wrote about the cost side in One Scrape Per Category, Not Per User and the matcher in Cheap Filters First, LLM Last.
Every job the scanner stores lands in a Firestore jobs collection. That firehose was private, so I added public pages on top of it: /jobs as an index and /jobs/[category] for each of the eleven categories. The idea is the standard aggregator play. Show a recent sample with snippet-length excerpts and outbound links, let search engines find "AI voice agent jobs on Upwork", and convert visitors to alerts.
The data layer is one function:
export const PUBLIC_WINDOW_DAYS = 7;
export const MIN_JOBS_TO_INDEX = 3; // below this a category page is noindex
export async function recentJobsByCategory() {
const buckets = new Map(CATEGORIES.map((c) => [c.id, []]));
try {
const { jobsCol } = await import("./db");
const cutoff = Date.now() - PUBLIC_WINDOW_DAYS * 24 * 3600_000;
const snap = await jobsCol()
.where("fetchedAt", ">", cutoff)
.orderBy("fetchedAt", "desc")
.limit(400)
.get();
// bucket each job into its categories, max 25 per bucket
} catch {
// unreachable datastore (e.g. env-less build) -> empty buckets
}
return buckets;
}
The pages use ISR with revalidate = 1800, and generateMetadata sets robots: { index: false, follow: true } when a category has fewer than three jobs.
There were a few decisions in here I still like. The public card shows a spend band ("$1k+ spent", "has hired before") instead of the client's exact spend, because the page is a teaser and exact figures belong to logged-in users. Excerpts are cut at 160 characters. Outbound links are nofollow. And the function never throws, because a public SEO page that 500s at build time is worse than one that renders empty.
That last decision is the one that came back around.
When I traced backwards from "every bucket is empty", I found four distinct paths, and they all converge on the same HTML.
1. Nobody subscribes to that category. This is by design. The scanner only scrapes categories that at least one active user has picked, most-subscribed first:
let activeCats = CATEGORIES.filter((c) => catCounts.has(c.id));
A category with zero subscribers never gets scraped, so its public page has nothing to show. The empty-state copy was written for exactly this case: "Scanning activates when subscribers pick a category."
2. The daily spend budget ran out. Scraping is billed per item, so the cron route checks a global daily budget before fetching. If the budget is spent, activeCats becomes an empty array and that run fetches nothing. Run it enough times with the budget exhausted and the seven-day window drains.
3. The scrape itself failed. Fetches run under Promise.allSettled, and a rejected category gets a console.error and is skipped. That is correct for alerts (one bad category should not block the other ten), but it means a scraper that fails every run for a week produces no jobs and no visible error anywhere a visitor or I would look.
4. The read failed. If the Firestore query on the public page throws, for credentials, quota, a missing index, anything, the bare catch {} returns eleven empty buckets. The comment says "env-less build", but the catch does not know it is at build time. It catches everything, forever.
Four causes. One of them is healthy (nobody picked that category), one is a business limit working as intended, and two are outages. The page renders all four identically, and the copy confidently asserts the first one.
That is the part that bothers me. An empty state is a claim. Mine claims "no subscribers yet" without checking.
The 400-row window is shared. The query takes the 400 newest jobs across all categories and then buckets them. The page header says "the last 7 days", but if a few busy categories produce 400 jobs in two days, the effective window for everyone shrinks to two days. A quiet category can fall under the three-job index threshold not because it is quiet, but because its neighbours are loud. I did this deliberately to avoid composite indexes (one range scan, filter in memory), and I would still make that trade at this size, but the "7 days" on the page is a ceiling, not a promise.
Every render reads the datastore twice. generateMetadata needs the count to decide noindex, and the page needs the jobs to render. Both call recentJobsByCategory(). Next only dedupes fetch automatically; a Firestore SDK call is not a fetch, so each regeneration of each category page runs the 400-document query twice. The Next docs call this out directly and recommend wrapping the function in React's cache:
import { cache } from "react";
export const recentJobsByCategory = cache(async () => { /* same body */ });
That is a two-line fix I should have written on day one.
None of this needs a rewrite. It needs the function to stop flattening information.
Return a status, not just data. Instead of Map<string, PublicJob[]>, return the buckets plus a reason: ok, read_failed, or stale. A failed read can still render an empty page for visitors, but it should log loudly and it should not pretend to be the healthy case.
Read what the scanner already knows. Here is the embarrassing bit. The cron route already writes meta/scan at the end of every run with lastRunAt and stats, including a categoriesScanned string listing exactly which categories it fetched, or (none). The public page never reads it. One extra document read per render would let the empty state say the true thing: "not scanned because no subscribers", "scanned, nothing new in 7 days", or "scanner has not run since Tuesday".
Separate the visitor message from the operator signal. Visitors do not need to know my spend budget ran out. I do. The page can keep a friendly empty state and still emit a log line, or a tiny /api/health/public-jobs endpoint I can point an uptime check at, that fails when no category has been scanned in 24 hours.
Memoize with cache. Halves the reads per regeneration.
Keep the noindex gate. This was the one part that worked perfectly. I originally built the sitemap from live counts too, listing only categories above the threshold. I changed it to list all eleven unconditionally and let the per-page noindex meta be the real signal, because a sitemap is a hint and was being generated from the same swallowable query. That turned out to be the right instinct: the sitemap is stable, and the page itself is the only place that decides whether it deserves indexing.
"Never throw" is a reasonable rule for a public page. But a function that never throws has to put the failure somewhere, and if the answer is "nowhere", you have built something that is indistinguishable from success with zero results. I made the same mistake in a different shape in You Can Retry a Write. You Cannot Retry an Email., where the question was which side effects are allowed to fail silently. The answer there was "decide per side effect, on purpose". The answer here is the same: decide per failure, on purpose, and write the decision down where you will see it.
The empty page is not the bug. The empty page that cannot explain itself is.
If you have built public pages on top of a pipeline that only runs on demand, how do you tell "nothing happened" from "nothing could happen"? I would like to steal your approach.