How I built a Reddit lead finder with RSS, an LLM scorer and a Node worker

A solo developer built a Reddit lead-finding system that polls public Atom RSS feeds across watched subreddits, matches posts against workspace keywords, and scores each match with an OpenAI model to judge whether the poster is looking for a product like the user's. The architecture splits a Next.js dashboard from a separate Node worker with its own polling and scoring loops, routes requests through a proxy pool with configurable lanes and retry limits, and drafts replies for users to edit and post themselves rather than posting automatically. The developer reports lessons including reading environment variables at call time rather than import time in ESM and the maintenance cost of duplicated Mongoose schemas across the two deploys.

Every time I shipped a side project, I lost evenings doing the same thing: searching Reddit for posts like "is there a tool that does X?" or "any alternative to Y?". Those posts are some of the warmest early leads a solo founder can get, but they are scattered across dozens of subreddits and fall off "new" within hours. So I built a small system to do the finding for me. This post is about how it's put together, what broke along the way, and what it still can't do. No growth hacks, just architecture. There are two deploys and one database: The app never talks to Reddit directly. The dashboard writes workspaces a product description, subreddits, keywords to Mongo; the worker reads them, fetches posts, stores matches and scores them; the dashboard reads matches back. When the app needs something right now validating a subreddit, triggering a backfill for a new workspace , it calls the worker's search API with a shared secret header. Why split it? Next.js on a container is great at serving requests and terrible at long-running loops. Keeping the loops in a separate process means a deploy of the web app never interrupts a polling cycle, and a stuck Reddit request never ties up a web worker. The obvious route is Reddit's .json listings. In my setup those got 403s once traffic went through proxies, so the worker reads the public Atom feeds instead: /r/{subreddit}/new/.rss for each watched subreddit .rss for keyword sweeps across Reddit RSS gives you less than JSON no scores, no comment counts , but for "has someone just asked for a thing like mine?" you mostly need title, body, author, link and timestamp, and Atom has all of those. Requests go out through a proxy pool with a configurable number of "lanes", a minimum gap between requests and a bounded retry count on 403s. All of that is configured through env variables, which brings me to the first lesson. worker/index.js is basically a handful of timers: new for all watched subreddits, match against keywords, store new matches. Splitting fetching from scoring was one of the better decisions. Fetching is cheap and bursty; scoring costs money and has rate limits. With separate loops, a slow LLM call never delays the next poll, and a Reddit outage never leaves the scorer idle with a backlog it can't reach. Keyword matching alone is noisy. "Looking for a CRM" matches a buyer, but so does "I hate every CRM I've tried, rant incoming" and "I built a CRM, roast it". The scorer sends each match plus the workspace's product description to an OpenAI model and asks how likely it is that this post is someone looking for a solution like this product. The interesting part isn't the prompt, it's the guardrails around it: lead-quality.json fixture of real-world-style posts with expected outcomes, and every matching bug fix adds a case. It's the closest thing to a test suite for "is the filter still sane?". For a lead worth answering, the app drafts a reply. The user edits it and posts it themselves. The tool never posts to Reddit on anyone's behalf. That's a product decision, not a technical limitation. Reddit communities are very quick to spot bots, and I didn't want to build a spam machine that gets its users banned. Drafting saves time; posting stays human. 1. Read env at call time, not at import. The worker is ESM, and imports are evaluated before the body of the entry file. If a module reads process.env.SOMETHING at the top level, it can run before the .env.local loader does, and you get undefined in production but not on your laptop. Now the env loader is the very first import, and modules read config inside functions getWebshareKey , pollExcludeUserIds , with a test that fails if a value gets captured at import time. 2. Duplicated schemas will bite you. The app and the worker each have their own Mongoose schemas for the same collections. It keeps the worker deployable without the app's code, but every field change has to be made twice. I've left myself a loud note about it; a shared package would be the real fix. 3. Anything that deletes data gets a kill switch. The cleanup job exports old matches and then deletes them. It ships with a disable flag and refuses to delete if email isn't configured so the export can't silently fail first . Boring, and exactly what you want at 2am. 4. Make one command the definition of "done". npm run verify runs a syntax check, the worker tests including the lead-quality fixture and a mocked backfill pipeline test , next build and Playwright end-to-end tests. CI runs the same command on every PR. Solo projects need this more than team ones, because nobody else is going to catch it. 5. Build the web app as an image, not on the server. GitHub Actions builds a Docker image Next standalone output, Node 22 Alpine, non-root and the host just pulls it. One gotcha: NEXT PUBLIC variables get inlined at build time, so changing them in the hosting dashboard does nothing until you rebuild. If you want to build something similar, the core is small: an RSS fetcher, a keyword matcher, a queue of unscored items and an LLM call with strict caps. The rest is making it boring and reliable. I turned this into a product called IndiePilot https://indiepilot.app there's a free plan for founders who'd rather not run the plumbing themselves. Either way, I'm happy to answer questions about the architecture in the comments: what would you do differently?