{"slug": "how-i-built-a-reddit-lead-finder-with-rss-an-llm-scorer-and-a-node-worker", "title": "How I built a Reddit lead finder with RSS, an LLM scorer and a Node worker", "summary": "A solo developer built a Reddit lead-finding system that polls public Atom RSS feeds across watched subreddits, matches posts against workspace keywords, and scores each match with an OpenAI model to judge whether the poster is looking for a product like the user's. The architecture splits a Next.js dashboard from a separate Node worker with its own polling and scoring loops, routes requests through a proxy pool with configurable lanes and retry limits, and drafts replies for users to edit and post themselves rather than posting automatically. The developer reports lessons including reading environment variables at call time rather than import time in ESM and the maintenance cost of duplicated Mongoose schemas across the two deploys.", "body_md": "Every time I shipped a side project, I lost evenings doing the same thing: searching Reddit for posts like \"is there a tool that does X?\" or \"any alternative to Y?\". Those posts are some of the warmest early leads a solo founder can get, but they are scattered across dozens of subreddits and fall off \"new\" within hours.\n\nSo I built a small system to do the finding for me. This post is about how it's put together, what broke along the way, and what it still can't do. No growth hacks, just architecture.\n\nThere are two deploys and one database:\n\nThe app never talks to Reddit directly. The dashboard writes workspaces (a product description, subreddits, keywords) to Mongo; the worker reads them, fetches posts, stores matches and scores them; the dashboard reads matches back. When the app needs something right now (validating a subreddit, triggering a backfill for a new workspace), it calls the worker's search API with a shared secret header.\n\nWhy split it? Next.js on a container is great at serving requests and terrible at long-running loops. Keeping the loops in a separate process means a deploy of the web app never interrupts a polling cycle, and a stuck Reddit request never ties up a web worker.\n\nThe obvious route is Reddit's `.json` listings. In my setup those got 403s once traffic went through proxies, so the worker reads the public Atom feeds instead:\n\n`/r/{subreddit}/new/.rss` for each watched subreddit`.rss` for keyword sweeps across Reddit\nRSS gives you less than JSON (no scores, no comment counts), but for \"has someone just asked for a thing like mine?\" you mostly need title, body, author, link and timestamp, and Atom has all of those.\n\nRequests go out through a proxy pool with a configurable number of \"lanes\", a minimum gap between requests and a bounded retry count on 403s. All of that is configured through env variables, which brings me to the first lesson.\n\n`worker/index.js` is basically a handful of timers:\n\n`new` for all watched subreddits, match against keywords, store new matches.\nSplitting fetching from scoring was one of the better decisions. Fetching is cheap and bursty; scoring costs money and has rate limits. With separate loops, a slow LLM call never delays the next poll, and a Reddit outage never leaves the scorer idle with a backlog it can't reach.\n\nKeyword matching alone is noisy. \"Looking for a CRM\" matches a buyer, but so does \"I hate every CRM I've tried, rant incoming\" and \"I built a CRM, roast it\". The scorer sends each match plus the workspace's product description to an OpenAI model and asks how likely it is that this post is someone looking for a solution like this product.\n\nThe interesting part isn't the prompt, it's the guardrails around it:\n\n`lead-quality.json` fixture of real-world-style posts with expected outcomes, and every matching bug fix adds a case. It's the closest thing to a test suite for \"is the filter still sane?\".\nFor a lead worth answering, the app drafts a reply. The user edits it and posts it themselves. The tool never posts to Reddit on anyone's behalf.\n\nThat's a product decision, not a technical limitation. Reddit communities are very quick to spot bots, and I didn't want to build a spam machine that gets its users banned. Drafting saves time; posting stays human.\n\n**1. Read env at call time, not at import.** The worker is ESM, and imports are evaluated before the body of the entry file. If a module reads `process.env.SOMETHING` at the top level, it can run before the `.env.local` loader does, and you get `undefined` in production but not on your laptop. Now the env loader is the very first import, and modules read config inside functions (`getWebshareKey()`, `pollExcludeUserIds()`), with a test that fails if a value gets captured at import time.\n\n**2. Duplicated schemas will bite you.** The app and the worker each have their own Mongoose schemas for the same collections. It keeps the worker deployable without the app's code, but every field change has to be made twice. I've left myself a loud note about it; a shared package would be the real fix.\n\n**3. Anything that deletes data gets a kill switch.** The cleanup job exports old matches and then deletes them. It ships with a disable flag and refuses to delete if email isn't configured (so the export can't silently fail first). Boring, and exactly what you want at 2am.\n\n**4. Make one command the definition of \"done\".** `npm run verify` runs a syntax check, the worker tests (including the lead-quality fixture and a mocked backfill pipeline test), `next build` and Playwright end-to-end tests. CI runs the same command on every PR. Solo projects need this more than team ones, because nobody else is going to catch it.\n\n**5. Build the web app as an image, not on the server.** GitHub Actions builds a Docker image (Next `standalone` output, Node 22 Alpine, non-root) and the host just pulls it. One gotcha: `NEXT_PUBLIC_*` variables get inlined at build time, so changing them in the hosting dashboard does nothing until you rebuild.\n\nIf you want to build something similar, the core is small: an RSS fetcher, a keyword matcher, a queue of unscored items and an LLM call with strict caps. The rest is making it boring and reliable.\n\nI turned this into a product called [IndiePilot](https://indiepilot.app) (there's a free plan) for founders who'd rather not run the plumbing themselves. Either way, I'm happy to answer questions about the architecture in the comments: what would you do differently?", "url": "https://wpnews.pro/news/how-i-built-a-reddit-lead-finder-with-rss-an-llm-scorer-and-a-node-worker", "canonical_source": "https://dev.to/matek075/how-i-built-a-reddit-lead-finder-with-rss-an-llm-scorer-and-a-node-worker-2ana", "published_at": "2026-10-19 13:37:16+00:00", "updated_at": "2026-10-19 13:44:50.678595+00:00", "lang": "en", "topics": ["ai-tools", "large-language-models", "ai-agents"], "entities": ["Reddit", "OpenAI", "Next.js", "Node.js", "MongoDB", "Mongoose"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/how-i-built-a-reddit-lead-finder-with-rss-an-llm-scorer-and-a-node-worker", "markdown": "https://wpnews.pro/news/how-i-built-a-reddit-lead-finder-with-rss-an-llm-scorer-and-a-node-worker.md", "text": "https://wpnews.pro/news/how-i-built-a-reddit-lead-finder-with-rss-an-llm-scorer-and-a-node-worker.txt", "jsonld": "https://wpnews.pro/news/how-i-built-a-reddit-lead-finder-with-rss-an-llm-scorer-and-a-node-worker.jsonld"}}