{"slug": "customer-research-on-reddit-with-an-ai-agent-pull-full-comment-threads-through", "title": "Customer research on Reddit with an AI agent: pull full comment threads through MCP", "summary": "A developer on the ProLevelMax team built an MCP server and REST API that lets AI agents pull full Reddit comment threads as structured data, exposing eight tools including discover, inspect, run and runs-get. The workflow returns a price quote before execution, flags runs above $0.10 for confirmation, and supports cursor-based paging plus JSON, CSV and XLSX downloads. The server connects to Claude.ai, Claude Desktop and ChatGPT via OAuth, and to Claude Code, Codex and Cursor via Bearer-token API keys.", "body_md": "If you have ever tried to answer \"what do people complain about in our category?\" by reading Reddit, you know the useful part is rarely the post. It is the comment thread underneath, three or four levels deep, where someone explains why they returned the product and five people reply with what they bought instead. This post walks through how to get those threads into an AI agent as structured data, using MCP for chat clients and a plain REST API for scripts.\n\nI work on ProLevelMax, the data source used in the examples below. The approach (discover a source, check the price, run it, page through results) is the part worth taking away.\n\nGetting one thread is easy. Getting a few hundred threads reliably is where the time goes:\n\nWhat I wanted instead: the agent asks for \"comments on this post\", gets a price before anything runs, and receives clean records it can count and quote.\n\nProLevelMax exposes the same data three ways:\n\n`https://prolevelmax.com/mcp`. Claude.ai, Claude Desktop and ChatGPT connect with OAuth (no key). Claude Code, Codex and Cursor can use an API key as a Bearer token.`https://prolevelmax.com/SKILL.md`. You create an API key in the dashboard, then send your AI the line `set up https://prolevelmax.com/SKILL.md` together with the key, and it configures itself.`https://prolevelmax.com/v1`, described in an OpenAPI 3.1 file.\nFor Claude Code, the setup line from SKILL.md is:\n\n```\nclaude mcp add --scope user --transport http prolevelmax https://prolevelmax.com/mcp --header \"Authorization: Bearer sk_procli_...\"\n```\n\nOnce connected, the agent has eight tools. The four that matter for research are `procli_discover`, `procli_inspect`, `procli_run` and `procli_runs_get`. The REST endpoints mirror them, so I will use REST for the rest of this post because it is easier to show.\n\n**1. Discover.** You describe what you want in plain words (English or Chinese) and get candidate sources back.\n\n```\ncurl https://prolevelmax.com/v1/discover \\\n  -H \"Authorization: Bearer sk_procli_...\" \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\"query\":\"reddit comments\"}'\n```\n\nThe response has `recommended` and `items`. Each source carries a `scraper` id, a `title`, `bestFor`, a `verdict` (`recommended`, `ok` or `not_recommended`), a `why`, its `pricing` and an `estimate`. For Reddit there are separate sources for search, subreddit feeds, a post with its comments, post comments page by page, and replies under one comment for deep threads.\n\n**2. Inspect.** Before running anything, ask for the input fields and a quote.\n\n```\ncurl https://prolevelmax.com/v1/inspect \\\n  -H \"Authorization: Bearer sk_procli_...\" \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\"scraper\":\"procli_reddit_b\",\"input\":{\"post_id\":\"t3_1abcde\"}}'\n```\n\nThe response shape (values omitted):\n\n```\n{\n  \"scraper\": \"procli_reddit_b\",\n  \"inputFields\": [\n    { \"name\": <string>, \"type\": <string>, \"required\": <bool>, \"description\": <string>, \"paging\": <bool> }\n  ],\n  \"exampleInput\": { ... },\n  \"pagingNote\": <string>,\n  \"quote\": {\n    \"maxCharge\": <number, USD>,\n    \"maxChargeDisplay\": <string>,\n    \"needsConfirmation\": <bool>,\n    \"balanceTooLow\": <bool>\n  }\n}\n```\n\n`quote.maxCharge` is the most this run can cost. `needsConfirmation` is true when the quote is above $0.10. Fields marked `paging: true` are the ones you fill from the previous result to get the next page.\n\n**3. Run.**\n\n```\ncurl https://prolevelmax.com/v1/run \\\n  -H \"Authorization: Bearer sk_procli_...\" \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\"scraper\":\"procli_reddit_b\",\"input\":{\"post_id\":\"t3_1abcde\"}}'\n```\n\nA finished run returns `runId`, `items`, `count`, a `page` object with cursor values for the next page, `cost` (with `amountDisplay`), `quote`, `wallet` (with `balanceDisplay`) and `download` links for `json`, `csv` and `xlsx`. For long jobs, send `\"async\": true` and you get a `202` with a `runId` to poll at `GET /v1/runs/:id`.\n\n**4. Download.** `GET /v1/runs/RUN_ID/download?format=csv` (or `xlsx`, `json`). Nested fields become `a.b` columns in CSV.\n\nThis pulls every page of comments for one Reddit post, refuses to run anything that needs confirmation, and writes the records to a JSON file. Replace the key and the post id.\n\n``` python\nimport json, sys, requests\n\nAPI = \"https://prolevelmax.com/v1\"\nH = {\"Authorization\": \"Bearer sk_procli_...\"}\nSOURCE = \"procli_reddit_b\"           # \"Post comments\", page by page\ninp = {\"post_id\": \"t3_1abcde\"}       # the post you want\ncomments, spent = [], []\n\nfor _ in range(10):                  # hard stop at 10 pages\n    info = requests.post(f\"{API}/inspect\", headers=H, json={\"scraper\": SOURCE, \"input\": inp}).json()\n    if info[\"quote\"][\"needsConfirmation\"]:\n        sys.exit(f\"Quote is {info['quote']['maxChargeDisplay']}, stopping for review\")\n    run = requests.post(f\"{API}/run\", headers=H, json={\"scraper\": SOURCE, \"input\": inp}).json()\n    comments += run[\"items\"]\n    spent.append(run[\"cost\"][\"amountDisplay\"])\n    paging = [f[\"name\"] for f in info[\"inputFields\"] if f.get(\"paging\")]\n    page = run.get(\"page\") or {}\n    if not paging or not any(page.get(n) for n in paging):\n        break                        # no next cursor: done\n    inp.update({n: page[n] for n in paging if page.get(n)})\n\njson.dump(comments, open(\"comments.json\", \"w\"), ensure_ascii=False, indent=2)\nprint(len(comments), \"comments, charges:\", spent, \"balance:\", run[\"wallet\"][\"balanceDisplay\"])\n```\n\nThe script does not hard-code the cursor field name. It reads which input fields are marked `paging` from inspect and copies matching values from the result's `page` object, which is how the docs describe paging for per-call sources. Error handling is left out to keep it short; errors come back as JSON with a stable `type` (for example `insufficient_balance`, `spend_limit`, `invalid_input`, `rate_limited`) you can branch on.\n\nFrom here, feed `comments.json` to whatever model you use and ask for themes with counts and verbatim quotes. Asking for quotes with links keeps the summary honest, because you can check each claim against the thread.\n\nRunning an agent against a paid API is only comfortable if spend has a ceiling. The pieces that give you one:\n\n`inspect` returns `maxCharge`, and the charge never exceeds the quote. The Skill tells agents to state the price and wait for your OK when the quote is above $0.10.`spend_limit` instead of charging.`POST /v1/schedules`), each schedule takes a `runCap` and an optional `monthlyCap`, and `dryRun: true` previews the cost without saving anything.\nBilling is pay per use: some sources charge per call, others per result plus a per-run fee with a minimum. Most Reddit sources are per call. Current rates are on the [pricing page](https://prolevelmax.com/pricing), and new accounts get $1 of credit after email verification, which is enough to try the flow above.\n\n`429 too_many_runs`.` maxItems`. A synchronous run waits at most 300 seconds (` timeout`), so use `async` for big jobs.\nIf something in the flow is unclear or broken, email [toph@prolevelmax.com](mailto:toph@prolevelmax.com).", "url": "https://wpnews.pro/news/customer-research-on-reddit-with-an-ai-agent-pull-full-comment-threads-through", "canonical_source": "https://dev.to/toph_a_0607897a5c9c5aca5c/customer-research-on-reddit-with-an-ai-agent-pull-full-comment-threads-through-mcp-2451", "published_at": "2026-10-18 10:43:05+00:00", "updated_at": "2026-10-18 10:44:48.612957+00:00", "lang": "en", "topics": ["ai-agents", "agent-protocols", "ai-tools", "developer-tools", "ai-products"], "entities": ["ProLevelMax", "Reddit", "Claude", "ChatGPT", "Claude Code", "Codex", "Cursor", "Anthropic"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/customer-research-on-reddit-with-an-ai-agent-pull-full-comment-threads-through", "markdown": "https://wpnews.pro/news/customer-research-on-reddit-with-an-ai-agent-pull-full-comment-threads-through.md", "text": "https://wpnews.pro/news/customer-research-on-reddit-with-an-ai-agent-pull-full-comment-threads-through.txt", "jsonld": "https://wpnews.pro/news/customer-research-on-reddit-with-an-ai-agent-pull-full-comment-threads-through.jsonld"}}