{"slug": "how-i-replaced-manual-company-research-with-an-apify-actor-my-cursor-agent-can", "title": "How I replaced manual company research with an Apify Actor my Cursor agent can call", "summary": "Developer Manojaditya Nadar built an Apify Actor called site-intent that visits a fixed set of pages per company domain and returns one structured JSON row per company, then connected it to the Cursor coding agent through the Apify MCP server at https://mcp.apify.com so the agent can call it as a tool. The Actor, built with Crawlee and Playwright, opens each homepage, optionally reads sitemap.xml, follows internal links to pricing, blog and careers pages, and samples posts or an RSS feed to fill fields including has_blog, pricing_page_exists, blog_cadence_band, qualify_ready and pre_segment. Nadar reports running the Actor on four real domains from Cursor for about three cents, replacing manual research that he said could take a week on a list of 800 names.", "body_md": "[Manojaditya Nadar](https://github.com/Manojaditya64)as part of\n\n[Write for Apify](https://apify.com/resources/write-for-apify)- a program for developers sharing original articles about what they've built with Apify.\n\nI run outbound for B2B SaaS. My lead lists tell me a company exists: industry, headcount band, headquarters. They do not tell me whether the blog is still active, pricing is public, or the homepage sounds like a product company or an agency.\n\nFor months I did that research by hand. I would open each site, click through to the blog and pricing pages, and try to remember what I saw. Five companies were manageable. Eight hundred were not.\n\nI built an Apify Actor called site-intent to automate that step. It visits a small, fixed set of pages per company and returns one structured summary row per domain. Then I connected it to Cursor through the [Apify MCP server](https://docs.apify.com/mcp) so my AI agent can run it like any other tool during a conversation.\n\nThis article explains what the Actor does, how I designed its inputs so agents do not send the wrong JSON, and what happened when I ran it on four real domains from Cursor for about three cents.\n\n## Prerequisites\n\nYou will need:\n\n- An [Apify account](https://console.apify.com) and API token\n- [Cursor](https://cursor.com) or another client that supports MCP (Model Context Protocol)\n- The Apify MCP server at `https://mcp.apify.com`\n- Basic familiarity with web scraping concepts\n\nI run site-intent as a private Actor in my Apify account. My Actor ID is `8YtTAk2XgTf3NL1yy`, but that only works on my account. Nobody else can call it.\n\nThe code is public on GitHub. To reproduce this Actor yourself, clone the [site-intent repo](https://github.com/Manojaditya64/site-intent), then deploy it to your own Apify account:\n\n```\nnpx apify-cli login\nnpx apify-cli push\n```\n\nApify assigns your Actor ID on first push. Pin or call that ID from MCP, not my ID. The MCP workflow in this article is the same; only the Actor ID changes.\n\n## The research step that slowed me down\n\nMy pipeline starts with a spreadsheet of domains, often filtered from a larger firmographic list (company size, industry, location, and similar fields).\n\nBefore I pay for contact enrichment (Prospeo, for example) or email verification, I want answers to simpler questions:\n\n- Does this company have a blog? When did it last publish?\n- Is pricing visible on the site?\n- Does the copy mention AEO (answer-engine optimization) or similar topics I care about?\n- Does the site look like a SaaS product company or a services agency?\n\nI used to answer those by opening each domain in a browser. On a list of 800 names, that alone could eat a week.\n\nWhat I wanted instead was a machine-readable record per domain: a single JSON object my scripts or AI agent could read and filter on before I spent money on enrichment.\n\n## What site-intent does\n\nOn Apify, an Actor is a small cloud app that takes JSON input, does work (often scraping), and writes results to a dataset (a table of JSON rows).\n\nsite-intent is built with [Crawlee](https://crawlee.dev) and Playwright. For each domain you pass in, the Actor:\n\n1. Opens the company homepage (and the `www` variant if needed)\n2. Optionally reads `sitemap.xml` to find blog-related URLs\n3. Follows internal links to locate pricing, blog, careers, and similar pages\n4. Samples a few blog posts or an RSS feed to estimate how recently the site published\n5. Combines everything into one output row for that company\n\nThe important design choice is the last step. Many scrapers return one row per page. site-intent returns one row per company, with fields like `has_blog`, `pricing_page_exists`, and `blog_cadence_band` already filled in. That is what makes it usable for lead qualification. In outbound, I do not need every paragraph from every page. A page dump forces me or an LLM to re-parse HTML for each domain. One row with `qualify_ready`, `pre_segment`, and `blog_cadence_band` lets me filter hundreds of names before I pay for contact enrichment or write a personalized email.\n\n### What it does not do\n\n- It does not call an LLM inside the Actor (that keeps runs cheap and repeatable)\n- It does not find email addresses\n- It does not crawl the entire website\n- It does not fetch revenue or employee counts (I use a separate enrichment step for that)\n\nIf I need an AI-written summary later, I run that in Python after the dataset comes back. The Actor only handles structured scraping.\n\n### How it decides a company is worth a closer look\n\nThe main output flag is `qualify_ready`. It is computed with simple rules, not an AI model:\n\n``` js\nconst qualify_ready =\n    !!home &&\n    (home.word_count || 0) > 80 &&\n    (pricing_page_exists || blog_index_fetched || has_blog);\n```\n\nIn plain terms: the homepage must have more than 80 words, and the site must have a pricing page or a blog. If `qualify_ready` is false, I skip paid enrichment for that domain.\n\n1. AEO / AI-search signal detection\n\n``` js\nconst AEO_RE =\n    /ai search|aeo\\b|answer engine|chatgpt|claude|perplexity|gemini|ai discovery|ai visibility|appear in chatgpt/i;\n// Evaluated against homepage H1, meta description, content, and pricing page text:\naeo_copy_present: AEO_RE.test(mergedForAeo)\n```\n\n2. Pre-segment routing\n\n```\nfunction inferPreSegment(intent: Partial<CompanyIntent>): string | null {\n    if (intent.aeo_copy_present && intent.pricing_page_exists && intent.business_model === 'saas') {\n        return 'aeo_curious';\n    }\n    if (intent.business_model === 'agency' || intent.is_agency_signals) return 'agency';\n    if (intent.aeo_copy_present) return 'aeo_curious';\n    if (!intent.has_blog) return 'no_blog';\n    if (intent.blog_cadence_band === 'stall') return 'blog_stall';\n    if (intent.blog_cadence_band === 'active') return 'blog_active';\n    return 'unknown';\n}\n```\n\n3. Qualification signal tags\n\n```\nfunction buildQualificationSignals(intent: Partial<CompanyIntent>): string[] {\n    const signals: string[] = [];\n    if (intent.is_agency_signals) signals.push('agency');\n    if (intent.aeo_copy_present) signals.push('aeo_copy');\n    if (intent.blog_cadence_band === 'stall') signals.push('blog_stall');\n    if (intent.hiring_content_keywords) signals.push('hiring_content');\n    if (!intent.pricing_page_exists && intent.business_model !== 'agency') {\n        signals.push('no_public_pricing');\n    }\n    return signals;\n}\n```\n\nOther fields I use for routing:\n\n| Field | What it tells you | \n|---|---|\n| `scrape_status` | Whether the scrape succeeded ( `ok` ,`partial` , or`failed` ) | \n| `scrape_coverage` | How much of the site the Actor actually reached ( `high` ,`medium` ,`low` ) | \n| `business_model` | Best guess: `saas` ,`agency` ,`services` , or`unknown` | \n| `blog_cadence_band` | Blog activity: `active` ,`stall` , or`unknown` (active ≈ post within ~45 days) | \n| `aeo_copy_present` | Whether the site mentions answer-engine / AI-search style copy | \n| `pre_segment` | A short label like `aeo_curious` or`blog_stall` for outbound routing | \n\n### How pages get fetched (without drowning in cost)\n\nMarketing sites fall into two buckets: mostly static HTML, or heavy JavaScript that needs a real browser.\n\nsite-intent supports three modes:\n\n| Mode | What it does | When I use it | \n|---|---|---|\n| `fast` | Lightweight HTML parser only | Simple static sites | \n| `auto` (default) | Tries the fast parser first; opens a browser only when a page looks dynamic | Most production runs | \n| `headless` | Real browser for every page | Sites that break the fast path | \n\nIn `auto` mode, most pages never spin up Playwright. That kept my Apify bill low across large batches: the four-domain run later in this article cost about three cents.\n\nThe Actor also caps how many URLs it will fetch per domain (default 16, including homepage, sitemap, pricing, blog index, and a few posts). That prevents a single huge site from burning the whole run budget.\n\n### Where it sits in my outbound flow\n\n```\nDomain spreadsheet\n  → Apify: site-intent\n  → Filter on qualify_ready and pre_segment\n  → LLM qualification on survivors (optional)\n  → Paid contact enrichment on winners only\n```\n\nApify handles scraping. Python scripts orchestrate batches. The Actor itself never holds API keys for AI services.\n\n## Designing the input so an agent sends the right JSON\n\nThe first time I let an AI agent call site-intent, it sent `start_url`. That field belongs to a different scrape mode, so the run failed.\n\nA separate production bug showed up in batch runs. I had hardcoded the Playwright handoff queue as `dynamic-queue`. On Apify, named request queues persist across runs, so stale handled counts from an old run could break URL limits on the next one. I switched to a per-run queue name using `APIFY_ACTOR_RUN_ID` (see [src/dynamic-queue.ts](https://github.com/Manojaditya64/site-intent/blob/main/src/dynamic-queue.ts) in the repo). That was the kind of bug you only see after real batch jobs, not in a five-domain smoke test.\n\nThe fields that matter for qualification runs are below (the full definition is in `.actor/input_schema.json` in the site-intent GitHub repo).\n\n**Input fields for qualification (`crawl_profile: \"intent\"`):**\n\n| Field | Type | Default | Purpose | \n|---|---|---|---|\n| `crawl_profile` | `\"intent\"` or`\"audit\"` | `\"intent\"` | Use `\"intent\"` for one row per domain. Ignore`\"audit\"` unless debugging. | \n| `domains` | string array | — | Company hostnames, e.g. `[\"acme.com\"]` | \n| `start_urls` | URL array | — | Extra seed URLs merged with each domain | \n| `mode` | `\"auto\"` ,`\"fast\"` ,`\"headless\"` | `\"auto\"` | How pages are fetched | \n| `max_urls_per_domain` | integer | `16` | Max HTTP fetches per company | \n| `max_post_pages` | integer | `4` | Blog articles to sample | \n| `discover_from_home` | boolean | `true` | Find pricing/blog links from homepage | \n| `parse_rss` | boolean | `true` | Read RSS/Atom for blog dates | \n| `fetch_sitemap` | boolean | `true` | Seed `sitemap.xml` | \n| `include_careers` | boolean | `true` | Try careers/jobs pages | \n\n**Audit-only fields (do not send for MCP qualification):** `start_url`, `max_pages`, `same_domain_only`.\n\nIf you are calling your own Actor over MCP, use `fetch-actor-details` with your Actor ID while logged into the owning account. That returns the live input schema without needing a public Store link.\n\nI tightened the input schema around three ideas:\n\n### 1. Pass a list of domains, not full URLs\n\nThe main input is `domains`: an array of hostnames like `growsurf.com`, not `https://www.growsurf.com/pricing`.\n\n```\n{\n  \"crawl_profile\": \"intent\",\n  \"domains\": [\"growsurf.com\", \"surferseo.com\"]\n}\n```\n\nSet `crawl_profile` to `\"intent\"` for lead qualification.\n\n### 2. Keep required fields small; let defaults do the rest\n\nMost runs only need `crawl_profile`, `domains`, and optionally `mode`.\n\nOptional flags like `discover_from_home`, `parse_rss`, and `fetch_sitemap` default to `true`, so an agent does not have to guess them.\n\n### 3. Mark legacy fields clearly\n\nFields like `start_url` and `max_pages` are labeled `(audit only)` in the schema so agents skip them during qualification runs.\n\n### Minimal input for MCP (what I used in Cursor)\n\n```\n{\n  \"crawl_profile\": \"intent\",\n  \"domains\": [\"growsurf.com\", \"surferseo.com\", \"beehiiv.com\", \"typedream.com\"],\n  \"mode\": \"auto\"\n}\n```\n\n### Full input when I want explicit control\n\n```\n{\n  \"crawl_profile\": \"intent\",\n  \"domains\": [\"example.com\"],\n  \"mode\": \"auto\",\n  \"max_urls_per_domain\": 16,\n  \"max_post_pages\": 4,\n  \"discover_from_home\": true,\n  \"parse_rss\": true,\n  \"fetch_sitemap\": true,\n  \"include_careers\": true\n}\n```\n\n### Common mistakes\n\n- Using a short Actor name like `site-intent` in MCP lookups. Call it by your own Actor ID instead.\n- Sending `start_url` when you mean a list of companies in`domains` .\n- Putting too many domains in one run. My batch script uses 20 per run by default to stay within Apify memory limits.\n\n## Wiring site-intent into Cursor with the Apify MCP server\n\nMCP lets an AI client call external tools. The Apify MCP server exposes Apify Actors as those tools, so Cursor can start a scrape, wait for results, and read the dataset without you leaving the chat.\n\nThis is different from [MCP connectors](https://docs.apify.com/integrations/mcp-connectors), which let an Actor call Notion or Slack during a run. Here, the flow is the other way around: my agent calls my Actor.\n\n### Step 1: Connect MCP in Cursor\n\nIn Cursor settings, add:\n\n```\n{\n  \"mcpServers\": {\n    \"apify\": {\n      \"url\": \"https://mcp.apify.com\"\n    }\n  }\n}\n```\n\nSign in with OAuth, or pass your Apify token in the `Authorization` header if you prefer. For a step-by-step walkthrough, see [how to add an MCP server to Cursor](https://blog.apify.com/how-to-add-mcp-server-to-cursor/).\n\nI also pinned *my* Actor in the [MCP configuration UI](https://mcp.apify.com) so it shows up in my tool list.\n\n### Step 2: Ask the agent to run the Actor\n\nFrom chat, I asked Cursor to qualify four SaaS domains I would normally have researched by hand. The agent called `call-actor` with the JSON input above.\n\n### Step 3: Read the results\n\nWhen the run finished, Apify wrote four rows to a dataset. These are the fields that mattered for my decision:\n\n| Domain | `qualify_ready` | `pre_segment` | blog cadence | pricing | AEO copy | \n|---|---|---|---|---|---|\n| growsurf.com | true | unknown | unknown | yes | no | \n| surferseo.com | true | aeo_curious | active | yes | yes | \n| beehiiv.com | true | blog_stall | stall | no | no | \n| typedream.com | true | unknown | unknown | no | no | \n\nThe pricing column shows whether the Actor found a dedicated pricing page (`pricing_page_exists`). `qualify_ready` does not require one. A domain can be `qualify_ready: true` with no pricing page if the homepage has enough content and the Actor found a blog index or other blog signal. That is what happened for typedream.com in this run.\n\nApify Console confirmed the run came from MCP (not a manual button click):\n\n| Field | Value | \n|---|---|\n| Run ID | `cR6up7A0PYmwwzeeg` | \n| Origin | MCP | \n| Duration | 1m 47s | \n| Cost | ~$0.031 | \n| Status | SUCCEEDED | \n\nWithout opening a browser, the agent could tell me that Surfer SEO looks like a strong fit (active blog, public pricing, AEO-related copy). Beehiiv shows a stalled blog and no public pricing page, so I might skip paid enrichment there.\n\n### How I prompt the agent after results arrive\n\nI give Cursor a short rubric:\n\n- If `qualify_ready` is false, skip enrichment.\n- If `pre_segment` is`aeo_curious` and the blog looks`active` , prioritize for my AEO outbound angle.\n- If `blog_stall` or`no_public_pricing` appears in`qualification_signals` , deprioritize or use a different sequence.\n\nThe agent reads structured fields. It does not need to parse raw HTML.\n\n### MCP tools I used\n\n1. `fetch-actor-details` with the Actor ID to read the input schema first (works because MCP uses my Apify account)\n2. `call-actor` with the JSON body to start the scrape\n3. `get-dataset-items` on the returned dataset ID to pull`qualify_ready` ,`pre_segment` , and related fields\n\n## Cost and time: manual research vs one tool call\n\nResearching four companies by hand takes me about 15 to 20 minutes if I click through each site properly.\n\nThe MCP run on the same four domains took 107 seconds and cost ~$0.031.\n\nFor large lists I still use a Python batch script (`scrape_icp_a_site_intent.py`) that calls Apify directly. MCP is what I use when I am mid-conversation in Cursor and want a quick answer on a short list of domains.\n\n| Approach | Best for | \n|---|---|\n| MCP from Cursor | Quick checks during an agent chat | \n| Python + Apify API | Hundreds of domains in scheduled batches | \n| Manual browser tabs | One-off curiosity only | \n\n## Conclusion\n\nI replaced a slow manual research step with an Apify Actor that returns one structured row per company. Then I exposed it to Cursor through the Apify MCP server. Four domains, under two minutes, about three cents, and my agent could tell me which names were worth enriching without me opening a single tab.\n\nIf you are building similar workflows, keep the Actor input simple, keep the output fields named for decisions you actually make, and only add LLM steps after the scrape returns structured data.\n\n**Next steps:**\n\n1. Connect the [Apify MCP server](https://mcp.apify.com) in your client\n2. Build or publish your own Actor, or use a Store Actor you trust\n3. Run five domains and read `qualify_ready` before you pay for enrichment\n\n## FAQ\n\n### Can I use a private Actor over MCP?\n\nYes. site-intent is private in my Apify account. MCP works as long as you sign in with the same account that owns the Actor. Pin or call it by *your* Actor ID after you push the repo. Other users cannot run it unless you share access or publish the Actor.\n\nTo read the input schema without a public Store page, use `fetch-actor-details` from MCP or open the Actor in Apify Console on your own account. Readers who clone the repo get their own private Actor in their own account. My ID in Prerequisites is only there as proof of my demo run.\n\n### Why is qualify_ready not an LLM score?\n\nI run this Actor across hundreds of domains. A simple rule-based flag is fast, cheap, and gives the same answer every time. I save LLM calls for the smaller set of companies that pass the scrape gate.\n\n### Is scraping these sites allowed?\n\nI only fetch public marketing pages at low volume. Respect each site's terms of service and use this for qualification research, not republishing their content.\n\n### Apify MCP server vs MCP connectors: what is the difference?\n\n- **MCP server (this article):** your AI agent calls Apify Actors as tools from outside (e.g. from Cursor).\n- **MCP connectors:** an Actor calls external services like Notion or Slack during its own run.\n\nThey solve different problems.\n\n### What if the agent sends the wrong input?\n\nStart with the minimal JSON in this article (`crawl_profile: \"intent\"` plus a `domains` array). If the run fails, open the input in Apify Console and check whether the agent used audit-only fields like `start_url` by mistake.", "url": "https://wpnews.pro/news/how-i-replaced-manual-company-research-with-an-apify-actor-my-cursor-agent-can", "canonical_source": "https://blog.apify.com/automate-company-research-with-cursor-and-apify/", "published_at": "2026-11-18 00:00:00+00:00", "updated_at": "2026-11-18 11:43:04.106552+00:00", "lang": "en", "topics": ["ai-agents", "agent-protocols", "ai-tools", "developer-tools"], "entities": ["Manojaditya Nadar", "Apify", "site-intent", "Cursor", "Apify MCP server", "Crawlee", "Playwright", "Prospeo"], "also_reported_by": [], "takeaway": "Return one structured row per company domain (has_blog, pricing_page_exists, blog_cadence_band) instead of one row per page, so an agent can filter hundreds of leads without re-parsing HTML for each domain.", "recommendation": {"advice": "Design Actor inputs so agents cannot send malformed JSON, and expose the Actor through the Apify MCP server so a Cursor agent can call it as a tool; Nadar's four-domain run cost about three cents.", "playbook": "agent-architecture", "playbook_url": "https://wpnews.pro/playbooks/agent-architecture"}, "alternates": {"html": "https://wpnews.pro/news/how-i-replaced-manual-company-research-with-an-apify-actor-my-cursor-agent-can", "markdown": "https://wpnews.pro/news/how-i-replaced-manual-company-research-with-an-apify-actor-my-cursor-agent-can.md", "text": "https://wpnews.pro/news/how-i-replaced-manual-company-research-with-an-apify-actor-my-cursor-agent-can.txt", "jsonld": "https://wpnews.pro/news/how-i-replaced-manual-company-research-with-an-apify-actor-my-cursor-agent-can.jsonld"}}