cd /news/ai-agents/how-i-replaced-manual-company-resear… · home › topics › ai-agents › article
[ARTICLE · art-171744] src=blog.apify.com ↗ pub= topic=ai-agents verified=true sentiment=↑ positive

How I replaced manual company research with an Apify Actor my Cursor agent can call

Developer Manojaditya Nadar built an Apify Actor called site-intent that visits a fixed set of pages per company domain and returns one structured JSON row per company, then connected it to the Cursor coding agent through the Apify MCP server at https://mcp.apify.com so the agent can call it as a tool. The Actor, built with Crawlee and Playwright, opens each homepage, optionally reads sitemap.xml, follows internal links to pricing, blog and careers pages, and samples posts or an RSS feed to fill fields including has_blog, pricing_page_exists, blog_cadence_band, qualify_ready and pre_segment. Nadar reports running the Actor on four real domains from Cursor for about three cents, replacing manual research that he said could take a week on a list of 800 names.

recommendation

Design Actor inputs so agents cannot send malformed JSON, and expose the Actor through the Apify MCP server so a Cursor agent can call it as a tool; Nadar's four-domain run cost about three cents.

more in playbook: Designing agent systems
by read14 min views2 publishedNov 18, 2026
How I replaced manual company research with an Apify Actor my Cursor agent can call
Image: Blog (auto-discovered)

Manojaditya Nadaras part of

Write for Apify- a program for developers sharing original articles about what they've built with Apify.

I run outbound for B2B SaaS. My lead lists tell me a company exists: industry, headcount band, headquarters. They do not tell me whether the blog is still active, pricing is public, or the homepage sounds like a product company or an agency.

For months I did that research by hand. I would open each site, click through to the blog and pricing pages, and try to remember what I saw. Five companies were manageable. Eight hundred were not.

I built an Apify Actor called site-intent to automate that step. It visits a small, fixed set of pages per company and returns one structured summary row per domain. Then I connected it to Cursor through the Apify MCP server so my AI agent can run it like any other tool during a conversation.

This article explains what the Actor does, how I designed its inputs so agents do not send the wrong JSON, and what happened when I ran it on four real domains from Cursor for about three cents.

Prerequisites #

You will need:

  • An Apify account and API token
  • Cursor or another client that supports MCP (Model Context Protocol)
  • The Apify MCP server at https://mcp.apify.com
  • Basic familiarity with web scraping concepts

I run site-intent as a private Actor in my Apify account. My Actor ID is 8YtTAk2XgTf3NL1yy, but that only works on my account. Nobody else can call it.

The code is public on GitHub. To reproduce this Actor yourself, clone the site-intent repo, then deploy it to your own Apify account:

npx apify-cli login
npx apify-cli push

Apify assigns your Actor ID on first push. Pin or call that ID from MCP, not my ID. The MCP workflow in this article is the same; only the Actor ID changes.

The research step that slowed me down #

My pipeline starts with a spreadsheet of domains, often filtered from a larger firmographic list (company size, industry, location, and similar fields).

Before I pay for contact enrichment (Prospeo, for example) or email verification, I want answers to simpler questions:

  • Does this company have a blog? When did it last publish?
  • Is pricing visible on the site?
  • Does the copy mention AEO (answer-engine optimization) or similar topics I care about?
  • Does the site look like a SaaS product company or a services agency?

I used to answer those by opening each domain in a browser. On a list of 800 names, that alone could eat a week.

What I wanted instead was a machine-readable record per domain: a single JSON object my scripts or AI agent could read and filter on before I spent money on enrichment.

What site-intent does #

On Apify, an Actor is a small cloud app that takes JSON input, does work (often scraping), and writes results to a dataset (a table of JSON rows).

site-intent is built with Crawlee and Playwright. For each domain you pass in, the Actor:

  1. Opens the company homepage (and the www variant if needed)
  2. Optionally reads sitemap.xml to find blog-related URLs
  3. Follows internal links to locate pricing, blog, careers, and similar pages
  4. Samples a few blog posts or an RSS feed to estimate how recently the site published
  5. Combines everything into one output row for that company

The important design choice is the last step. Many scrapers return one row per page. site-intent returns one row per company, with fields like has_blog, pricing_page_exists, and blog_cadence_band already filled in. That is what makes it usable for lead qualification. In outbound, I do not need every paragraph from every page. A page dump forces me or an LLM to re-parse HTML for each domain. One row with qualify_ready, pre_segment, and blog_cadence_band lets me filter hundreds of names before I pay for contact enrichment or write a personalized email.

What it does not do

  • It does not call an LLM inside the Actor (that keeps runs cheap and repeatable)
  • It does not find email addresses
  • It does not crawl the entire website
  • It does not fetch revenue or employee counts (I use a separate enrichment step for that)

If I need an AI-written summary later, I run that in Python after the dataset comes back. The Actor only handles structured scraping.

How it decides a company is worth a closer look

The main output flag is qualify_ready. It is computed with simple rules, not an AI model:

const qualify_ready =
    !!home &&
    (home.word_count || 0) > 80 &&
    (pricing_page_exists || blog_index_fetched || has_blog);

In plain terms: the homepage must have more than 80 words, and the site must have a pricing page or a blog. If qualify_ready is false, I skip paid enrichment for that domain.

  1. AEO / AI-search signal detection
const AEO_RE =
    /ai search|aeo\b|answer engine|chatgpt|claude|perplexity|gemini|ai discovery|ai visibility|appear in chatgpt/i;
// Evaluated against homepage H1, meta description, content, and pricing page text:
aeo_copy_present: AEO_RE.test(mergedForAeo)
  1. Pre-segment routing
function inferPreSegment(intent: Partial<CompanyIntent>): string | null {
    if (intent.aeo_copy_present && intent.pricing_page_exists && intent.business_model === 'saas') {
        return 'aeo_curious';
    }
    if (intent.business_model === 'agency' || intent.is_agency_signals) return 'agency';
    if (intent.aeo_copy_present) return 'aeo_curious';
    if (!intent.has_blog) return 'no_blog';
    if (intent.blog_cadence_band === 'stall') return 'blog_stall';
    if (intent.blog_cadence_band === 'active') return 'blog_active';
    return 'unknown';
}
  1. Qualification signal tags
function buildQualificationSignals(intent: Partial<CompanyIntent>): string[] {
    const signals: string[] = [];
    if (intent.is_agency_signals) signals.push('agency');
    if (intent.aeo_copy_present) signals.push('aeo_copy');
    if (intent.blog_cadence_band === 'stall') signals.push('blog_stall');
    if (intent.hiring_content_keywords) signals.push('hiring_content');
    if (!intent.pricing_page_exists && intent.business_model !== 'agency') {
        signals.push('no_public_pricing');
    }
    return signals;
}

Other fields I use for routing:

Field What it tells you
scrape_status Whether the scrape succeeded ( ok ,partial , orfailed )
scrape_coverage How much of the site the Actor actually reached ( high ,medium ,low )
business_model Best guess: saas ,agency ,services , orunknown
blog_cadence_band Blog activity: active ,stall , orunknown (active ≈ post within ~45 days)
aeo_copy_present Whether the site mentions answer-engine / AI-search style copy
pre_segment A short label like aeo_curious orblog_stall for outbound routing

How pages get fetched (without drowning in cost)

Marketing sites fall into two buckets: mostly static HTML, or heavy JavaScript that needs a real browser.

site-intent supports three modes:

Mode What it does When I use it
fast Lightweight HTML parser only Simple static sites
auto (default) Tries the fast parser first; opens a browser only when a page looks dynamic Most production runs
headless Real browser for every page Sites that break the fast path

In auto mode, most pages never spin up Playwright. That kept my Apify bill low across large batches: the four-domain run later in this article cost about three cents.

The Actor also caps how many URLs it will fetch per domain (default 16, including homepage, sitemap, pricing, blog index, and a few posts). That prevents a single huge site from burning the whole run budget.

Where it sits in my outbound flow

Domain spreadsheet
  → Apify: site-intent
  → Filter on qualify_ready and pre_segment
  → LLM qualification on survivors (optional)
  → Paid contact enrichment on winners only

Apify handles scraping. Python scripts orchestrate batches. The Actor itself never holds API keys for AI services.

Designing the input so an agent sends the right JSON #

The first time I let an AI agent call site-intent, it sent start_url. That field belongs to a different scrape mode, so the run failed.

A separate production bug showed up in batch runs. I had hardcoded the Playwright handoff queue as dynamic-queue. On Apify, named request queues persist across runs, so stale handled counts from an old run could break URL limits on the next one. I switched to a per-run queue name using APIFY_ACTOR_RUN_ID (see src/dynamic-queue.ts in the repo). That was the kind of bug you only see after real batch jobs, not in a five-domain smoke test.

The fields that matter for qualification runs are below (the full definition is in .actor/input_schema.json in the site-intent GitHub repo).

Input fields for qualification (crawl_profile: "intent"):

Field Type Default Purpose
crawl_profile "intent" or"audit" "intent" Use "intent" for one row per domain. Ignore"audit" unless debugging.
domains string array — Company hostnames, e.g. ["acme.com"]
start_urls URL array — Extra seed URLs merged with each domain
mode "auto" ,"fast" ,"headless" "auto" How pages are fetched
max_urls_per_domain integer 16 Max HTTP fetches per company
max_post_pages integer 4 Blog articles to sample
discover_from_home boolean true Find pricing/blog links from homepage
parse_rss boolean true Read RSS/Atom for blog dates
fetch_sitemap boolean true Seed sitemap.xml
include_careers boolean true Try careers/jobs pages

Audit-only fields (do not send for MCP qualification): start_url, max_pages, same_domain_only.

If you are calling your own Actor over MCP, use fetch-actor-details with your Actor ID while logged into the owning account. That returns the live input schema without needing a public Store link.

I tightened the input schema around three ideas:

1. Pass a list of domains, not full URLs

The main input is domains: an array of hostnames like growsurf.com, not https://www.growsurf.com/pricing.

{
  "crawl_profile": "intent",
  "domains": ["growsurf.com", "surferseo.com"]
}

Set crawl_profile to "intent" for lead qualification.

2. Keep required fields small; let defaults do the rest

Most runs only need crawl_profile, domains, and optionally mode.

Optional flags like discover_from_home, parse_rss, and fetch_sitemap default to true, so an agent does not have to guess them.

3. Mark legacy fields clearly

Fields like start_url and max_pages are labeled (audit only) in the schema so agents skip them during qualification runs.

Minimal input for MCP (what I used in Cursor)

{
  "crawl_profile": "intent",
  "domains": ["growsurf.com", "surferseo.com", "beehiiv.com", "typedream.com"],
  "mode": "auto"
}

Full input when I want explicit control

{
  "crawl_profile": "intent",
  "domains": ["example.com"],
  "mode": "auto",
  "max_urls_per_domain": 16,
  "max_post_pages": 4,
  "discover_from_home": true,
  "parse_rss": true,
  "fetch_sitemap": true,
  "include_careers": true
}

Common mistakes

  • Using a short Actor name like site-intent in MCP lookups. Call it by your own Actor ID instead.
  • Sending start_url when you mean a list of companies indomains .
  • Putting too many domains in one run. My batch script uses 20 per run by default to stay within Apify memory limits.

Wiring site-intent into Cursor with the Apify MCP server #

MCP lets an AI client call external tools. The Apify MCP server exposes Apify Actors as those tools, so Cursor can start a scrape, wait for results, and read the dataset without you leaving the chat.

This is different from MCP connectors, which let an Actor call Notion or Slack during a run. Here, the flow is the other way around: my agent calls my Actor.

Step 1: Connect MCP in Cursor

In Cursor settings, add:

{
  "mcpServers": {
    "apify": {
      "url": "https://mcp.apify.com"
    }
  }
}

Sign in with OAuth, or pass your Apify token in the Authorization header if you prefer. For a step-by-step walkthrough, see how to add an MCP server to Cursor.

I also pinned my Actor in the MCP configuration UI so it shows up in my tool list.

Step 2: Ask the agent to run the Actor

From chat, I asked Cursor to qualify four SaaS domains I would normally have researched by hand. The agent called call-actor with the JSON input above.

Step 3: Read the results

When the run finished, Apify wrote four rows to a dataset. These are the fields that mattered for my decision:

Domain qualify_ready pre_segment blog cadence pricing AEO copy
growsurf.com true unknown unknown yes no
surferseo.com true aeo_curious active yes yes
beehiiv.com true blog_stall stall no no
typedream.com true unknown unknown no no

The pricing column shows whether the Actor found a dedicated pricing page (pricing_page_exists). qualify_ready does not require one. A domain can be qualify_ready: true with no pricing page if the homepage has enough content and the Actor found a blog index or other blog signal. That is what happened for typedream.com in this run.

Apify Console confirmed the run came from MCP (not a manual button click):

Field Value
Run ID cR6up7A0PYmwwzeeg
Origin MCP
Duration 1m 47s
Cost ~$0.031
Status SUCCEEDED

Without opening a browser, the agent could tell me that Surfer SEO looks like a strong fit (active blog, public pricing, AEO-related copy). Beehiiv shows a stalled blog and no public pricing page, so I might skip paid enrichment there.

How I prompt the agent after results arrive

I give Cursor a short rubric:

  • If qualify_ready is false, skip enrichment.
  • If pre_segment isaeo_curious and the blog looksactive , prioritize for my AEO outbound angle.
  • If blog_stall orno_public_pricing appears inqualification_signals , deprioritize or use a different sequence.

The agent reads structured fields. It does not need to parse raw HTML.

MCP tools I used

  1. fetch-actor-details with the Actor ID to read the input schema first (works because MCP uses my Apify account)
  2. call-actor with the JSON body to start the scrape
  3. get-dataset-items on the returned dataset ID to pullqualify_ready ,pre_segment , and related fields

Cost and time: manual research vs one tool call #

Researching four companies by hand takes me about 15 to 20 minutes if I click through each site properly.

The MCP run on the same four domains took 107 seconds and cost ~$0.031.

For large lists I still use a Python batch script (scrape_icp_a_site_intent.py) that calls Apify directly. MCP is what I use when I am mid-conversation in Cursor and want a quick answer on a short list of domains.

Approach Best for
MCP from Cursor Quick checks during an agent chat
Python + Apify API Hundreds of domains in scheduled batches
Manual browser tabs One-off curiosity only

Conclusion #

I replaced a slow manual research step with an Apify Actor that returns one structured row per company. Then I exposed it to Cursor through the Apify MCP server. Four domains, under two minutes, about three cents, and my agent could tell me which names were worth enriching without me opening a single tab.

If you are building similar workflows, keep the Actor input simple, keep the output fields named for decisions you actually make, and only add LLM steps after the scrape returns structured data.

Next steps:

  1. Connect the Apify MCP server in your client
  2. Build or publish your own Actor, or use a Store Actor you trust
  3. Run five domains and read qualify_ready before you pay for enrichment

FAQ #

Can I use a private Actor over MCP?

Yes. site-intent is private in my Apify account. MCP works as long as you sign in with the same account that owns the Actor. Pin or call it by your Actor ID after you push the repo. Other users cannot run it unless you share access or publish the Actor.

To read the input schema without a public Store page, use fetch-actor-details from MCP or open the Actor in Apify Console on your own account. Readers who clone the repo get their own private Actor in their own account. My ID in Prerequisites is only there as proof of my demo run.

Why is qualify_ready not an LLM score?

I run this Actor across hundreds of domains. A simple rule-based flag is fast, cheap, and gives the same answer every time. I save LLM calls for the smaller set of companies that pass the scrape gate.

Is scraping these sites allowed?

I only fetch public marketing pages at low volume. Respect each site's terms of service and use this for qualification research, not republishing their content.

Apify MCP server vs MCP connectors: what is the difference?

  • MCP server (this article): your AI agent calls Apify Actors as tools from outside (e.g. from Cursor).
  • MCP connectors: an Actor calls external services like Notion or Slack during its own run.

They solve different problems.

What if the agent sends the wrong input?

Start with the minimal JSON in this article (crawl_profile: "intent" plus a domains array). If the run fails, open the input in Apify Console and check whether the agent used audit-only fields like start_url by mistake.

── more in #ai-agents 4 stories · sorted by recency
── more on @manojaditya nadar 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/how-i-replaced-manua…] indexed:0 read:14min 2026-11-18 · —