Give your LLM live web search: query top results clean page Markdown (no search API key)

A developer published a method for giving LLMs live web search without a paid search API key, using the Apify actor get_anything/ai-web-search-scraper to run a DuckDuckGo query, fetch the top results, and return each page's body as cleaned Markdown with rank, title, URL, domain and a token estimate. The approach can be called from Python via the Apify client or attached to Claude, ChatGPT or Cursor as a tool through the Apify MCP server, with per-result pricing and no subscription.

LLMs are stuck at their training cutoff. The usual fix is to give them a web search step : run a query, take the top results, read the pages, and put the relevant text in the prompt. Most tutorials stop at "get a search API key", which means a paid account with Google, Bing or Brave and their rate limits. Here's a way to do the whole loop search → fetch pages → clean Markdown in one call, with no search API key. Query: best open source vector database 2026 , top 5 results, with page content. This is the real output Markdown trimmed : { "query": "best open source vector database 2026", "engine": "duckduckgo", "position": 1, "title": "9 Best Open Source Vector Databases in 2026 Tested - Geekflare", "url": "https://geekflare.com/open-source/vector-database/", "domain": "geekflare.com", "contentSuccess": true, "tokensApprox": 200, "markdown": "| Qdrant | Yes — sparse and dense vectors fused in one query | ..." }, { "position": 5, "title": "7 Best Open-Source Vector Databases in 2026", "domain": "krunalkanojiya.com", "markdown": "Qdrant is the best balanced open-source vector database for most teams starting a dedicated retrieval service in 2026. It combines ..." } Every result has its rank, title, URL and domain . With fetchPageContent on, it also has the page body as Markdown with boilerplate stripped, plus a token estimate. That's exactly what you want to paste into a prompt with citations. python from apify client import ApifyClient client = ApifyClient "YOUR APIFY TOKEN" def web search query, n=5, chars=3000 : run = client.actor "get anything/ai-web-search-scraper" .call run input={ "queries": query , "engine": "duckduckgo", "maxResults": n, "fetchPageContent": True, "maxCharsPerPage": chars, keep the prompt within budget } return r for r in client.dataset run "defaultDatasetId" .iterate items if r.get "contentSuccess" def build context results : return "\n\n".join f" {r 'position' } {r 'title' } {r 'url' } \n{r 'markdown' }" for r in results results = web search "best open source vector database 2026" context = build context results prompt = f"""Answer using only the sources below and cite them like 1 . {context} Question: Which open-source vector database should a small team start with in 2026, and why?""" send prompt to the model you use Claude, GPT, a local Llama... A few tips from using it: maxCharsPerPage . position lets the model cite 1 – 5 , and you can show the links in your UI. queries takes a list, so you can fan out a question into 3–4 phrasings in one run and de-duplicate by URL. If you use Claude, ChatGPT, Cursor or another MCP client, add the scraper as a tool through the Apify MCP server: { "mcpServers": { "apify": { "url": "https://mcp.apify.com/?tools=get anything/ai-web-search-scraper" } } } Now you can ask "Search the web for this week's news on X and summarise the top 5 sources with links" , and the assistant runs the search and reads the pages itself. Setup guide: Connect any Apify scraper to Claude or ChatGPT in 2 minutes https://dev.to/get anything/connect-any-apify-scraper-to-claude-or-chatgpt-in-2-minutes-mcp-37he . If you don't need the search step say you have a list of docs pages or articles , use the sister tool that only converts URLs to clean Markdown and can crawl a whole site: LLM Web Scraper https://apify.com/get anything/llm-web-scraper . Pay per result, no subscription. See the Pricing tab on the Actor page. The Apify free plan's monthly credit is enough to try it properly. Try it: AI Web Search Scraper on Apify https://apify.com/get anything/ai-web-search-scraper What are you using for grounding today, and where does it break? I'm curious which sites give people the most trouble.