LLMs are stuck at their training cutoff. The usual fix is to give them a web search step: run a query, take the top results, read the pages, and put the relevant text in the prompt. Most tutorials stop at "get a search API key", which means a paid account with Google, Bing or Brave and their rate limits.
Here's a way to do the whole loop (search → fetch pages → clean Markdown) in one call, with no search API key.
Query: best open source vector database 2026, top 5 results, with page content. This is the real output (Markdown trimmed):
[
{
"query": "best open source vector database 2026",
"engine": "duckduckgo",
"position": 1,
"title": "9 Best Open Source Vector Databases in 2026 (Tested) - Geekflare",
"url": "https://geekflare.com/open-source/vector-database/",
"domain": "geekflare.com",
"contentSuccess": true,
"tokensApprox": 200,
"markdown": "| Qdrant | Yes — sparse and dense vectors fused in one query | ..."
},
{
"position": 5,
"title": "7 Best Open-Source Vector Databases in 2026",
"domain": "krunalkanojiya.com",
"markdown": "Qdrant is the best balanced open-source vector database for most teams starting a dedicated retrieval service in 2026. It combines ..."
}
]
Every result has its rank, title, URL and domain. With fetchPageContent on, it also has the page body as Markdown with boilerplate stripped, plus a token estimate. That's exactly what you want to paste into a prompt with citations.
from apify_client import ApifyClient
client = ApifyClient("YOUR_APIFY_TOKEN")
def web_search(query, n=5, chars=3000):
run = client.actor("get_anything/ai-web-search-scraper").call(run_input={
"queries": [query],
"engine": "duckduckgo",
"maxResults": n,
"fetchPageContent": True,
"maxCharsPerPage": chars, # keep the prompt within budget
})
return [r for r in client.dataset(run["defaultDatasetId"]).iterate_items() if r.get("contentSuccess")]
def build_context(results):
return "\n\n".join(
f"[{r['position']}] {r['title']} ({r['url']})\n{r['markdown']}" for r in results
)
results = web_search("best open source vector database 2026")
context = build_context(results)
prompt = f"""Answer using only the sources below and cite them like [1].
{context}
Question: Which open-source vector database should a small team start with in 2026, and why?"""
A few tips from using it:
maxCharsPerPage. position lets the model cite [1]–[5], and you can show the links in your UI. queries takes a list, so you can fan out a question into 3–4 phrasings in one run and de-duplicate by URL.
If you use Claude, ChatGPT, Cursor or another MCP client, add the scraper as a tool through the Apify MCP server:
{ "mcpServers": { "apify": { "url": "https://mcp.apify.com/?tools=get_anything/ai-web-search-scraper" } } }
Now you can ask "Search the web for this week's news on X and summarise the top 5 sources with links", and the assistant runs the search and reads the pages itself. Setup guide: Connect any Apify scraper to Claude or ChatGPT in 2 minutes.
If you don't need the search step (say you have a list of docs pages or articles), use the sister tool that only converts URLs to clean Markdown and can crawl a whole site: LLM Web Scraper.
Pay per result, no subscription. See the Pricing tab on the Actor page. The Apify free plan's monthly credit is enough to try it properly.
Try it: AI Web Search Scraper on Apify
What are you using for grounding today, and where does it break? I'm curious which sites give people the most trouble.