{"slug": "give-your-llm-live-web-search-query-top-results-clean-page-markdown-no-search", "title": "Give your LLM live web search: query top results clean page Markdown (no search API key)", "summary": "A developer published a method for giving LLMs live web search without a paid search API key, using the Apify actor get_anything/ai-web-search-scraper to run a DuckDuckGo query, fetch the top results, and return each page's body as cleaned Markdown with rank, title, URL, domain and a token estimate. The approach can be called from Python via the Apify client or attached to Claude, ChatGPT or Cursor as a tool through the Apify MCP server, with per-result pricing and no subscription.", "body_md": "LLMs are stuck at their training cutoff. The usual fix is to give them a **web search step**: run a query, take the top results, read the pages, and put the relevant text in the prompt. Most tutorials stop at \"get a search API key\", which means a paid account with Google, Bing or Brave and their rate limits.\n\nHere's a way to do the whole loop (**search → fetch pages → clean Markdown**) in one call, with no search API key.\n\nQuery: `best open source vector database 2026`, top 5 results, with page content. This is the real output (Markdown trimmed):\n\n```\n[\n  {\n    \"query\": \"best open source vector database 2026\",\n    \"engine\": \"duckduckgo\",\n    \"position\": 1,\n    \"title\": \"9 Best Open Source Vector Databases in 2026 (Tested) - Geekflare\",\n    \"url\": \"https://geekflare.com/open-source/vector-database/\",\n    \"domain\": \"geekflare.com\",\n    \"contentSuccess\": true,\n    \"tokensApprox\": 200,\n    \"markdown\": \"| Qdrant | Yes — sparse and dense vectors fused in one query | ...\"\n  },\n  {\n    \"position\": 5,\n    \"title\": \"7 Best Open-Source Vector Databases in 2026\",\n    \"domain\": \"krunalkanojiya.com\",\n    \"markdown\": \"Qdrant is the best balanced open-source vector database for most teams starting a dedicated retrieval service in 2026. It combines ...\"\n  }\n]\n```\n\nEvery result has its **rank, title, URL and domain**. With `fetchPageContent` on, it also has the **page body as Markdown** with boilerplate stripped, plus a token estimate. That's exactly what you want to paste into a prompt with citations.\n\n``` python\nfrom apify_client import ApifyClient\n\nclient = ApifyClient(\"YOUR_APIFY_TOKEN\")\n\ndef web_search(query, n=5, chars=3000):\n    run = client.actor(\"get_anything/ai-web-search-scraper\").call(run_input={\n        \"queries\": [query],\n        \"engine\": \"duckduckgo\",\n        \"maxResults\": n,\n        \"fetchPageContent\": True,\n        \"maxCharsPerPage\": chars,      # keep the prompt within budget\n    })\n    return [r for r in client.dataset(run[\"defaultDatasetId\"]).iterate_items() if r.get(\"contentSuccess\")]\n\ndef build_context(results):\n    return \"\\n\\n\".join(\n        f\"[{r['position']}] {r['title']} ({r['url']})\\n{r['markdown']}\" for r in results\n    )\n\nresults = web_search(\"best open source vector database 2026\")\ncontext = build_context(results)\n\nprompt = f\"\"\"Answer using only the sources below and cite them like [1].\n\n{context}\n\nQuestion: Which open-source vector database should a small team start with in 2026, and why?\"\"\"\n# send `prompt` to the model you use (Claude, GPT, a local Llama...)\n```\n\nA few tips from using it:\n\n`maxCharsPerPage`.` position` lets the model cite `[1]`–`[5]`, and you can show the links in your UI.` queries` takes a list, so you can fan out a question into 3–4 phrasings in one run and de-duplicate by URL.\nIf you use Claude, ChatGPT, Cursor or another MCP client, add the scraper as a tool through the Apify MCP server:\n\n```\n{ \"mcpServers\": { \"apify\": { \"url\": \"https://mcp.apify.com/?tools=get_anything/ai-web-search-scraper\" } } }\n```\n\nNow you can ask *\"Search the web for this week's news on X and summarise the top 5 sources with links\"*, and the assistant runs the search and reads the pages itself. Setup guide: [Connect any Apify scraper to Claude or ChatGPT in 2 minutes](https://dev.to/get_anything/connect-any-apify-scraper-to-claude-or-chatgpt-in-2-minutes-mcp-37he).\n\nIf you don't need the search step (say you have a list of docs pages or articles), use the sister tool that only converts URLs to clean Markdown and can crawl a whole site: [LLM Web Scraper](https://apify.com/get_anything/llm-web-scraper).\n\nPay per result, no subscription. See the Pricing tab on the Actor page. The Apify free plan's monthly credit is enough to try it properly.\n\n**Try it:** [AI Web Search Scraper on Apify](https://apify.com/get_anything/ai-web-search-scraper)\n\nWhat are you using for grounding today, and where does it break? I'm curious which sites give people the most trouble.", "url": "https://wpnews.pro/news/give-your-llm-live-web-search-query-top-results-clean-page-markdown-no-search", "canonical_source": "https://dev.to/get_anything/give-your-llm-live-web-search-query-top-results-clean-page-markdown-no-search-api-key-3apj", "published_at": "2026-10-30 06:39:29+00:00", "updated_at": "2026-10-30 06:52:47.072310+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "large-language-models", "agent-protocols", "ai-search"], "entities": ["Apify", "DuckDuckGo", "Claude", "ChatGPT", "Cursor", "Qdrant"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/give-your-llm-live-web-search-query-top-results-clean-page-markdown-no-search", "markdown": "https://wpnews.pro/news/give-your-llm-live-web-search-query-top-results-clean-page-markdown-no-search.md", "text": "https://wpnews.pro/news/give-your-llm-live-web-search-query-top-results-clean-page-markdown-no-search.txt", "jsonld": "https://wpnews.pro/news/give-your-llm-live-web-search-query-top-results-clean-page-markdown-no-search.jsonld"}}