How to Add Web Search to an AI Agent (Python and MCP)

TinySearch, an open-source web retrieval layer that runs locally without a search API key, can add web search to an AI agent by giving it two tools — one that searches and one that reads chosen pages — so the model answers from what it read with source links, according to the guide. Setup takes about two minutes in Claude Code, Cursor or Codex via an MCP command, or about 70 lines of Python for a custom agent using any model. TinySearch's scrape_urls chunks each page and ranks chunks against the question locally, returning a few hundred relevant tokens per page instead of raw HTML.

Engineering How to Add Web Search to an AI Agent Python and MCP To add web search to an AI agent, give it two tools, one that searches and one that reads the pages it chooses, and let the model answer from what it read, with source links. This guide does that with TinySearch https://tinysuite.dev/products/tinysearch/ , an open-source web retrieval layer that runs on your machine and needs no search API key. Setup takes about two minutes in Claude Code, Cursor or Codex, and about 70 lines of Python for your own agent with any model. You will finish with an agent that can answer questions about current events and fresh documentation, cites its sources, and doesn’t fill its context window with raw HTML. The short version 1. Install a search tool. Run claude mcp add --scope user tinysearch -- uvx --python 3.12 --from "tinysuite-search server " tinysearch , or add the same command to your client’s MCP config. 2. Tell the agent how to research. Search first, read the best two or three pages, answer only from what it read, cite URLs. 3. Check it works. Ask a question whose answer changed recently and confirm the agent called search and scrape urls before answering. The rest of this page walks through each path in detail: MCP clients, your own Python agent, and a self-hosted Docker stack. How web search works in an agent Every reliable agent search setup follows the same loop: 1. Search. The model turns the question into a query. The tool returns titles, URLs and short snippets. 2. Pick. The model chooses the two or three results most likely to contain the answer. 3. Read. The tool fetches those pages, strips navigation and boilerplate, and returns only the passages relevant to the question, each with its URL. 4. Answer. The model writes the answer from those passages and cites them. Most bad agent search comes from skipping step 3, where the agent answers from snippets and guesses, or from doing step 3 badly by pasting whole pages into the prompt. A typical page is mostly menus, cookie banners and related links. Pasting it burns thousands of tokens and buries the paragraph that mattered more on that https://tinysuite.dev/blog/raw-webpages-are-bad-context/ . TinySearch’s scrape urls chunks each page and ranks the chunks against the question locally, so the model gets a few hundred relevant tokens per page instead of the whole thing. Before you start You need: - uv https://docs.astral.sh/uv/getting-started/installation/ installed uv --version should print a version . - An MCP-capable agent client Claude Code, Cursor, Codex, Claude Desktop, VS Code and others , or a Python 3.12+ project for the code path. - A model with reliable tool calling. No API key, account or Docker is required. The first page read downloads Chromium and a small embedding model. Run this once to get that out of the way: Option 1: Add web search to Claude Code, Cursor or Codex Step 1: Register the search server Claude Code. --scope user makes search available in every project: Cursor, Claude Desktop and other JSON-configured clients. Add this to the client’s MCP config for Cursor, ~/.cursor/mcp.json ; for Claude Desktop, claude desktop config.json : Codex CLI. Add this to ~/.codex/config.toml : Restart the client. The agent now has get current datetime , search , scrape urls , and two browser tools browser navigate , browser act for pages that need clicking or JavaScript. Configs for VS Code and other clients are on the agent setup page https://tinysuite.dev/agents/ . For Open WebUI specifically, follow the Open WebUI walkthrough https://tinysuite.dev/blog/add-web-search-to-local-llm-agent/ . Step 2: Tell the agent how to research Capable models use search tools without being told, but a few rules make the results much more reliable. Add this to CLAUDE.md , AGENTS.md , Cursor rules or your system prompt: Step 3: Verify it works Ask something the model can’t know from training data: What is the latest stable release of Python, and what are its headline changes? Cite the release notes. Open the tool activity. A working setup shows: 1. A get current datetime call. 2. A search call that returns python.org or docs.python.org URLs. 3. A scrape urls call on one or more of those URLs. 4. An answer with links you can open and check. An answer that comes back without tool calls is the model answering from memory, not a working search setup. See Troubleshooting troubleshooting . Option 2: Add web search to your own Python agent If you’re building the agent, call TinySearch as a library. It works with any model that speaks the OpenAI chat API: OpenAI, OpenRouter, Ollama, LM Studio, vLLM or llama.cpp. Step 1: Install Point the OpenAI SDK at your model: Step 2: The complete agent Save this as search agent.py : Step 3: Test it A good answer says to pass format=json , mentions that the format has to be enabled in settings.yml or the instance returns 403 Forbidden, and links docs.searxng.org . Add print call.function.name, args inside the loop to watch the agent search, pick and read. Making it production-ready - Restrict domains when you can. tinysearch.search {"query": q, "domains": "docs.python.org" } keeps a docs assistant on official sources. - Search several angles at once. search and scrape urls each take up to five items per call, run concurrently. - Control the budget. max tokens on scrape urls caps the evidence per page. Between 1,000 and 2,000 tokens is enough for most questions. - Pages that need JavaScript or a click. Use the MCP server’s browser navigate and browser act tools, or read the browser tool docs https://tinysuite.dev/docs/tinysearch/ mcp-tools . - Choose your search backend. DDGS is the default. A self-hosted SearXNG instance gives you control over which engines are queried. See SearXNG vs DuckDuckGo and DDGS https://tinysuite.dev/blog/searxng-vs-duckduckgo-agent-search/ . Option 3: Run a self-hosted search stack with Docker For a shared, always-on service, run TinySearch with its own SearXNG instance: Then connect any MCP client to the Streamable HTTP endpoint: Keep it on a trusted network or behind an authenticated proxy. Configuration options are in the TinySearch docs https://tinysuite.dev/docs/tinysearch/ configuration . Why this beats the usual alternatives | Approach | How it works | Where it breaks down | |---|---|---| | Your model provider’s built-in search | The provider runs search on its side | Billed per search, tied to that provider’s models, and not available for local models | | Hosted search APIs Tavily, Exa, Brave, Perplexity Sonar | An API returns results or pre-written answers | API key, per-request billing, and every query goes through a third party detailed comparison https://tinysuite.dev/blog/tinysearch-vs-tavily-exa-perplexity-brave/ | | Fetch the page and paste it | requests.get plus the raw HTML or text in the prompt | Thousands of wasted tokens per page, and the relevant paragraph gets buried | | SearXNG or DuckDuckGo alone | Free search results | Links and snippets only. Reading, cleaning and ranking pages is left to you | | Full browser agents | The model drives a browser through screenshots | Slow and expensive per step. Overkill for reading a documentation page | | TinySearch | Search, read, chunk and rerank locally, return cited passages | Not a managed high-volume API with an SLA | What TinySearch does differently: - No API key, no per-search bill. It uses DDGS by default, or your own SearXNG. Your only cost is the model, and nothing if the model is local. - The model sees evidence, not pages. Pages are chunked and ranked against the question with BM25 plus local embeddings, and only the best passages come back, each with its source URL. - Your model writes the answer. TinySearch makes no LLM call of its own. Nothing is pre-summarised by a model you didn’t choose, and every claim traces back to a passage. - Works with any model and client. MCP for Claude Code, Cursor, Codex and Open WebUI, a Python library for your own code, and Docker for a shared service. - Bounded by design. Up to five items per call, token budgets per page, and related links returned to the model rather than crawled automatically, so one question can’t spiral into a crawl why https://tinysuite.dev/blog/tinysearch-related-links-link-discovery-mcp/ . - Handles the awkward pages. PDFs and DOCX files are read directly, and the browser tools cover JavaScript-rendered pages, consent banners and “load more” buttons. When to choose something else: if you need a managed search API with an SLA at very high request volumes, use a hosted provider. If your agent can’t run local processes at all, for example a browser-only app with no server, a hosted API is the only option. Be clear about privacy: inference and reranking can stay entirely on your machine, but search queries still go to external search engines, and reading a page contacts that website. Troubleshooting | Problem | Fix | |---|---| | The tools don’t appear | Restart the client after editing its config, and check that uvx --version works in the shell the client uses | | The first page read is slow | Run tinysearch setup once to download Chromium and the embedding model in advance | | The agent answers without searching | Add the research rules from Step 2, and check that your model supports tool calling | | Search returns errors or nothing | DDGS backends can rate-limit. Retry, or configure SearXNG https://tinysuite.dev/blog/searxng-vs-duckduckgo-agent-search/ | | A page comes back empty | It probably needs JavaScript or interaction. Use browser navigate , or let the agent pick another source | | Answers cite pages that don’t say that | Tell the model to quote or closely paraphrase passages, and spot-check the cited URLs | Run tinysearch doctor in the same environment as your client to check its setup without downloading anything. Next steps Web search lets an agent find what it doesn’t know. To make it remember what it has already learned, and what you’ve told it, add long-term memory to your agent https://tinysuite.dev/blog/how-to-add-memory-to-ai-agent/ with TinyContext, using the same setup. If you’d rather have a ready-made chat app with search and memory already connected, run npx tinywebui TinyWebUI https://tinysuite.dev/products/tinywebui/ . The full tool reference and configuration are in the TinySearch docs https://tinysuite.dev/docs/tinysearch/ . The source is on GitHub https://github.com/TinySuiteHQ/TinySearch . Frequently asked questions How do I add web search to an AI agent? Give the agent two tools: one that searches the web and returns titles, URLs and snippets, and one that reads the pages it picks and returns the relevant passages with their URLs. The model calls them, then answers from the evidence with citations. With TinySearch you add one MCP server to Claude Code, Cursor or Codex, or call search and scrape urls from your own Python agent. No search API key is required. Can I add web search to an AI agent for free? Yes. TinySearch is open source and uses the DDGS metasearch library by default, so it needs no paid search API key or account. You can also point it at a self-hosted SearXNG instance. You only pay for the model that writes the answer, and nothing if that model runs locally. Why does an agent need to read pages, not just search? Search results give a title and a one or two sentence snippet. That is enough to choose which pages to open, but rarely enough to answer correctly. Agents that answer from snippets alone guess. Reading the chosen pages and keeping only the passages relevant to the question gives the model real evidence and a URL to cite. Does web search work with local models like Ollama or LM Studio? Yes. Web search is a tool, so any model with reliable tool calling can use it, including local models served by Ollama, LM Studio, vLLM or llama.cpp. TinySearch also runs its reranking embeddings locally. The search queries themselves still go to external search engines, and reading a page contacts that website. How do I stop web pages from hijacking my agent? Treat everything returned by a search or page read as untrusted data. Say so in the system prompt, keep tool results clearly separated from instructions, don't give a browsing agent tools that can send email, spend money or change files without confirmation, and limit how many tool calls one answer can make. What is the difference between TinySearch and Tavily or Exa? Tavily, Exa, Brave Search API and Perplexity Sonar are hosted APIs that need an API key and bill per request. TinySearch runs on your own machine, needs no key by default, and does the page reading and passage ranking locally before anything reaches the model. Hosted APIs are a better fit when you need a managed service with an SLA at high volume.