{"slug": "how-to-add-web-search-to-an-ai-agent-python-and-mcp", "title": "How to Add Web Search to an AI Agent (Python and MCP)", "summary": "TinySearch, an open-source web retrieval layer that runs locally without a search API key, can add web search to an AI agent by giving it two tools — one that searches and one that reads chosen pages — so the model answers from what it read with source links, according to the guide. Setup takes about two minutes in Claude Code, Cursor or Codex via an MCP command, or about 70 lines of Python for a custom agent using any model. TinySearch's scrape_urls chunks each page and ranks chunks against the question locally, returning a few hundred relevant tokens per page instead of raw HTML.", "body_md": "Engineering\n\n# How to Add Web Search to an AI Agent (Python and MCP)\n\n**To add web search to an AI agent, give it two tools, one that searches and one that reads the pages it chooses, and let the model answer from what it read, with source links.** This guide does that with [TinySearch](https://tinysuite.dev/products/tinysearch/), an open-source web retrieval layer that runs on your machine and needs no search API key. Setup takes about two minutes in Claude Code, Cursor or Codex, and about 70 lines of Python for your own agent with any model.\n\nYou will finish with an agent that can answer questions about current events and fresh documentation, cites its sources, and doesn’t fill its context window with raw HTML.\n\n## The short version\n\n1. **Install a search tool.** Run`claude mcp add --scope user tinysearch -- uvx --python 3.12 --from \"tinysuite-search[server]\" tinysearch` , or add the same command to your client’s MCP config.\n2. **Tell the agent how to research.** Search first, read the best two or three pages, answer only from what it read, cite URLs.\n3. **Check it works.** Ask a question whose answer changed recently and confirm the agent called`search` and`scrape_urls` before answering.\n\nThe rest of this page walks through each path in detail: MCP clients, your own Python agent, and a self-hosted Docker stack.\n\n## How web search works in an agent\n\nEvery reliable agent search setup follows the same loop:\n\n1. **Search.** The model turns the question into a query. The tool returns titles, URLs and short snippets.\n2. **Pick.** The model chooses the two or three results most likely to contain the answer.\n3. **Read.** The tool fetches those pages, strips navigation and boilerplate, and returns only the passages relevant to the question, each with its URL.\n4. **Answer.** The model writes the answer from those passages and cites them.\n\nMost bad agent search comes from skipping step 3, where the agent answers from snippets and guesses, or from doing step 3 badly by pasting whole pages into the prompt. A typical page is mostly menus, cookie banners and related links. Pasting it burns thousands of tokens and buries the paragraph that mattered ([more on that](https://tinysuite.dev/blog/raw-webpages-are-bad-context/)). TinySearch’s `scrape_urls` chunks each page and ranks the chunks against the question locally, so the model gets a few hundred relevant tokens per page instead of the whole thing.\n\n## Before you start\n\nYou need:\n\n- [uv](https://docs.astral.sh/uv/getting-started/installation/) installed (`uv --version` should print a version).\n- An MCP-capable agent client (Claude Code, Cursor, Codex, Claude Desktop, VS Code and others), **or** a Python 3.12+ project for the code path.\n- A model with reliable tool calling.\n\nNo API key, account or Docker is required. The first page read downloads Chromium and a small embedding model. Run this once to get that out of the way:\n\n## Option 1: Add web search to Claude Code, Cursor or Codex\n\n### Step 1: Register the search server\n\n**Claude Code.** `--scope user` makes search available in every project:\n\n**Cursor, Claude Desktop and other JSON-configured clients.** Add this to the client’s MCP config (for Cursor, `~/.cursor/mcp.json`; for Claude Desktop, `claude_desktop_config.json`):\n\n**Codex CLI.** Add this to `~/.codex/config.toml`:\n\nRestart the client. The agent now has `get_current_datetime`, `search`, `scrape_urls`, and two browser tools (` browser_navigate`, `browser_act`) for pages that need clicking or JavaScript. Configs for VS Code and other clients are on the [agent setup page](https://tinysuite.dev/agents/). For Open WebUI specifically, follow [the Open WebUI walkthrough](https://tinysuite.dev/blog/add-web-search-to-local-llm-agent/).\n\n### Step 2: Tell the agent how to research\n\nCapable models use search tools without being told, but a few rules make the results much more reliable. Add this to `CLAUDE.md`, `AGENTS.md`, Cursor rules or your system prompt:\n\n### Step 3: Verify it works\n\nAsk something the model can’t know from training data:\n\nWhat is the latest stable release of Python, and what are its headline changes? Cite the release notes.\n\nOpen the tool activity. A working setup shows:\n\n1. A `get_current_datetime` call.\n2. A `search` call that returns python.org or docs.python.org URLs.\n3. A `scrape_urls` call on one or more of those URLs.\n4. An answer with links you can open and check.\n\nAn answer that comes back without tool calls is the model answering from memory, not a working search setup. See [Troubleshooting](#troubleshooting).\n\n## Option 2: Add web search to your own Python agent\n\nIf you’re building the agent, call TinySearch as a library. It works with any model that speaks the OpenAI chat API: OpenAI, OpenRouter, Ollama, LM Studio, vLLM or llama.cpp.\n\n### Step 1: Install\n\nPoint the OpenAI SDK at your model:\n\n### Step 2: The complete agent\n\nSave this as `search_agent.py`:\n\n### Step 3: Test it\n\nA good answer says to pass `format=json`, mentions that the format has to be enabled in `settings.yml` or the instance returns 403 Forbidden, and links `docs.searxng.org`. Add `print(call.function.name, args)` inside the loop to watch the agent search, pick and read.\n\n### Making it production-ready\n\n- **Restrict domains when you can.**`tinysearch.search([{\"query\": q, \"domains\": [\"docs.python.org\"]}])` keeps a docs assistant on official sources.\n- **Search several angles at once.**`search()` and`scrape_urls()` each take up to five items per call, run concurrently.\n- **Control the budget.**`max_tokens` on`scrape_urls` caps the evidence per page. Between 1,000 and 2,000 tokens is enough for most questions.\n- **Pages that need JavaScript or a click.** Use the MCP server’s`browser_navigate` and`browser_act` tools, or read the[browser tool docs](https://tinysuite.dev/docs/tinysearch/#mcp-tools) .\n- **Choose your search backend.** DDGS is the default. A self-hosted SearXNG instance gives you control over which engines are queried. See[SearXNG vs DuckDuckGo and DDGS](https://tinysuite.dev/blog/searxng-vs-duckduckgo-agent-search/) .\n\n## Option 3: Run a self-hosted search stack with Docker\n\nFor a shared, always-on service, run TinySearch with its own SearXNG instance:\n\nThen connect any MCP client to the Streamable HTTP endpoint:\n\nKeep it on a trusted network or behind an authenticated proxy. Configuration options are in the [TinySearch docs](https://tinysuite.dev/docs/tinysearch/#configuration).\n\n## Why this beats the usual alternatives\n\n| Approach | How it works | Where it breaks down | \n|---|---|---|\n| **Your model provider’s built-in search** | The provider runs search on its side | Billed per search, tied to that provider’s models, and not available for local models | \n| **Hosted search APIs** (Tavily, Exa, Brave, Perplexity Sonar) | An API returns results or pre-written answers | API key, per-request billing, and every query goes through a third party ( [detailed comparison](https://tinysuite.dev/blog/tinysearch-vs-tavily-exa-perplexity-brave/) ) | \n| **Fetch the page and paste it** | `requests.get()` plus the raw HTML or text in the prompt | Thousands of wasted tokens per page, and the relevant paragraph gets buried | \n| **SearXNG or DuckDuckGo alone** | Free search results | Links and snippets only. Reading, cleaning and ranking pages is left to you | \n| **Full browser agents** | The model drives a browser through screenshots | Slow and expensive per step. Overkill for reading a documentation page | \n| **TinySearch** | Search, read, chunk and rerank locally, return cited passages | Not a managed high-volume API with an SLA | \n\nWhat TinySearch does differently:\n\n- **No API key, no per-search bill.** It uses DDGS by default, or your own SearXNG. Your only cost is the model, and nothing if the model is local.\n- **The model sees evidence, not pages.** Pages are chunked and ranked against the question with BM25 plus local embeddings, and only the best passages come back, each with its source URL.\n- **Your model writes the answer.** TinySearch makes no LLM call of its own. Nothing is pre-summarised by a model you didn’t choose, and every claim traces back to a passage.\n- **Works with any model and client.** MCP for Claude Code, Cursor, Codex and Open WebUI, a Python library for your own code, and Docker for a shared service.\n- **Bounded by design.** Up to five items per call, token budgets per page, and related links returned to the model rather than crawled automatically, so one question can’t spiral into a crawl ([why](https://tinysuite.dev/blog/tinysearch-related-links-link-discovery-mcp/) ).\n- **Handles the awkward pages.** PDFs and DOCX files are read directly, and the browser tools cover JavaScript-rendered pages, consent banners and “load more” buttons.\n\n**When to choose something else:** if you need a managed search API with an SLA at very high request volumes, use a hosted provider. If your agent can’t run local processes at all, for example a browser-only app with no server, a hosted API is the only option.\n\nBe clear about privacy: inference and reranking can stay entirely on your machine, but search queries still go to external search engines, and reading a page contacts that website.\n\n## Troubleshooting\n\n| Problem | Fix | \n|---|---|\n| The tools don’t appear | Restart the client after editing its config, and check that `uvx --version` works in the shell the client uses | \n| The first page read is slow | Run `tinysearch setup` once to download Chromium and the embedding model in advance | \n| The agent answers without searching | Add the research rules from Step 2, and check that your model supports tool calling | \n| Search returns errors or nothing | DDGS backends can rate-limit. Retry, or [configure SearXNG](https://tinysuite.dev/blog/searxng-vs-duckduckgo-agent-search/) | \n| A page comes back empty | It probably needs JavaScript or interaction. Use `browser_navigate` , or let the agent pick another source | \n| Answers cite pages that don’t say that | Tell the model to quote or closely paraphrase passages, and spot-check the cited URLs | \n\nRun `tinysearch doctor` in the same environment as your client to check its setup without downloading anything.\n\n## Next steps\n\nWeb search lets an agent find what it doesn’t know. To make it remember what it has already learned, and what you’ve told it, [add long-term memory to your agent](https://tinysuite.dev/blog/how-to-add-memory-to-ai-agent/) with TinyContext, using the same setup. If you’d rather have a ready-made chat app with search and memory already connected, run `npx tinywebui` ([TinyWebUI](https://tinysuite.dev/products/tinywebui/)).\n\nThe full tool reference and configuration are in the [TinySearch docs](https://tinysuite.dev/docs/tinysearch/). The source is on [GitHub](https://github.com/TinySuiteHQ/TinySearch).\n\n## Frequently asked questions\n\n### How do I add web search to an AI agent?\n\nGive the agent two tools: one that searches the web and returns titles, URLs and snippets, and one that reads the pages it picks and returns the relevant passages with their URLs. The model calls them, then answers from the evidence with citations. With TinySearch you add one MCP server to Claude Code, Cursor or Codex, or call search() and scrape_urls() from your own Python agent. No search API key is required.\n\n### Can I add web search to an AI agent for free?\n\nYes. TinySearch is open source and uses the DDGS metasearch library by default, so it needs no paid search API key or account. You can also point it at a self-hosted SearXNG instance. You only pay for the model that writes the answer, and nothing if that model runs locally.\n\n### Why does an agent need to read pages, not just search?\n\nSearch results give a title and a one or two sentence snippet. That is enough to choose which pages to open, but rarely enough to answer correctly. Agents that answer from snippets alone guess. Reading the chosen pages and keeping only the passages relevant to the question gives the model real evidence and a URL to cite.\n\n### Does web search work with local models like Ollama or LM Studio?\n\nYes. Web search is a tool, so any model with reliable tool calling can use it, including local models served by Ollama, LM Studio, vLLM or llama.cpp. TinySearch also runs its reranking embeddings locally. The search queries themselves still go to external search engines, and reading a page contacts that website.\n\n### How do I stop web pages from hijacking my agent?\n\nTreat everything returned by a search or page read as untrusted data. Say so in the system prompt, keep tool results clearly separated from instructions, don't give a browsing agent tools that can send email, spend money or change files without confirmation, and limit how many tool calls one answer can make.\n\n### What is the difference between TinySearch and Tavily or Exa?\n\nTavily, Exa, Brave Search API and Perplexity Sonar are hosted APIs that need an API key and bill per request. TinySearch runs on your own machine, needs no key by default, and does the page reading and passage ranking locally before anything reaches the model. Hosted APIs are a better fit when you need a managed service with an SLA at high volume.", "url": "https://wpnews.pro/news/how-to-add-web-search-to-an-ai-agent-python-and-mcp", "canonical_source": "https://tinysuite.dev/blog/how-to-add-web-search-to-ai-agent/", "published_at": "2026-11-02 00:00:00+00:00", "updated_at": "2026-11-02 11:19:56.677564+00:00", "lang": "en", "topics": ["ai-agents", "agent-protocols", "ai-tools", "developer-tools", "ai-search"], "entities": ["TinySearch", "Claude Code", "Cursor", "Codex", "Claude Desktop", "VS Code", "Open WebUI", "uv"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/how-to-add-web-search-to-an-ai-agent-python-and-mcp", "markdown": "https://wpnews.pro/news/how-to-add-web-search-to-an-ai-agent-python-and-mcp.md", "text": "https://wpnews.pro/news/how-to-add-web-search-to-an-ai-agent-python-and-mcp.txt", "jsonld": "https://wpnews.pro/news/how-to-add-web-search-to-an-ai-agent-python-and-mcp.jsonld"}}