SimpleQA Verified is a benchmark from Google DeepMind and Google Research for testing how accurately AI models answer factual questions. It contains 1,000 questions, selected from OpenAI's original SimpleQA dataset after checking reference answers, removing similar questions, and balancing topics.
We wanted to see how Google search results and web page content help AI agents search the web and answer factual questions. So we connected SerpApi's Google Search API and Extract API to GPT-6.1 Sol, using low reasoning effort.
After reviewing the saved model inputs, we excluded 40 questions where we detected exposure to public benchmark answers or guidance. The model answered 937 of the remaining 960 questions correctly, giving us 97.6% accuracy.
That puts SerpApi joint #1 by reported accuracy among the results below, matching the 97.6% in Tavily's published evaluation. We tested only SerpApi in the eval; the other providers' scores come from Tavily's and Parallel's published evaluation reports.
How we tested Google Search and Extract for AI agents #
A web search API helps an AI agent find sources, but a search result may not contain enough detail to answer the question. Our setup let the agent use Google SERP data to find relevant pages, read them with Extract, and search again if it needed more information.
The model decided when to search and when to read a page, within the limits below.
| Setting | What we used |
|---|---|
| Answer model | GPT-6.1 Sol ( `gpt-6.1-sol` ), low reasoning |
| Search | SerpApi Google Search API, engine=google |
| Search output | Markdown, using output=md |
| Page content | SerpApi Extract API |
| Search & tool call limit | Multiple searches and fetches allowed per question |
| Grader | GPT-5.6 Sol, high reasoning |
Google SERP data in Markdown
We used the regular SerpApi Google Search API with Markdown output. This gave the model the useful content returned by Google, including organic results, answer boxes, and AI Overviews when available.
We removed metadata and pagination details before passing the results to the model. Searches used no_cache=true to fetch fresh results for every run.
Web page extraction with SerpApi Extract API
When the model needed more detail from a source, it could call the SerpApi Extract API. We used engine=extract with mode=scrape and returned the page's Markdown through include_markdown=true and json_restrictor=text_content.markdown.
The model could then use that page content to answer or make another search. We limited each page fetch to 48,000 characters and each model input to 56,000 tokens.
Grading and exclusions
After the final answers were saved, the grader reviewed each question, reference answer, and prediction without access to search tools.
We detected benchmark leaks by reviewing the search results and page content shown to the model. We excluded 40 responses where search results exposed benchmark answers or evaluation hints.
SimpleQA Verified results and sources #
To put our result in context, we've included the SimpleQA Verified scores published by Tavily and Parallel below.
| Search provider | Accuracy | Source |
|---|---|---|
| SerpApi Google + Extract | 97.6% | Our run, 937/960 after exposure exclusions |
| Tavily Search | 97.6% | [Tavily's evaluation](https://www.tavily.com/blog/tavily-ranks-1-on-sealqa-and-simpleqa) |
| Parallel Advanced | 97.0% | [Parallel's evaluation](https://parallel.ai/articles/best-web-search-api-for-ai-applications) |
| Perplexity Search | 95.0% | [Parallel's evaluation](https://parallel.ai/articles/best-web-search-api-for-ai-applications) |
| Exa Auto | 91.0% | [Parallel's evaluation](https://parallel.ai/articles/best-web-search-api-for-ai-applications) |
Our 97.6% matches Tavily’s published score and is the joint-highest reported accuracy in this comparison. Our agent used Google search data and SerpApi Extract to find sources, read pages, and make follow-up searches when needed.
These scores come from separate evaluations with different setups, so they provide context rather than a direct comparison under identical conditions.
What's next? #
We are also evaluating SealQA, which includes questions with conflicting or unhelpful search evidence. Our completed Seal-0 run scored 58.56% (65/111) using GPT-6.1 Sol at medium reasoning with SerpApi Google and Extract.
We'll share the full SealQA results and methodology in a follow-up post.