# 1,656 Sources, 0 Inline Citations: What Grok’s web_search API Returns, and How to Count It

> Source: <https://pub.towardsai.net/1-656-sources-0-inline-citations-what-groks-web-search-api-returns-and-how-to-count-it-233131c1048a?source=rss----98111c9905da---4>
> Published: 2026-11-19 16:31:02+00:00

**Recommendation:** When measuring citations from Grok's Responses API, check each annotation for start_index or end_index before counting it as inline, and report raw, per-answer unique and global unique URL totals separately.
More in the playbook "Evals and testing for LLM apps": <https://wpnews.pro/playbooks/evals-testing>

*Disclosure: this article was written with the help of AI tools. The data collection, numbers and conclusions were checked by the Mention Impossible team against the raw API logs.*

*A single-run evaluation of grok-4.7 on 24 questions, compared with ChatGPT, Claude, Gemini and Perplexity: three valid URL counts from the same responses, and the setup choices that can mislead a comparison.*

If you evaluate LLM answers with web search, at some point you write a line like citations = annotations. For OpenAI-style responses that is usually fine. For Grok's Responses API, in our run it wasn't: 24 answers came back with 1,656 URL annotations, and not one of them pointed to a place in the answer text. They were the model's full reading list, not citations.

This post documents what we measured, how we counted it, where the counting can go wrong, and what the numbers look like next to four other engines asked the same questions. The data comes from Mention Impossible, a weekly measurement of which brands web-search LLMs name in answers to software-buying questions; the method is published at [data.mentionimpossible.com/method](https://data.mentionimpossible.com/method/) [10]. An earlier post [9] covered general traps in brand-mention measurement. This one is about a single API’s output format.

The request body, minus the auth header:

```
{  "model": "grok-4.7",  "input": [{"role": "user", "content": "<question>"}],  "tools": [{"type": "web_search"}],  "max_output_tokens": 2500}
```

A Grok response with web search puts URLs in three places. They are easy to mix up.

xAI’s citations page [4] explains the split. It describes two kinds of citation data: a list of all sources the agent encountered, which includes pages it looked at and judged irrelevant, and inline citations, which are markdown links placed at the point where a source is used. For the Responses API, the page says inline citations are on by default, and when they are off, annotations “only list sources encountered during search” with no positions. Our responses looked like the second case, even though we left the default in place. The same page notes that enabling inline citations does not guarantee the model will cite anything. We can’t tell from our data which explanation applies.

We checked the overlap directly: in every answer, the set of annotated URLs equals the set of unique URLs returned by that answer’s search actions (1,656 of 1,656). So in this run, the annotations are a de-duplicated copy of the search results.

“How many sources did Grok use?” has at least three defensible answers:

The gap between the second and third is URLs that appear in more than one answer. Use per-answer counts when the question is “what did this answer rely on?”, and global counts when it is “how many distinct pages does this engine draw on?”. Just don’t report one under the other’s label.

The counting logic, cut down from our verification script:

``` python
import json, glob
raw_total = per_answer_unique = positioned = 0global_unique = set()for path in glob.glob("GROK/raw/grok__*.json"):    r = json.load(open(path))    consulted = []    for item in r["output"]:        if item["type"] == "web_search_call":            a = item.get("action") or {}            consulted += [s["url"] for s in a.get("sources") or []]            if a.get("url"):                 # open_page carries one URL                consulted.append(a["url"])        elif item["type"] == "message":            for c in item["content"]:                for ann in c.get("annotations") or []:                    if ann.get("start_index") or ann.get("end_index"):                        positioned += 1      # a real inline position    raw_total += len(consulted)    per_answer_unique += len(set(consulted))    global_unique |= set(consulted)
print(raw_total, per_answer_unique, len(global_unique), positioned)# 15 Nov 2026 run: 1862 1656 1524 0
```

The other four engines answered the same 24 questions that week through their official APIs with web search on: ChatGPT (gpt-6.1-sol, OpenAI web search [5]), Claude (claude-sonnet-5-5, Anthropic web search tool [6]), Gemini (gemini-3.8-flash, Grounding with Google Search [7]) and Perplexity (Agent API, "fast" preset [8]). Perplexity ran three times per question, so it has 72 answers; the rest have 24.

Raw URL counts don’t compare well across these APIs. Gemini returned redirect links, so our extracted Gemini rows carry no usable URL, the domain comes from the annotation title, and the 1,272 raw rows repeat heavily. So the unit we compare is **distinct domains per answer**. Each URL is normalised to a lowercased host with www., m. and amp. stripped, and each domain counts once per answer.

**Vendor share** is the fraction of those (answer, domain) pairs where the domain belongs to a brand on our tracked list. A domain matches a brand if it equals one of the brand’s domains or is a subdomain of one. This is a dictionary lookup, so a vendor missing from the list counts as third-party.

Distinct domains per answer, and vendor share:

By raw links instead of domains, Grok’s vendor share is 7.2% (119 of 1,656). Grok’s 24 answers drew on 935 distinct domains, and 764 of them appeared in only one answer.

On these numbers, Grok’s list has about five times as many domains as Claude’s or Gemini’s and about 13 times as many as ChatGPT’s. But that compares a reading list with citation lists. The next section explains why that matters.

**1. Treating the source list as citations.** This is the big one, and our first draft fell into it: it used “sources cited per answer” in the way you would for ChatGPT. A fairer comparison sets Grok next to what other engines consulted. OpenAI’s API exposes this separately: with include: ["web_search_call.action.sources"], the sources field lists every URL the model consulted, and OpenAI's docs note that it is often longer than the citation list [5]. On the same 24 questions, ChatGPT consulted 1,340 unique URLs per answer, summed (about 56 per answer), and cited 117 (4.9 per answer). Grok's 69 per answer belongs next to ChatGPT's 56, not its 4.9.

**2. An instruction asymmetry we introduced.** The other four engines got a short instruction that included “cite your sources” (Perplexity’s added “Cite sources inline like [1][2]”). Grok’s request had no instructions. That may explain part of the missing inline citations, and it makes the inline-citation comparison unfair to Grok. The next run should send the same instruction to every engine.

**3. Absence of X is not evidence about X.** None of the 1,862 consulted URLs was on x.com, twitter.com or t.co, and the usage block reported x_search_calls: 0. That follows from the configuration: X content reaches Grok through x_search [3], which we didn't enable. All this shows is that web_search alone didn't surface X posts for these questions.

**4. Search budgets differ by configuration, not only by model.** Claude ran with max_uses: 2 and used both searches in all 24 answers. ChatGPT used search_context_size: "low" and made two to four search calls per answer. Grok made 5 to 11 billed searches per answer (193 in total). Part of Grok's breadth comes from those settings.

**5. Counters disagree slightly.** The usage block bills 193 web searches; the output contains 194 search actions (172 + 22). We count from output items and treat usage as a billing figure. Usage also reported num_sources_used: 0 for every answer, which we don't interpret.

**6. One run.** Each question ran once. Our Perplexity repeats in the earlier post [9] showed that the brand lists can shift between runs of the same question. Grok’s variance here is unmeasured, not zero.

The general lesson: for every engine, check what the API’s “citations” field actually contains before you aggregate it. In our run, the same word covered a 69-item reading list for Grok and a five-item citation list for ChatGPT.

[1] xAI, Generate text (Responses API), [https://docs.x.ai/developers/model-capabilities/text/generate-text](https://docs.x.ai/developers/model-capabilities/text/generate-text)

[2] xAI, Web Search tool, [https://docs.x.ai/developers/tools/web-search](https://docs.x.ai/developers/tools/web-search)

[3] xAI, X Search tool, [https://docs.x.ai/developers/tools/x-search](https://docs.x.ai/developers/tools/x-search)

[4] xAI, Citations, [https://docs.x.ai/developers/tools/citations](https://docs.x.ai/developers/tools/citations)

[5] OpenAI, Web search (sources and citations), [https://developers.openai.com/api/docs/guides/tools-web-search](https://developers.openai.com/api/docs/guides/tools-web-search)

[6] Anthropic, Web search tool, [https://platform.claude.com/docs/en/agents-and-tools/tool-use/web-search-tool](https://platform.claude.com/docs/en/agents-and-tools/tool-use/web-search-tool)

[7] Google, Grounding with Google Search, [https://ai.google.dev/gemini-api/docs/google-search](https://ai.google.dev/gemini-api/docs/google-search)

[8] Perplexity, Agent API quickstart, [https://docs.perplexity.ai/docs/agent-api/quickstart](https://docs.perplexity.ai/docs/agent-api/quickstart)

[9] WhiskeyJack, Measuring Brand Mentions in LLM Answers Is Easy. Measuring Them Well Isn’t (Towards AI, 2026), [https://medium.com/@whiskeyjackmi/measuring-brand-mentions-in-llm-answers-is-easy-measuring-them-well-isnt-d00af2d111c5](https://medium.com/@whiskeyjackmi/measuring-brand-mentions-in-llm-answers-is-easy-measuring-them-well-isnt-d00af2d111c5)

[10] Mention Impossible, Method, [https://data.mentionimpossible.com/method/](https://data.mentionimpossible.com/method/)

[1,656 Sources, 0 Inline Citations: What Grok’s web_search API Returns, and How to Count It](https://pub.towardsai.net/1-656-sources-0-inline-citations-what-groks-web-search-api-returns-and-how-to-count-it-233131c1048a) was originally published in [Towards AI](https://pub.towardsai.net) on Medium, where people are continuing the conversation by highlighting and responding to this story.
