{"slug": "simulate-google-s-query-fan-out-to-find-your-content-gaps-python", "title": "Simulate Google's Query Fan-Out to Find Your Content Gaps (Python)", "summary": "A developer published a Python script that approximates Google's query fan-out by using Claude to generate likely sub-queries for a seed question, then splitting a target page into passages, embedding them with a small open-source model, and scoring each sub-query against its best-matching passage to produce a coverage report that surfaces content gaps first. The approach is motivated by Google's documentation that AI Overviews and AI Mode may issue multiple related searches across subtopics, and by Bing Webmaster Tools' AI Performance report exposing the grounding queries Copilot generates before citing pages. The author recommends running the generation step 3–5 times and merging results, since real fan-out is not deterministic.", "body_md": "When someone asks Google's AI Mode \"what's the best way to host a Next.js app for a small team?\", Google doesn't run one search. It splits the question into sub-questions and runs those in parallel. Google calls this **query fan-out**. Its docs say AI Overviews and AI Mode \"may use a 'query fan-out' technique, issuing multiple related searches across subtopics and data sources, to develop a response\" ([Search Central](https://developers.google.com/search/docs/appearance/ai-features)).\n\nMicrosoft lets you see this happening. Bing Webmaster Tools' AI Performance report lists **grounding queries**, the background queries Copilot generated before citing your pages. They're shorter and more specific than what users type.\n\nThat changes what \"covering a topic\" means. Your page isn't competing for one query. It's competing for 5–15 machine-generated ones, and every sub-question you skip is a retrieval you hand to someone else.\n\nYou can't see Google's real fan-out. But you can approximate it closely enough to find gaps. Here's a script that does exactly that.\n\n```\nseed question ──► LLM generates likely sub-queries\n                             │\nyour page ──► split into passages ──► embed\n                             │\n              for each sub-query: best-matching passage + score\n                             │\n                     coverage report (gaps first)\n```\n\nTwo parts:\n\n```\npip install anthropic pydantic sentence-transformers requests beautifulsoup4\nexport ANTHROPIC_API_KEY=...\n```\n\nI'm using Claude for generation and a small open-source embedding model for scoring. Both are swappable.\n\nStructured output keeps this clean: we ask for a typed list and get a validated Python object back, with no regex over model text.\n\n``` python\n# fanout.py\nfrom typing import Literal\nimport anthropic\nfrom pydantic import BaseModel\n\nclass SubQuery(BaseModel):\n    query: str\n    intent: Literal[\"definition\", \"comparison\", \"how-to\", \"pricing\", \"constraint\", \"alternative\", \"follow-up\"]\n\nclass FanOut(BaseModel):\n    sub_queries: list[SubQuery]\n\nclient = anthropic.Anthropic()\n\nPROMPT = \"\"\"You are simulating the query fan-out step of an AI search engine.\nGiven a user's question, list the 12 short web searches the engine would most\nlikely run in parallel to build a complete, trustworthy answer.\n\nCover different facets: definitions, comparisons, pricing, constraints\n(size, budget, region, stack), how-to steps, alternatives, and the follow-up a\ncareful user would ask next. Write each one the way a search engine would\n(short, keyword-like), not as a full sentence.\n\nUser question: {question}\"\"\"\n\ndef fan_out(question: str) -> list[SubQuery]:\n    response = client.messages.parse(\n        model=\"claude-opus-5-5\",\n        max_tokens=16000,\n        output_config={\"effort\": \"low\"},  # this is a simple task; keep it cheap\n        messages=[{\"role\": \"user\", \"content\": PROMPT.format(question=question)}],\n        output_format=FanOut,\n    )\n    return response.parsed_output.sub_queries\n```\n\nRun it on a real question:\n\n```\nfor sq in fan_out(\"best way to host a Next.js app for a small team\"):\n    print(f\"[{sq.intent:11}] {sq.query}\")\n[comparison ] vercel vs netlify vs cloudflare pages nextjs\n[pricing    ] vercel pro pricing per seat 2026\n[constraint ] nextjs self hosting docker small team\n[how-to     ] deploy nextjs app router to cloudflare\n[constraint ] nextjs ISR support hosting providers\n[alternative] nextjs hosting without vercel\n[follow-up  ] nextjs hosting cold starts serverless\n...\n```\n\nGenerated output varies between runs. That's fine. Run it 3–5 times and merge the results, because real fan-out isn't deterministic either.\n\nAI systems retrieve **passages**, not whole pages. We'll split by headings and paragraphs, keeping each heading attached to its text so the passage still makes sense on its own.\n\n``` python\n# passages.py\nimport requests\nfrom bs4 import BeautifulSoup\n\ndef page_passages(url: str, min_words: int = 25) -> list[str]:\n    html = requests.get(url, timeout=20, headers={\"User-Agent\": \"fanout-audit/1.0\"}).text\n    soup = BeautifulSoup(html, \"html.parser\")\n    for tag in soup([\"script\", \"style\", \"nav\", \"footer\", \"header\", \"aside\"]):\n        tag.decompose()\n\n    root = soup.find(\"main\") or soup.find(\"article\") or soup.body\n    passages, heading = [], \"\"\n    for el in root.find_all([\"h1\", \"h2\", \"h3\", \"p\", \"li\", \"td\"]):\n        text = \" \".join(el.get_text(\" \", strip=True).split())\n        if el.name in (\"h1\", \"h2\", \"h3\"):\n            heading = text\n            continue\n        if len(text.split()) >= min_words:\n            passages.append(f\"{heading}: {text}\" if heading else text)\n    return passages\n```\n\nNote that this reads the **raw HTML**, the way many AI fetchers do. If your content only appears after JavaScript runs, this returns almost nothing, and that's a finding in its own right (see Part 9).\n\n``` python\n# coverage.py\nfrom sentence_transformers import SentenceTransformer, util\n\nmodel = SentenceTransformer(\"BAAI/bge-small-en-v1.5\")\n\ndef coverage(sub_queries, passages):\n    q_emb = model.encode([sq.query for sq in sub_queries], normalize_embeddings=True)\n    p_emb = model.encode(passages, normalize_embeddings=True)\n    sims = util.cos_sim(q_emb, p_emb)  # [n_queries, n_passages]\n\n    rows = []\n    for i, sq in enumerate(sub_queries):\n        best = int(sims[i].argmax())\n        rows.append({\n            \"query\": sq.query,\n            \"intent\": sq.intent,\n            \"score\": float(sims[i][best]),\n            \"best_passage\": passages[best][:110] + \"…\",\n        })\n    return sorted(rows, key=lambda r: r[\"score\"])  # gaps first\npython\n# main.py\nimport sys\nfrom fanout import fan_out\nfrom passages import page_passages\nfrom coverage import coverage\n\nquestion, url = sys.argv[1], sys.argv[2]\n\n# Merge a few runs to smooth out randomness.\nseen, subs = set(), []\nfor _ in range(3):\n    for sq in fan_out(question):\n        key = sq.query.lower()\n        if key not in seen:\n            seen.add(key)\n            subs.append(sq)\n\npassages = page_passages(url)\nprint(f\"{len(subs)} sub-queries · {len(passages)} passages\\n\")\n\nfor r in coverage(subs, passages):\n    flag = \"GAP \" if r[\"score\"] < 0.55 else \"weak\" if r[\"score\"] < 0.68 else \"ok  \"\n    print(f\"{flag} {r['score']:.2f}  [{r['intent']}] {r['query']}\")\n    if flag != \"ok  \":\n        print(f\"           closest: {r['best_passage']}\")\npython main.py \"best way to host a Next.js app for a small team\" https://your-site.com/blog/nextjs-hosting\n```\n\n*(Illustrative output. Your numbers will differ.)*\n\n```\n33 sub-queries · 41 passages\n\nGAP  0.41  [pricing] vercel pro pricing per seat 2026\n           closest: Why we moved: Our bill grew faster than our traffic…\nGAP  0.47  [constraint] nextjs ISR support hosting providers\n           closest: Deploying to Cloudflare: The adapter handles most…\nweak 0.61  [follow-up] nextjs hosting cold starts serverless\n...\nok   0.79  [comparison] vercel vs netlify vs cloudflare pages nextjs\n```\n\n`bge-small` on English technical content. Calibrate them by checking a few passages yourself: find the score below which the \"closest passage\" clearly doesn't answer the question.\nIf you want a reality check, compare your generated sub-queries with Bing's real grounding queries (Bing Webmaster Tools → AI Performance, CSV export). Matching phrasing there is the closest thing to ground truth you'll get today.\n\nCoverage tells you whether your page *can* answer the sub-questions. It doesn't tell you whether AI engines actually *do* pick you. For that you need to run the real prompts across ChatGPT, Perplexity, Gemini and others over time and track who gets mentioned. We built [Vista AI's AI rank tracker](https://www.vistaai.io/features/ai-rank-tracker) for that part. The script above is the free, local half of the loop: find gaps, fix them, then measure.\n\nThe full script is ~120 lines. If you adapt it (different embedding model, multilingual, or batch mode over a sitemap), share it in the comments.\n\nNext in the series: **Track ChatGPT, Perplexity and Claude referrals properly.**", "url": "https://wpnews.pro/news/simulate-google-s-query-fan-out-to-find-your-content-gaps-python", "canonical_source": "https://dev.to/furqank729/simulate-googles-query-fan-out-to-find-your-content-gaps-python-4e0c", "published_at": "2026-10-22 06:00:00+00:00", "updated_at": "2026-10-22 06:10:53.376450+00:00", "lang": "en", "topics": ["generative-engine-optimization", "ai-search", "ai-tools", "large-language-models"], "entities": ["Google", "Bing Webmaster Tools", "Microsoft", "Copilot", "Anthropic", "Claude", "Next.js", "Vercel"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/simulate-google-s-query-fan-out-to-find-your-content-gaps-python", "markdown": "https://wpnews.pro/news/simulate-google-s-query-fan-out-to-find-your-content-gaps-python.md", "text": "https://wpnews.pro/news/simulate-google-s-query-fan-out-to-find-your-content-gaps-python.txt", "jsonld": "https://wpnews.pro/news/simulate-google-s-query-fan-out-to-find-your-content-gaps-python.jsonld"}}