{"slug": "which-data-sources-should-you-care-about-for-ai-search", "title": "Which Data Sources Should You Care About For AI Search?", "summary": "Search Engine Journal published a tiered reference table categorizing data sources by their usefulness for AI search, ranking Google Search, Bing Search, Google Merchant Center, OpenAI merchant feeds, and Google Maps as Tier 1 \"confirmed + current\" sources used for RAG, grounding, and actions. The table cites GPT-3 using filtered Common Crawl as roughly 60% of its sampling mixture and LLaMA 1 reporting 67%, with C4 at 15% of LLaMA's pretraining mixture, and notes OpenAI accepts merchant CSV/JSON feed refreshes as often as every 15 minutes. The piece argues that as AI chatbots and tools like AI Overviews and Copilot pull from an increasingly diverse range of sources, practitioners risk \"search source myopia.", "body_md": "If you’re new to “search,” or one of the “old guard,” there’s a risk of some kind of search source myopia. A short-sightedness around what sources really matter for your day-to-day work.\n\nThis isn’t new in AI search, but it’s another bruise that’s getting punched again, and again (and again). As AI chatbots and different tools like AI Overviews, Copilot, etc., pull from a more and more [diverse range of data sources](https://www.searchenginejournal.com/research-suggests-ai-engines-assign-ranking-roles-to-sources/578620/), our job gets a lot harder. And more interesting!\n\n## It’s Hard When You Need To Be Focused On Everything\n\nWhen anything is hard or challenging, that is the place you have an opportunity to really get ahead. So this is the opportunity to really start reviewing the different data sources that could be preferenced or relied upon by AI tools in the future.\n\nI have built a table of search sources and categorised them by how useful I think they’ll be to you right now.\n\nUsing This Data\n\n| Tier | Meaning | \n|---|---|\n| 1 | Confirmed + current — RAG / grounding / actions | \n| 2 | Confirmed + current — training / licensing | \n| 3 | Confirmed historical — pretraining | \n| 4 | Strong evidence / highly likely | \n\nApproach the Tier 1 sources with the most interest – as they’ll more-than-likely be worthwhile. Tier 2 and 3 may be less easy to achieve or even be confident they’ll be beneficial and Tier 4 are highly likely, but lacking confirmation.\n\nAI Data Sources Reference\n\n| Tier | Typical use | Source | Evidence status | What the evidence says | Reference | \n|---|---|---|---|---|---|\n| 1 | Web & search discovery | Google Search | Confirmed + current | Grounding with Google Search connects Gemini to real-time web content, returning inline citations to source URLs. | [Google — Gemini API docs, Grounding with Google Search](https://ai.google.dev/gemini-api/docs/google-search) | \n| 1 |  | Bing Search | Confirmed + current | Microsoft documents Bing results being used to enhance Copilot responses. Not re-verified in this pass. | [Microsoft Bing](https://www.microsoft.com/bing/) | \n| 3 |  | Common Crawl | Confirmed historical | GPT-3 used filtered Common Crawl as roughly 60% of its sampling mixture; LLaMA 1 reported 67%. | [Common Crawl](https://commoncrawl.org/) | \n| 3 |  | Historical web corpora (C4 etc.) | Confirmed historical | C4 is a cleaned derivative of Common Crawl; LLaMA reported C4 at 15% of its pretraining mixture. | [TensorFlow Datasets — C4](https://www.tensorflow.org/datasets/catalog/c4) | \n| 4 |  | Web grounding services | Strong evidence / likely | Category inference covering third-party grounding/retrieval intermediaries. No single canonical source. | — | \n| 1 | Products & shopping | Google Merchant Center | Confirmed + current | Merchant feed data underpins Google’s shopping surfaces. Retained on user instruction; Google Shopping removed as it is a surface, not a source. | [Google Merchant Center Help](https://support.google.com/merchants) | \n| 2 |  | Merchant / retail feeds (OpenAI) | Confirmed + current | Merchants share a secure, regularly refreshed CSV/JSON feed of identifiers, descriptions, pricing, inventory, media and fulfilment so ChatGPT can surface products accurately. Refreshes accepted as often as every 15 minutes. | [OpenAI Developers — Agentic Commerce, product feeds](https://developers.openai.com/commerce/specs) | \n| 4 |  | Microsoft Merchant Center | Strong evidence / likely | Equivalent commercial feed infrastructure; inferred parallel to Google Merchant Center rather than separately evidenced. | — | \n| 4 |  | Marketplace feeds | Strong evidence / likely | Category inference. Shopify catalog data is already integrated into ChatGPT, which supports the pattern. | [OpenAI Help — Shopping with ChatGPT Search](https://help.openai.com/en/articles/11128490-shopping-with-chatgpt-search) | \n| 1 | Local & places | Google Maps | Confirmed + current | Grounding with Google Maps is a documented tool alongside Search grounding, giving models geospatial context. | [Google Cloud — Grounding API](https://docs.cloud.google.com/vertex-ai/generative-ai/docs/model-reference/grounding) | \n| 1 |  | Google Business Profile | Confirmed + current | Business profile data feeds Google’s local surfaces. Carried from the source table; not separately re-verified. | — | \n| 1 |  | Yelp | Confirmed + current | Yelp licenses reviews, photos and business information to OpenAI for real-time local recommendations. Beyond grounding it also drives actions: ChatGPT users can book a table or join a waitlist, and Request a Quote lets users contact providers in-chat. Yelp’s 10-Q confirms it is live. | [Axios; Yelp blog; Yelp 10-Q FY2026](https://www.axios.com/2026/07/23/yelp-reviews-chatgpt-geo-partnership) | \n| 4 |  | OpenStreetMap | Strong evidence / likely | Widely used open geospatial corpus; inferred rather than confirmed for any named model. | — | \n| 4 |  | Foursquare | Strong evidence / likely | Left in Tier 4 deliberately: the OpenAI deal is Yelp’s, and no equivalent evidence exists for Foursquare. | — | \n| 4 |  | Tripadvisor | Strong evidence / likely | Category inference for review/travel data. No confirmed deal identified in this pass. | — | \n| 1 | Knowledge & reference | Wikipedia | Confirmed + current | Explicitly present in GPT-3’s disclosed mixture and LLaMA (June-Aug 2022 dumps, 20 languages); also widely used as a live reference/RAG corpus. | [Wikimedia dumps](https://dumps.wikimedia.org/) | \n| 1 |  | Wikimedia | Confirmed + current | Same corpus family as Wikipedia. Licensing is unusually clear: principally CC BY-SA with attribution/share-alike obligations. | [Wikimedia dumps](https://dumps.wikimedia.org/) | \n| 4 |  | Wikidata | Strong evidence / likely | Structured entity layer; strongly implied by knowledge-graph use but not separately confirmed. | — | \n| 1 | Community / Q&A / social |  | Confirmed + current | The Google deal gave access to the Reddit Data API for ‘real-time, structured, unique content’, and allows Reddit content to be displayed across Google products — i.e. live grounding, not only training. | [Tom’s Guide (Google/Reddit deal)](https://www.tomsguide.com/ai/google-strikes-dollar60m-deal-with-reddit-for-ai-training-data-what-you-need-to-know) | \n| 2 |  |  | Confirmed + current | Same deal, training side: Google may use Reddit posts to train its AI models and improve services such as Search; reported at roughly $60m/yr. NOTE: Reddit is reportedly weighing whether to renew — treat as unstable. | [Fortune; Neowin/WSJ on renewal doubt](https://fortune.com/2024/02/23/reddit-60m-deal-google-search-giant-train-ai-models-on-posts) | \n| 4 |  | Social platforms | Strong evidence / likely | Category inference covering platform-wide social corpora. | — | \n| 4 |  | Forums / communities | Strong evidence / likely | Category inference. Overlaps Reddit but generalised to non-Reddit forums. | — | \n| 1 | News & publisher content | Live publisher pages | Confirmed + current | Reached at inference time via search grounding rather than pretraining; retrieval selection and crawlability govern inclusion. | [Google — Grounding with Google Search](https://ai.google.dev/gemini-api/docs/google-search) | \n| 2 |  | Licensed publisher content | Confirmed + current | OpenAI has multiple explicit licensing partnerships (FT, Axel Springer, AP, News Corp). Terms differ per partner on training vs grounding vs attribution. | [OpenAI — FT content partnership](https://openai.com/index/content-partnership-with-financial-times/) | \n| 2 |  | Publisher partnerships | Confirmed + current | Axel Springer’s deal includes otherwise paywalled material in answers; AP licensed part of its text archive. | [OpenAI — Axel Springer partnership](https://openai.com/index/axel-springer-partnership/) | \n| 3 |  | Historical news corpora | Confirmed historical | Archive material absorbed in pretraining; distinct from live licensed access. | — | \n| 2 | Developer / technical | GitHub | Confirmed + current | LLaMA used GitHub’s public BigQuery dataset, restricted to Apache/BSD/MIT projects; The Pile separately includes GitHub. Public visibility is not an open licence. | [GitHub](https://github.com/) | \n| 2 |  | Stack Overflow | Confirmed + current | Named in licensing-deal mapping alongside Reddit and Shutterstock as a data platform powering multiple buyers. | [LLM Pulse — AI content licensing deals mapped](https://llmpulse.ai/blog/ai-content-licensing-deals/) | \n| 2 |  | Technical docs | Confirmed + current | Vendor documentation corpora; widely used but not tied to a single disclosed agreement. | — | \n| 4 |  | npm / PyPI registries | Strong evidence / likely | WEAKEST ENTRY IN THE TABLE. Relabelled from ‘package registries’ to name examples. No disclosed agreement or documented retrieval use found — consider cutting. | — | \n| 1 | Travel & commerce actions | Google Hotel Center feeds | Confirmed + current | When Gemini or AI Mode show hotel options with real-time prices, that data comes from the Google Hotels feed. In Aug 2026 Google added hotel booking inside AI Mode completed with Google Pay, so this is grounding plus actions. | [TechCrunch — AI Mode travel update](https://techcrunch.com/2026/08/27/googles-ai-mode-can-now-track-flight-prices-help-book-hotels-and-more/) | \n| 4 |  | Booking / partner feeds | Strong evidence / likely | Booking Holdings and IHG are reported as participants in Google’s agentic booking pilot, which supports the direction but stops short of a documented feed spec. | [InfosTourisme (IHG/Booking pilot)](https://infostourisme.com/en/front-page/google-is-testing-ai-powered-hotel-reservations-this-detail-could-change-distribution/2026/) | \n| 4 |  | OTA / commerce sources | Strong evidence / likely | Category inference. OTAs run their own rate feeds into these surfaces. | — | \n| 4 |  | Reservation / inventory APIs | Strong evidence / likely | Category inference covering booking/inventory endpoints exposed to agents. | — | \n\nSource: Chris Green.\n\nThere is a chance this table may date badly – an occupational hazard of “AI Search” at the moment. But another thing you could try is look ahead to where *may* be good sources of data AI providers will be looking for in the future.\n\nIf you’re in a niche or a country (even) where the above data sources have less traction (i.e., Yelp isn’t huge in the UK), perhaps there are other “big players” you may want to look into, even if there isn’t a confirmed relationship yet.\n\nThe most meaningful research you can do as to which of these data sources are most likely shaping AI results today. That’ll require some work studying generated results from some [queries your customers are likely searching for](https://www.searchenginejournal.com/ai-visibility-measurement-what-to-track-what-to-ignore/582009/) – look for the gaps, or the areas you can get ahead.\n\nAs always, any feedback or suggestions welcome. Good luck!\n\n**More Resources:**\n\n- [New Data Finds Gap Between Google Rankings And LLM Citations](https://www.searchenginejournal.com/new-data-finds-gap-between-google-rankings-and-llm-citations/561492/)\n- [Complete Crawler List For AI User-Agents \\[Dec 2025\\]](https://www.searchenginejournal.com/ai-crawler-user-agents-list/558130/)\n- [Timeline Of ChatGPT Updates & Key Events](https://www.searchenginejournal.com/history-of-chatgpt-timeline/488370/)\n\nThis post was originally published on [Chris Green SEO](https://chrisgreenseo.substack.com/p/which-data-sources-should-you-care).\n\n*Featured Image: Gorodenkoff/Shutterstock*\n\n[AI Search](https://www.searchenginejournal.com/category/digital/generative-ai/)", "url": "https://wpnews.pro/news/which-data-sources-should-you-care-about-for-ai-search", "canonical_source": "https://www.searchenginejournal.com/which-data-sources-should-you-care-about-for-ai-search/590086/", "published_at": "2026-09-22 19:00:31+00:00", "updated_at": "2026-09-22 19:22:53.484548+00:00", "lang": "en", "topics": ["ai-search", "generative-engine-optimization", "ai-crawlers", "structured-data", "artificial-intelligence"], "entities": ["Search Engine Journal", "Google Search", "Bing Search", "Common Crawl", "Google Merchant Center", "OpenAI", "ChatGPT", "Google Maps"], "alternates": {"html": "https://wpnews.pro/news/which-data-sources-should-you-care-about-for-ai-search", "markdown": "https://wpnews.pro/news/which-data-sources-should-you-care-about-for-ai-search.md", "text": "https://wpnews.pro/news/which-data-sources-should-you-care-about-for-ai-search.txt", "jsonld": "https://wpnews.pro/news/which-data-sources-should-you-care-about-for-ai-search.jsonld"}}