TIME Is Serving AI Bots a Different Website, With Ads Built In
TIME is now serving two different versions of its website. Humans get the magazine. AI crawlers get a stripped down markdown copy…
Google is an answer engine now. The search box is still there, but the product is built to hand you an answer, and searching the web has turned into a side feature.
Two years ago I wrote that Google now defaults to not indexing your content. Back then the problem was the index. New pages from small sites stayed out unless Google decided it needed them. Since then it has only gotten worse, and by now Google even made this the official direction. At I/O in May, Google announced the biggest upgrade to its search box in 25 years, "completely reimagined with AI", with Gemini 3.5 Flash as the default model in AI Mode for everyone. Google still says you get the normal results, and in June it denied any plan to make AI Mode the default in Chrome. But by Similarweb's count, AI Overviews went from 15% of searches to 43% in one year, and AI Mode visits went from 126 million in June 2025 to 279 million in May 2026. Google wants you to read the answer, and the ten links below are just an afterthought, a legacy feature.
A search engine, the kind Google used to be, tries to index every page it can find and gives you a ranked list for your search term. You can go to page 5, page 20, page 40 and keep finding things. That is how I research. When I want to understand something properly, I dig through the results, open pages that never made it to page one, and find the forum thread or the odd personal site that has the one detail nobody else has. You cannot do that on Google anymore. In September 2025, Google switched off the URL parameter that showed 100 results per page and told Search Engine Land it was "not something that we formally support". In April, I searched for my own name and got two pages of results, most of them my own profiles, although my name appears on plenty of well-known sites. Google's own explainer still says its index holds "hundreds of billions of webpages". You just do not get to see them. You get a generated summary, often shallow and incomplete, and a short list of pages Google picked for you.
Bing is no help. Bing has copied Google for as long as it has existed, and it still does. Microsoft also made the same turn toward answers last year. It retired the Bing Search APIs on August 11, 2025 and pointed developers to "Grounding with Bing Search", which feeds web data into an LLM so the LLM can write a response. Nobody can buy a list of Bing results anymore, only an answer that used Bing along the way.
Mojeek and Brave Search try hard, and I respect both. Mojeek runs its own crawler and passed 9 billion pages in 2025. Brave says its independent index has over 30 billion pages. Both are small operations next to an index of hundreds of billions, and in daily use they are not good enough. Once you leave the obvious queries, the results get thin.
Kagi comes up in every one of these discussions. I pay for it and I like it. The UI is great, and it is the best search tool I use. But it is not a search engine, because it runs on top of other people's indexes. Kagi stopped using Bing years ago after Microsoft raised API prices by up to 10x in 2023, and Microsoft shut the API down anyway. Google does not sell a search API at all. In January, Kagi wrote that it gets its Google-style results through "third-party API providers" and that it would rather "walk through the front door" than depend on "gray-market workarounds". As far as I can tell, that means they are just scraping Google. When Google's results get worse, Kagi's get worse with them.
The EU is trying, sort of. The publicly funded OpenWebSearch.eu project ended in August after 42 months and 8.5 million euros with a pilot index of about 39 billion crawled URLs and a note that it needs more partners and long-term funding to scale. Ecosia and Qwant's joint venture launched a German index of around one billion pages in July, and in March it wrote to all 27 EU governments asking them to fund national search indexes, priced by its CEO at about 50 million euros. One billion pages is a start, but it is not the web. I do not have much faith that a project which needs letters to 27 governments will turn into anything usable.
The best engine for the kind of search I mean, where I do not want an answer but want to search the web, is Yandex, and I am not happy about that. It is a Russian search engine, and if you search for anything political that touches Russia, you get manipulated results. It feels wrong to use it. But for technical and non-political topics it is very good and feels like the old Google. It has a deep index and it lets you page through it. I keep ending up there because there is nowhere else to go.
Just to give the full picture, Baidu and Naver exist too. I have tried both, and they return results that have nothing to do with my query, so they are out for me.
What we need is a full index of the web with ranked results for a query and pages you can click through, run by a company that wants to be a search engine and nothing else. The web is the biggest collection of writing that has ever existed, and there is no working way to search it. We had that fifteen years ago, and losing it is a step backwards.
Give Vroni a GitHub issue, bug report, spec, or rough idea. It reads the repo, plans the change, writes code, runs checks, and works toward a review-ready pull request.
Take a look at vroni.com