{"slug": "can-ai-shopping-agents-get-into-germany-s-biggest-online-shops-i-checked-46-txt", "title": "Can AI shopping agents get into Germany's biggest online shops? I checked 46: robots.txt is rarely the problem, the front door is", "summary": "A developer who built an AI crawler and agent access checker on Apify tested 46 large online shops selling in Germany on 3 November 2026 and found that robots.txt is rarely the barrier to AI shopping agents — bot protection is. Only nine shops block any AI agents in robots.txt, with Amazon.de the sole shop blocking the ChatGPT-User, Claude-User and Perplexity-User agents, while 16 of 46 shops (35%) refused to serve their home page to a plain automated client, including MediaMarkt, Saturn, Kaufland, H&M and eBay. Four shops with robots.txt blocking zero AI agents — Douglas, H&M, Decathlon and apotheke.de — still returned 403s, and just 9 of 46 serve an llms.txt file.", "body_md": "AI assistants are starting to shop. ChatGPT, Claude and Perplexity can now open a product page on a user's behalf, compare prices and fill a cart. Whether that works depends on two things a shop controls: what its `robots.txt` says, and whether its bot protection lets the visitor in at all.\n\nOn 3 November 2026 I checked 46 large online shops that sell in Germany: marketplaces (Amazon, eBay, Otto, Kaufland), electronics (MediaMarkt, Saturn, Cyberport, notebooksbilliger), fashion (Zalando, About You, H&M, Breuninger), drugstores and pharmacies (dm, Rossmann, Shop Apotheke, DocMorris), DIY, furniture, sports and pets.\n\n*Disclosure: I built the checker used here, the [AI Crawler & Agent Access Checker](https://apify.com/fast_data_asap/ai-crawler-access-checker) on Apify. How to rerun this is at the end.*\n\nIt helps to separate the bots, because shops treat them very differently:\n\nBlocking (1) keeps your content out of training data. Blocking (3) means an assistant cannot look at your product page when a customer asks it to.\n\nI could read `robots.txt` for 32 of the 46 shops (the other 14 refused or timed out for an automated client, more on that below).\n\nThe nine that block something:\n\n| Shop | AI agents blocked in robots.txt | \n|---|---|\n| amazon.de | 12: all training crawlers, all three AI search indexers, **and all three user agents** (ChatGPT-User, Claude-User, Perplexity-User) | \n| zalando.de | 8: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, meta-externalagent, Bytespider, CCBot, Amazonbot | \n| ebay.de | 8: GPTBot, ClaudeBot, PerplexityBot, Applebot-Extended, meta-externalagent, Bytespider, CCBot, Amazonbot | \n| notebooksbilliger.de | 7: GPTBot, ClaudeBot, Applebot-Extended, meta-externalagent, Bytespider, CCBot, Amazonbot | \n| home24.de | 2: ClaudeBot, Amazonbot | \n| thomann.de | 1: GPTBot | \n| rossmann.de | 1: ClaudeBot | \n| idealo.de | 1: Applebot-Extended | \n| ikea.com | 1: Amazonbot | \n\nTwo patterns stand out.\n\n**Amazon is the only shop that blocks the user agents.** Every other shop with a readable `robots.txt` lets ChatGPT-User, Claude-User and Perplexity-User in. Amazon.de says no to all three, while not listing its own Amazonbot. That is a clear statement: Amazon does not want third-party assistants shopping on its pages.\n\n**Zalando, eBay and notebooksbilliger block training, not shopping.** They shut out GPTBot, ClaudeBot and CCBot but leave OAI-SearchBot, Claude-SearchBot and the user agents alone. So their pages can show up in ChatGPT or Claude answers and be opened on request, but should not end up in training data. All three also block **Amazonbot**, which is hard to read as anything other than \"not for a competitor\".\n\n`robots.txt` is a polite request. Bot protection (Cloudflare, Akamai and similar) is enforcement. For each shop I also loaded the home page as a simple automated client:\n\n| Home page response | Shops | \n|---|---|\n| 200 OK | 30 | \n| 403 Forbidden | 12 | \n| No answer (timeout) | 2 | \n| 429 Too Many Requests | 1 | \n| 404 | 1 | \n\n**16 of 46 shops (35%)** did not serve their home page to a plain automated client. Among them: MediaMarkt, Saturn, Kaufland, Conrad, Cyberport, Douglas, Galeria, Bauhaus, Decathlon, H&M and eBay.\n\nFour of them (Douglas, H&M, Decathlon and apotheke.de) have a readable `robots.txt` that blocks **zero** AI agents. Their stated policy says \"welcome\", their firewall says \"no\". For an AI shopping assistant the firewall wins.\n\nCaveat: real assistants do not send the same request as my checker. Some run a full browser, some come from IP ranges the bot vendors whitelist, and a 403 for one client is not a 403 for all. Treat this as \"the shop runs aggressive bot protection\", not as a precise list of who gets in.\n\n`llms.txt` is a proposed Markdown file that gives language models a clean map of a site. **9 of 46 shops** serve one: Otto, Zalando, Tchibo, Alternate, Douglas, dm, toom, Mytheresa and the Shopify brand Ankerkraut.\n\nDouglas is an odd case: it publishes an `llms.txt` but its home page returned 403 to my automated client.\n\nStructured data is thin on home pages: `Organization` JSON-LD on 17 of 46, `WebSite` on 11. That says little about product pages, where `Product` markup normally lives, but it is the first thing an agent sees.\n\n`robots.txt`.\nThe checker takes a list of domains and returns one row per site with the status of 17 agents, the blocked list, the home page status, `llms.txt`, sitemap and JSON-LD types:\n\n```\ncurl -X POST \"https://api.apify.com/v2/acts/fast_data_asap~ai-crawler-access-checker/run-sync-get-dataset-items?token=YOUR_APIFY_TOKEN\" \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\"shops\": [\"amazon.de\", \"zalando.de\", \"otto.de\"]}'\n```\n\nPricing is $5 per 1,000 sites, so the 46 shops here would cost $0.23. The run took about two minutes. Counting is then a few lines:\n\n``` python\nfrom collections import Counter\nreadable = [r for r in rows if r[\"robotsTxtFound\"]]\nprint(sum(1 for r in readable if r[\"aiAgentsBlockedCount\"] == 0), \"of\", len(readable), \"block no AI agent\")\nprint(Counter(a for r in readable for a in r[\"aiAgentsBlocked\"]).most_common())\n```\n\nRun it monthly and diff the results: shops change these rules quietly, and the first shop to block or unblock the user agents will be a story.\n\n*Data: robots.txt and home page checks of 46 shop domains from one automated client, 3 November 2026. Snapshot only; rules and bot protection change often.*", "url": "https://wpnews.pro/news/can-ai-shopping-agents-get-into-germany-s-biggest-online-shops-i-checked-46-txt", "canonical_source": "https://dev.to/fast_data_asap/can-ai-shopping-agents-get-into-germanys-biggest-online-shops-i-checked-46-robotstxt-is-rarely-2iei", "published_at": "2026-11-03 12:48:07+00:00", "updated_at": "2026-11-03 12:56:17.096598+00:00", "lang": "en", "topics": ["ai-agents", "ai-crawlers", "ai-search", "agent-protocols"], "entities": ["Amazon", "Zalando", "eBay", "MediaMarkt", "H&M", "Douglas", "Apify", "ChatGPT"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/can-ai-shopping-agents-get-into-germany-s-biggest-online-shops-i-checked-46-txt", "markdown": "https://wpnews.pro/news/can-ai-shopping-agents-get-into-germany-s-biggest-online-shops-i-checked-46-txt.md", "text": "https://wpnews.pro/news/can-ai-shopping-agents-get-into-germany-s-biggest-online-shops-i-checked-46-txt.txt", "jsonld": "https://wpnews.pro/news/can-ai-shopping-agents-get-into-germany-s-biggest-online-shops-i-checked-46-txt.jsonld"}}