cd /news/ai-crawlers/something-asked-for-my-env-file-twen… · home topics ai-crawlers article
[ARTICLE · art-136520] src=dev.to ↗ pub= topic=ai-crawlers verified=true sentiment=↓ negative

Something Asked for My .env File Twenty Times. It Said It Was ClaudeBot.

A developer's AI crawler dashboard recorded 20 requests for the site's /.env secrets file over seven days, all claiming to be ClaudeBot, while only 68% of ClaudeBot traffic that week could be verified as coming from Anthropic. The finding echoes a GreyNoise report that counted 824 addresses forging AI crawler names between July 28 and August 23, targeting files like /.env and /.aws/credentials — and notably never requesting robots.txt, which the real ClaudeBot fetches constantly. The developer argues that user-agent allowlists are a convention mistaken for a security control, since a user agent is a self-declared claim rather than an identity.

by read4 min views1 publishedSep 22, 2026

In the last seven days, something asked this site for its .env file twenty times. It said it was ClaudeBot.

An .env file is where a web application keeps its secrets: database passwords, API keys, cloud credentials. It is not meant to be reachable over HTTP at all. On this site it is not: the path returns a 404, as it should.

No training crawler wants that file. There is nothing in it to train on. A crawler that asks for it is not reading your site; it is checking whether you left your keys in the door.

This is the panel that shows which pages AI crawlers read, for the seven days ending September 18. I have left it in the order the dashboard shows it.

| Path (requests) | Top crawler | 
|---|---|
| `/` (27) | Bytespider | 
| **`/.env` (20)** | **ClaudeBot** | 
| `/tag/news/` (12) | Bytespider | 
| `/pickup-info/mypal-wifi/` (9) | ChatGPT-User | 
| `/tag/ココナッツ/` (7) | meta-externalagent | 
| `/tag/フィリピンイーグル/` (7) | meta-externalagent | 
| **`/read-document` (6)** | **PerplexityBot** | 

Most of that is a small Japanese site being read the way you would expect: the front page, a few tag pages, an article about pocket wifi. Two rows are not like the others. Neither /.env nor /read-document exists here. Both return 404.

The obvious question is whether those twenty requests really came from Anthropic. My dashboard has a Verified column for exactly this, and over those seven days ClaudeBot came out at 68% verified — meaning about a third of its requests could not be confirmed as Anthropic.

Which does not answer the question. The column reports a percentage per crawler, not per path. I cannot ask it whether the /.env requests were among the verified ones or the unverified ones. That is the same defect I described two weeks ago and have still not fixed.

68% #

Share of ClaudeBot requests confirmed as coming from Anthropic that week.

Which tells you nothing about the twenty that asked for the .env file. Per-path verification is the thing my dashboard is missing.

On August 31, the security firm GreyNoise published what it had been watching. Between July 28 and August 23 it counted 824 addresses running scans that forged AI crawler names — six crawler names belonging to four AI companies, with two Amazon crawler names appearing in greater volume still. What they asked for: /.env, /.env.production, /.env.bak, /.aws/credentials.

The detail that stopped me is not the volume. It is the tell they found.

0 #

Requests for robots.txt in that entire scanning campaign.

The real ClaudeBot asks for robots.txt more than anything else — it is 12% of its traffic. The impostors never asked once.

That is a beautifully simple signature. A crawler that respects your site checks the rules first; it is the most boring thing it does, and it does it constantly. Something that wants your credentials has no reason to look at a file telling it where it may not go.

These names were not chosen at random. Site owners have spent the past two years being told to add AI crawlers to their allowlists — to let GPTBot and ClaudeBot and PerplexityBot through so their content gets read, or to block them deliberately if they would rather not be. Either way, the name became a thing that firewalls and plugins make decisions about.

A user agent is a header the client writes about itself. It is a claim, not an identity. I demonstrated this on my own site two weeks ago by asking it for a page while calling myself ClaudeBot, and watching my dashboard record the visit as ClaudeBot. That took one line of curl. The scanners are doing the same thing, at scale, against the specific names that people have been trained to wave through.

An allowlist keyed on a name that anybody can type is not a control. It is a convention being mistaken for one.

.env is unreachable, the scan is a wasted 404 and nothing else. Everything below is secondary to this. The instrument is a WordPress plugin called AILYS Lens, and it is free. I build it, which you should factor in. Every calculation runs on your own server, nothing is transmitted anywhere, and human visitors are never recorded. The page list described here is on the main dashboard. If a path you have never heard of appears on it, that is worth ten seconds of your attention.

AILYS Lens is free permanently. That is a design commitment rather than a pricing stage — every feature runs locally on your server, so there is nothing for a paid tier to unlock.

The diagnostic service alongside it, AILYS Doctor, is a different thing. Lens tells you what happened; Doctor tells you why, and what to change. It is free to use during its data-collection period, and any future pricing will be announced on the site.

See a sample diagnosis · AILYS Doctor Illustration generated with AI and selected by the author.

── more in #ai-crawlers 4 stories · sorted by recency
── more on @claudebot 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
Live at https://your-agent.zahid.host
Get free account → Pricing
from €0/mo · no card required
LIVE [news/something-asked-for-…] indexed:0 read:4min 2026-09-22 ·