Anyone Can Say They're ClaudeBot — And Attackers Are

Two years ago, every site operator had to make a call: block the AI crawlers or let them in. Plenty let them in. The problem is that the thing they built the allowlist around — a User-Agent string — was never designed to prove anything.

Fair warning: this isn’t a hot-take on a trending thread. I went looking for fresh community discussion and came up mostly dry. What follows is a synthesis of vendor documentation, standards work in progress, and what people running production edges keep quietly reporting. The topic holds up on its own. Ignore it now and you pay later.

How the welcome mat got rolled out

Around 2023, the calculus shifted. AI companies started scraping at scale for training data, and site operators split into two camps.

One camp slammed the door with robots.txt. The other camp opened it wider.

The open-door logic was straightforward: if ChatGPT or Claude cites your site in an answer, that’s a new referral channel. This is where the whole “generative engine optimization” pitch came from — GEO as the successor to SEO, with publishers racing to be the source an assistant quotes. So sites added ClaudeBot, GPTBot, and PerplexityBot to their firewall exceptions. Some went further and exempted them from rate limits entirely, because crawlers crawl a lot and nobody wants to throttle their own distribution.

Here’s the part that matters: what exactly got allowlisted? In most configurations, a single line of text in an HTTP request header.

A User-Agent is an introduction, not an ID

Quick primer for anyone who doesn’t spend their days in nginx configs.

When your browser hits a website, it introduces itself. Something like “I’m Chrome 130 on Windows.” That’s the User-Agent header. It’s a string. Nothing signs it. Nothing verifies it. There is no issuing authority, no revocation, no cryptography of any kind.

One curl flag and you’re ClaudeBot. That’s not a simplification — it’s literally one flag. The technical barrier is zero.

Picture a building lobby where saying “delivery” gets you waved through. No uniform check, no badge, no clipboard. Just the words. Now imagine the building also posted a sign reading “delivery drivers skip the line.” That’s the current state of a lot of production edge configs.

Why this is a bargain for attackers

Anyone who’s run a vulnerability scanner knows the two enemies of mass scanning: rate limits and outright blocking. Hammer weird paths from one IP a few dozen times per second and the WAF cuts you off.

Dressing up as an AI crawler dissolves a good chunk of that.

Start with the rate limits. If the User-Agent is on the allowlist, the throttle often doesn’t apply at all. Operators exempted crawlers on purpose, because crawlers are supposed to be noisy.

Then there’s log camouflage. When an access log shows a few thousand ClaudeBot lines, the reflexive read is “crawler came through.” Nobody scrolls. That’s true even when the requested paths include /wp-admin/, /.env, /.git/config, and /actuator/health — the standard scanner shopping list.

And the human layer is soft too. “GPTBot traffic is a bit heavy this week” lands very differently in a security channel than “unidentified scanner hitting us.” One gets triaged next sprint. The other pages someone.

One spoofed header buys detection evasion and block evasion at the same time. Effort-to-payoff, it’s hard to beat.

How you actually verify a crawler

The good news: the methods already exist. The bad news: not enough people are using them.

Step one — check the IP. The major AI companies publish the IP ranges their crawlers operate from. Anthropic documents theirs, OpenAI documents theirs, and several publish machine-readable JSON you can pull on a schedule. If the User-Agent says ClaudeBot and the source IP isn’t on the list, it’s fake. This catches most of it.

Step two — reverse DNS. Google has used this for Googlebot verification for years. Take the connecting IP, do a reverse lookup to get a hostname, then forward-resolve that hostname and confirm it returns the original IP. That’s forward-confirmed rDNS. It’s useful when a provider’s IP ranges shift often enough that list-syncing becomes a chore.

Step three — Web Bot Auth. This is the real fix. It’s a specification moving through the IETF that has bots sign their requests with a private key, built on top of HTTP Message Signatures.

The bot publishes its public key at a well-known location and attaches a signature header to every request. Your server verifies the signature against that key. Without the private key, you can’t forge the signature — so User-Agent spoofing simply stops being a viable move.

The operational win is that you stop maintaining IP lists. A crawler can move data centers, switch CDNs, or spin up new egress ranges, and the signature still validates. Cloudflare has been pushing this hard, and bot operators are adopting it one at a time.

Two things to check today

There’s no reason to wait for the standard to land. Two concrete tasks.

Your config. Whether it’s nginx, Cloudflare, or something else — go find the rules that let traffic through based on a User-Agent string alone. Add IP verification, or just delete the exception. The case for giving AI crawlers a rate-limit exemption was always thinner than it looked.

Your logs. Pull the last few months of access logs, filter for AI bot User-Agents, and cross-reference the source IPs against the official published ranges. Then look at what the non-matching requests actually asked for. If you see .env, .git, or admin paths, this already happened to you.

Neither task is hard. Most organizations skip them not because they’re difficult but because nobody told them the impersonation was possible.

The underlying problem is proof

AI crawler spoofing isn’t a new vulnerability. It’s an old habit — treating the User-Agent header as identification — that got dangerous when the AI boom gave it something valuable to unlock.

The web spent two years opening its doors to AI. Building the gatehouse got pushed to the backlog. Until Web Bot Auth or something like it is broadly deployed, that gap belongs to whoever wants it.

So open your logs. Two questions: who is your server letting in right now, and do you have any way to confirm they are who they claim to be? If either answer stalls, you’ve found where to start.