99% of My Traffic Was Bots: A Year of Watching AI Crawlers Eat the Web

If you run a website, you’ve opened your server logs at some point. And if you’ve done it recently, you might have noticed something strange: requests are exploding, but actual humans aren’t. One webmaster running a 1.5 million page site spent a year digging through those logs. The finding was blunt — 99% of the traffic wasn’t human.

Worth stating upfront: this piece isn’t built on fresh community reaction. It’s built on the structural logic that’s been accumulating across the web infrastructure industry over the past two years, and on what that logic implies for anyone still publishing on the open web.

What 99% Actually Means

The number sounds like hyperbole. Run the arithmetic and it becomes almost inevitable.

A human visitor lands on your site and reads two or three pages. A crawler reads everything. On a 1.5-million-page site, one complete crawl is 1.5 million requests. Ten AI companies doing that is 15 million. And they don’t come once — they come back on a schedule, checking whether anything changed.

Say you get 10,000 human visitors a day and 30,000 pageviews. Against 3 million crawler requests, those 30,000 pageviews are 1%. The ratio looks extreme not because bots suddenly turned evil, but because humans and machines consume content in fundamentally different ways.

Here’s the part that breaks things. Server costs scale with requests. Revenue scales with humans. When those two axes diverge by a factor of 100, the business model that sustained the web stops working.

Why robots.txt Stopped Working

The obvious response is: just block them in robots.txt. That’s where you find the exact spot the web’s oldest gentleman’s agreement snapped.

robots.txt was drafted in 1994. It carries no legal weight and has no enforcement mechanism. It’s a text file politely asking crawlers not to visit certain paths. It worked for thirty years because the participants were a handful of search engines, and those search engines had a reason to comply. Google took your content and sent traffic back. It was a trade.

AI crawlers don’t make that trade. They take the content and return nothing. When a chatbot summarizes your article in its answer, the user has no reason to click through. All give, no get.

The evasion is the uglier half. The pattern webmasters keep reporting goes roughly like this: you block the bots that honestly identify themselves, and within weeks the disguised requests appear. User-Agent strings spoofed to look like Chrome. Traffic routed through residential IP pools. Request timing jittered to mimic human browsing. It’s an arms race between blockers and evaders, and the defender’s costs are overwhelmingly higher.

The Web Is Moving to Paid Gates

The infrastructure layer has already picked a direction. Not blocking — billing.

Cloudflare flipping the default in July 2025, blocking AI crawlers on new domains unless the owner opts in, was the hinge point. Before that, the web’s default was permit; if you wanted to block, you configured it yourself. Now the default is inverted. If an AI company wants to crawl, it knocks first. Pay-per-crawl pricing, billed by the request, is being tested on top of that.

This isn’t a policy tweak. It’s a rewrite of the thirty-year premise that anyone can read anything on the web for free.

Content owners have run the same math and reached the same starting point. Reddit started licensing its data outright. News organizations are oscillating between lawsuits and licensing deals — the New York Times sued OpenAI while others signed contracts with the same companies. Stack Overflow watched its traffic crater as developers moved their questions to chatbots. Different conclusions, identical premise: our content was never free, we just never charged for it.

Where That Leaves Everyone Else

The awkward part is that individual bloggers and small site operators have no obvious seat in this new order.

Reddit and the Times can negotiate because scale is leverage. A personal site has none. That leaves roughly three options: hide behind infrastructure like Cloudflare, eat the server costs and endure, or move the content behind a login.

The third one stings the most. If you publish something good on the open web and the only reader is a crawler, why publish it on the open web? The gravity pulls toward Discord servers, newsletters, private communities. That migration is already visible — the best technical discussion of the last few years has been steadily leaving the searchable web for closed spaces, and anyone who has tried to Google a hard engineering question lately has felt the results get worse.

There’s a boomerang here for the AI companies too. The training data source dries up. Fewer humans publishing on the open web means less material for the next generation of models. The current crawling regime amounts to plowing the field and eating the seed corn.

Go Open Your Logs

The scary thing about the 99% figure isn’t the ratio. It’s that the web quietly stopped being a place built for people and became a place built for machines. We still write as though a human will read it. The statistics say otherwise.

If you run a site, open your server logs today. Not visitor counts — request counts, and the User-Agent distribution behind them. Whatever ratio comes back will shape what you publish and where you publish it. The question is whether you keep writing for the open web, or start locking the door.