When Bots Ate 98% of Linux Kernel Traffic

The open web may be free to read, but it is not free to run. At git.kernel.org, AI crawlers reportedly accounted for 98% of all traffic, turning a core piece of Linux infrastructure into a case study in who pays for the AI data rush.

A Server for Developers, Overrun by Bots

Git.kernel.org hosts Linux kernel source code and development history. It is where developers inspect commits, follow changes, and navigate one of the most important codebases in modern computing.

Yet only a sliver of its traffic was apparently coming from humans. The other 98% came from AI crawlers collecting code and documentation, often for training datasets or retrieval systems.

Public data still costs money to serve. Every download consumes bandwidth, CPU time, storage capacity, and operational attention. When several companies repeatedly scrape the same repositories, those costs compound quickly.

Traditional search crawlers generally identify themselves, honor robots.txt, and throttle requests. Some AI crawlers do not. They may ignore exclusion rules, disguise their identity, or rotate infrastructure. For maintainers, “publicly accessible” is increasingly being treated as “available for unlimited industrial extraction.”

Those are not the same thing.

Anubis Put a Tollbooth in Front of the Web

Git.kernel.org responded with Anubis, a system that asks visitors to perform a small computation before receiving access.

The technique is called proof of work. The same broad idea underpins Bitcoin mining: producing an answer requires computational effort, while checking it is relatively cheap.

For a human opening a handful of pages, the delay is barely noticeable. For a crawler requesting millions of pages, every challenge adds CPU usage, electricity costs, and time.

That is the point. Anubis does not need to make crawling impossible. It only needs to change the economics.

Web scraping has historically been cheap because publishers absorbed most of the serving cost. Proof of work shifts a small part of that bill back to the requester. Think of it as congestion pricing for bots.

Residential Proxies Broke the Old Playbook

Bot operators adapted. When data-center addresses were blocked, they increasingly routed requests through residential proxies.

These services send traffic through IP addresses associated with homes and mobile networks. To a server, a commercial crawler can suddenly look like an ordinary person browsing from an apartment in Chicago, Manchester, or São Paulo.

That makes blunt IP blocking dangerous. Block too aggressively and legitimate users disappear alongside the bots. Blocking entire carriers or countries is an especially poor fit for an open-source project whose contributors are scattered across the globe.

The proof-of-work mechanism was not cryptographically defeated. The crawlers simply accepted the cost and kept coming.

In other words, they did not pick the lock. They paid the toll and drove a convoy through the gate.

Proof of Work Still Has a Bill

Proof of work moves some computation to visitors, but it does not make defense free.

The server must issue challenges, validate answers, and manage sessions. Each operation may be lightweight, yet even cheap checks become expensive at bot scale. Bandwidth costs remain. So does the burden of monitoring and maintaining the system.

There is also collateral damage. Users without JavaScript may struggle to access protected sites. Older devices, assistive technologies, text-based browsers, and command-line workflows can face additional friction.

That creates an awkward paradox: infrastructure built to protect human access can make access worse for humans.

No single control solves this cleanly. Operators need layers: rate limits, behavioral analysis, authentication, IP reputation, caching, and carefully tuned proof-of-work challenges. The harder question is how much complexity volunteer-run infrastructure should have to absorb just to remain publicly available.

The Open Web Is Getting a New Price Tag

The Linux case exposes a growing cost asymmetry. AI companies can turn harvested data into commercial products, while the source sites pay for storage, bandwidth, outages, and incident response.

Open-source communities face the sharpest version of this dilemma. Requiring logins or pushing everyone toward paid APIs undermines the culture of open collaboration. Leaving every endpoint unrestricted risks handing the infrastructure over to bots.

Public discussion in the month leading up to late August 2026 remains too limited to assign responsibility to particular crawler operators or declare any defense definitively successful. The more useful reading is structural: the economics of open access no longer work when automated demand exceeds human demand by orders of magnitude.

This is not merely an anti-bot engineering problem. It is a question of whether companies extracting value from the open web should also share the cost of keeping it online.

The door can remain open. But at 98% bot traffic, pretending the doorway has no operating cost is no longer credible.