Developers Stopped Trusting Google and Started Indexing the Web Themselves

“I know I read this somewhere. I just can’t find it again.”

That complaint shows up constantly in developer forums now. A blog post from six months ago. An issue comment. One line in a doc. The memory is vivid, but no combination of keywords brings it back. What comes back instead is twenty AI-generated summaries with plausible titles. So a handful of people gave up on searching the whole internet and started building indexes that cover only what they have already read.

The thing that broke is re-finding, not finding

The complaint isn’t about discovery. Discovery still works well enough. What broke is re-finding — pulling back something you have already seen.

Information retrieval researchers have studied this for decades, and their estimates put a substantial share of all search activity in this category. Re-finding is uniquely fragile because what people remember is not a title but a texture. “It was about Linux kernel scheduling, it had one chart, and the author was kind of snide about it.” That memory does not convert into a query string.

It used to work anyway, because the web was smaller. Type “kernel scheduler latency benchmark” and the post landed in the top three. Today the same query returns a page that summarizes that post, plus a page summarizing the summary, plus a summary of that. The original sits five pages down, or it’s gone.

AI slop is a new kind of background noise

The term AI slop has settled into common usage: mass-produced generative content that is neither right nor wrong, just vacant.

What makes it different from spam is that it doesn’t look like spam. Old spam was obvious — unnatural keyword repetition, broken grammar, weird link patterns. Search engines learned to filter it. AI slop has flawless grammar. Clean structure. Well-formed subheadings. Tidy bulleted lists. It is excellent writing by machine standards and empty by human ones.

For a ranking algorithm, this is the worst possible outcome, because the quality signals it relied on have all been neutralized. Sentence length. Structural completeness. Topical coherence. A model does all of these better than most humans. So the top of the results page erodes gradually — not overnight, but over months.

Layer on top of that the web closing itself off. Content owners started blocking crawlers to keep their work out of training sets. Those blocks don’t discriminate. Archiving bots, small independent search engines, and personal tools get shut out along with the training crawlers. The share of source documents reachable on the open web keeps shrinking.

The alternative: an index of what you actually read

This is the context that produced personal full-text search indexes. Tools like Hister get cited as representative examples, and the idea is refreshingly simple.

A browser extension saves the full body text of every page you visit into a local database. That’s the divergence from your browser’s built-in history, which keeps only URLs and titles. Because you hold the text itself, you get real full-text search. You can search for the sentence you remember from the article. The server runs on your own machine or homelab, so nothing leaves the building.

The signal-to-noise ratio isn’t close. Every document in the index is one you clicked and read. It already passed a human filter once. There is almost no surface for AI slop to attach to. A problem of picking good results out of a million documents becomes a problem of finding one thing among five thousand. The second problem is far easier.

Content surviving link rot matters more than people expect. Domains expire. Companies fold. Blogging platforms shut down. A local copy means the page stays readable after the original disappears.

And nothing in the results is bought. How much someone spent on SEO does not affect the ordering. It’s text matching and recency, and that’s it.

So why isn’t everyone doing this

It’s a romantic idea, and the people who have actually run it have real objections.

Start with storage. Text alone is smaller than you’d guess — read 100 pages a day and you’re looking at a few hundred megabytes a year. Full snapshots with images and CSS are a different story: several megabytes per page. This is where most tools compromise. Keep the body text, discard the rest.

Security is the harder problem. Saving every page means saving your bank balance screen, your company’s internal wiki, your medical records. If that database leaks, the blast radius is nothing like a browser history leak. Domain blocklists, incognito exclusions, and local encryption become mandatory — and whether they’re configured correctly is ultimately on the user.

Operational burden is the last one. Self-hosting means backups, updates, and outage response are yours. There’s a recurring, well-earned bit of cynicism in these threads: it gets neglected after a couple of years, and one day the disk dies and takes the archive with it. Personal wikis, bookmark systems, and note apps have died exactly this way, over and over.

The real question is who owns the index

For twenty years we outsourced memory. There was no reason to save anything, because searching found it. A thin browser history didn’t matter. Google was effectively our external storage.

That assumption is wobbling. The web got bigger, low-quality content multiplied, crawling got blocked, and the business model tilted toward ads and AI-generated answers. There is no longer any guarantee that a shared public index stays tuned to what you personally need.

A personal index is a decision to build that guarantee yourself. You trade scale for control. You can’t search the whole internet, but you can always retrieve what you have read.

The interesting part is that this requires no hostility toward AI. Quite the opposite. Bolt retrieval-augmented generation onto a personal index and it becomes genuinely useful, because you’re querying a small corpus of documents you already trust. The archive you built because of AI slop turns out to be the best possible input for AI.

What to take from this

Public discussion around this specific trend has been thin. There wasn’t enough conversation over the past 30 days to draw conclusions from, so treat everything above as structural context and long-running arguments rather than a read on current sentiment. Real-world reactions to any particular tool, and hard numbers on traction, aren’t in here.

The direction is clear enough regardless. The feeling that search has gotten worse isn’t personal imagination — it’s several structural shifts converging. The response taking shape is small indexes under individual control.

Starting doesn’t require much. Save the text of one good article you read today. That post from six months ago — can you find it right now?