An AGI Benchmark Contest Just Got Beaten by AI Slop

Some events matter less for what happened than for what they symbolize. A competition built to answer the question of how we measure progress toward artificial general intelligence just gave its grand prize to a submission that many people believe an AI wrote. That is the argument currently rippling through the AI community, and it is a good one.

A caveat before we go further. I went looking for the discussion threads on this — the Hacker News post-mortems, the X threads, the Reddit teardowns — and came up empty for the last 30 days. So this piece leans on structural analysis rather than quoting anyone. Check the primary sources for the specific factual details; what follows is about why this was coming regardless.

Slop Has Reached the Academy

The word slop originally meant the wet mush you feed livestock. It now means AI-generated content that is plausible on the surface and empty underneath. It started as a complaint about SEO blogspam and auto-generated YouTube. Then it hit Amazon’s Kindle store. Then Stack Overflow answers. Then open source bug reports, where maintainers started publicly begging people to stop filing hallucinated CVEs.

Research papers were always next. The fact that it landed in an AGI-measurement contest is just the universe being on the nose about it. And the scandal here is not that an AI wrote something. It is that the judging process could not tell.

Why Review Can’t Catch It

The judges were not lazy. This is structural.

Academic peer review was designed on an assumption of good faith, and more importantly on an assumption of cost. The whole system was built around the idea that a paper takes months to write. Submission was expensive, so review could afford to be expensive too. The economics balanced.

LLMs broke that balance. The cost of submitting collapsed to roughly zero. The cost of reviewing did not budge — it went up, because now reviewers have to add “is a human behind this” to their checklist.

And good slop is genuinely good, at least on the surface. The prose flows. The citations are formatted perfectly. The structure is textbook. For a review process that keys on surface-quality signals, that is the worst possible combination: the shinier the paper, the better its odds. The filter rewards exactly the thing it should be catching.

The Irony Is the Whole Point

Sit with the setup for a second. This was a contest about measuring AGI. Measuring AGI is hard precisely because it is hard to tell whether a system is reasoning or producing a convincing imitation of reasoning. And the contest’s judging process could not tell whether a submission was real research or a convincing imitation of research.

So this is both an operational screwup and an accidental answer to the contest’s own question. The answer is: we do not yet have a reliable way to distinguish imitation from the real thing. That finding is worth considerably more than the $25,000 prize.

What Would Actually Fix It

Three fixes get floated.

Raise the cost of submitting again. Entry fees, pre-registration, identity verification — deliberate friction. The problem is obvious: friction is a barrier, and barriers hit unknown-but-capable researchers hardest. You filter out slop and independent researchers in the same pass.

Demand verifiable artifacts. Reproducible code, experiments that run, public data. Text is cheap to generate; a working experiment is hard to fake. This is the realistic option, and it is the one I would bet on. It also happens to be something the ML community already half-does — NeurIPS has pushed reproducibility checklists for years. The infrastructure exists. It just is not mandatory.

Deploy AI detectors. I am skeptical. Detectors and generators are built on the same substrate, so this is a treadmill you never get off. And every false positive is a real researcher wrongly accused, which is a worse outcome than letting some slop through.

The Preview, Not the Incident

This is not one contest’s bad day. It is a trailer.

When production cost goes to zero, every system that depends on review and curation feels the same pressure. Conferences. Hiring. Open source contribution. Grant committees. Anything where a human reads a well-formatted artifact and infers effort from polish.

Which leaves the real question: how fast can we unlearn the habit of treating good writing as evidence of good thinking? That heuristic worked for centuries because writing well was expensive. It is not anymore. Reproducibility is the strongest candidate to replace it — but the honest answer is that nobody has fully worked out what comes next.