If It Can't Draw a Pelican on a Bike, It's Out

A new model drops and the benchmark table goes up within minutes. Nobody reads it. What actually happens is that a developer opens a chat window and types “draw me a pelican riding a bicycle, in SVG.” It sounds like a bit. It is currently the most widely used unofficial evaluation in AI.

Why a pelican, and why a bicycle

The test got famous because Simon Willison ran the same prompt against every new model release and posted the results on his blog. Andrej Karpathy picked it up, and that pushed it further. Now every launch day fills timelines with lopsided birds on lopsided bikes.

The pairing is deliberately mean. A pelican has absurd proportions — the bill is roughly half the body, the neck is short, the legs sit way back. A bicycle is the opposite: rigidly structured. Two wheels, a triangular frame, pedals, handlebars. Move any one of those and it stops reading as a bicycle. The prompt asks a model to balance a ridiculous animal on top of a strict machine.

And SVG is not image generation. The model isn’t drawing. It’s emitting coordinates and curve commands as text, with no canvas to glance at and no chance to correct course. Every shape has to be positioned mentally, then written out as code. The human equivalent is drafting a blueprint with your eyes closed.

That’s why the failures are so entertaining. Three-wheeled bicycles. Pelicans floating a hundred units above the frame. Bills that skewer the front wheel. The community named one specific failure mode the Habsburg jaw — after the famously inbred royal family’s protruding chin — for the versions where the model loses control of the bill length and the bird ends up with a comically jutting face.

What the official scores stopped telling us

Be honest: the standard metrics have gone quiet. The top models are bunched so tightly that a one- or two-point gap tells you nothing about which one to use. And contamination keeps coming up — the real possibility that benchmark questions leaked into training data. It’s the same problem as trusting a test score from a student who saw the exam in advance.

The pelican test sidesteps all of it, because there’s no answer key. No rubric, no leaderboard. You just look. Does it look like a bicycle. Does it look like a pelican. Is the pelican on the bicycle. Humans resolve that in half a second.

It’s also a bundled test. Spatial reasoning, structural understanding, code generation, instruction following — all at once, and being good at one won’t carry you. That’s exactly what benchmarks with neat per-category scores miss. Real work doesn’t arrive pre-sorted by capability.

Everyone has their own private exam

The pelican is famous, not unique. Most developers have one test they reach for by muscle memory.

Some ask for a frog — awkward limb placement, a weird crouch, genuinely hard to get right in SVG. Some paste in the actual bug they burned three hours on last week. Some ask about the documentation for an obscure, unmaintained library to see whether the model admits it doesn’t know. Some hand over deliberately vague requirements and watch whether it asks a clarifying question or just invents the missing details.

The common thread: none of these have ever been published. They live on one laptop, in one person’s head. There is no path for them into a training set. It turns out the most reliable defense against benchmark contamination is writing your own questions and telling nobody.

So is this actually evaluation

The limits are real. Sample size of one. Run the same prompt five times and you get five different birds — and most people post the best one. There’s also the reflexive problem: the moment a test gets famous, labs may start optimizing for it. A benchmark that becomes a target stops being a measurement.

More fundamentally, drawing a decent bird in SVG doesn’t mean a model reviews code well. There’s no evidence the two correlate.

It survives anyway because it answers a question the official scores don’t. Developers don’t actually want to know what a model scored on GPQA. They want to know whether they can use it for their work. There’s no way to learn that except by giving it a few things to do. The pelican is just the fastest, cheapest version of that gut check.

It’s not a benchmark, it’s a filter

The pelican test doesn’t measure capability. It exposes where a model decisively fails. A good drawing tells you almost nothing. A catastrophic one tells you something specific and reliable: this model cannot arrange shapes in its head.

That’s screening, not scoring. It isn’t there to set a passing grade — it’s there to eliminate candidates. And screening tools only stay in rotation if they’re short, cheap, and a little bit fun. The pelican is all three.

That’s also why private tests took off exactly as the public scores converged. When everything looks equivalent on paper, the only option left is checking with your own eyes. So what’s the first thing you throw at a new model? If an answer came to mind immediately, you already have a benchmark.