Are AI labs pelicanmaxxing?

Simon Willison's habit of asking every new model to draw a pelican riding a bicycle has turned into one of the internet's favorite unofficial AI benchmarks, the kind of thing that lights up Hacker News threads the moment a lab drops a new release. Naturally, that popularity raises a suspicion: with so much money riding on looking good in public, would a lab quietly train on exactly this prompt to juice the result? Dylan Castillo decided to actually check, rather than just argue about it in comments.

The setup was straightforward but not tiny. Castillo built a grid of 8 animals crossed with 6 vehicles, including the original pelican-and-bicycle cell, and generated three samples per combination across seven frontier models — GPT-5.6 Terra, Claude Sonnet 5, Gemini 3.5 Flash, Grok 4.5, Qwen3.7-Max, GLM-5.2, and DeepSeek V4 Pro. That's 1,008 SVGs total, rendered to PNGs and scored by GPT-5.6 Luna on three 1-to-5 scales: how well it drew the animal, how well it drew the vehicle, and how coherent the whole scene looked. A second model, Gemini 3.1 Flash-Lite, tagged each image for which way the subject faced and what extra scene elements showed up.

The results don't support the pelicanmaxxing theory. Pooled across all seven labs, pelicans ranked sixth out of eight animals, trailing cats, whales, raccoons, and herons. Bicycles fared worse still, landing second-to-last among vehicles, just ahead of planes. Put the two together and the pelican-on-bicycle cell sits at 42nd out of 48 possible combinations — nowhere near the top spot you'd expect if a lab had specifically trained on it.

Castillo went further and ran a regression that adjusts for how inherently hard each animal-vehicle pairing is to draw, since a bicycle with two matched wheels and a frame is objectively fussier than a skateboard. Even after that adjustment, every lab's pelican-specific boost stayed statistically insignificant, and the same held for the exact pelican-bicycle cell. The one number that did cross the p<0.05 threshold was Gemini 3.5 Flash getting slightly better at bicycles generally — but with 21 separate tests run, that's roughly the one false positive chance alone predicts, and it doesn't survive a stricter multiple-comparisons correction. GLM-5.2 showed the largest (still insignificant) bump on the pelican-bicycle cell specifically, which lines up with Castillo noticing a genuinely nice sample from that model during manual review.

The closest thing to a real pattern was directional, not qualitative: all 21 pelican-bicycle images across every lab faced right. That sounds damning until you notice 60% of all 1,008 images in the entire dataset face right, and three other animal-vehicle combos landed at 90% or higher agreement too. Scene elements told a similar story — flamingos on boats always got a sun, otters on planes got scarves over a third of the time — but pelicans on bicycles showed nothing unusually memorized compared to any other pairing. Castillo's own conclusion is that if labs are gaming anything, it's probably broader SVG-drawing ability as a whole, something Google/DeepMind has said outright it optimizes for, rather than this one viral prompt specifically.