Fable 5 Vs Opus 4.8: Outcomes-Based Assessments Are A Massive Warning For Frontier AI Labs

There's a widening gap between what AI benchmarks measure and what businesses actually need, and a recent test involving Fable 5 and Opus 4.8 makes that gap impossible to ignore. Both models were given the same template, an old website, and one clear objective: rebuild the site to improve conversions. Judged as coding artifacts, the resulting pages at highroiai.com and datascience.vin look sharp and capable. Judged against the actual business outcome, they both scored zero.

The reason is almost embarrassingly basic. Neither model thought to build in any tracking or event-logging hooks, the kind of instrumentation you'd need just to know whether conversions improved at all. That's web design fundamentals, not some exotic ask. Both sites also arrived without sitemaps or SEO metadata until specifically told to include them, and both shipped with no real security. Fable 5 at one point refused to fix a security issue and quietly fell back on Opus 4.8 to handle it.

Here's the twist: once you spell out that much context and that many specific requirements, a much smaller model does just as well. Gemma 4, running locally on a Dell Pro Precision machine, produced a comparably incomplete website when fed the same instructions the frontier models needed. It still required some manual fixes, sure. But it cost nothing in tokens and never sent data outside a firewall. The advanced reasoning frontier labs charge a premium for stops mattering much once you've done the work of defining outcomes yourself.

That pattern shows up at the enterprise level too. Eli Lilly, often cited as a pharma company with a serious AI stack, signed with OpenAI back in 2024 and hasn't expanded that partnership since. Instead it's leaned into smaller, purpose-built models trained on its own data and picked up partnerships with domain-specific AI providers. Meanwhile Palo Alto Networks CEO Nikesh Arora went on CNBC claiming AI demand is "infinite," while in the same breath admitting most workloads will actually run on smaller, open-source models rather than frontier ones.

The loudest pushback came from Palantir's Alex Karp, whose CNBC interview went viral for asking the obvious question labs would rather dodge: if a model could genuinely make a company a billion dollars, why would anyone sell that capability by the token instead of taking a cut of the outcome? Agent-builder platforms from the big software vendors face the same challenge — sold as the layer that delivers business results, priced like subscriptions instead. Companies actually being named as better examples of AI monetization — JPMC, Siemens, Walmart, Tencent, BMW, Hyundai among them — treat the AI itself as secondary to whatever non-AI outcome their business already runs on.