🔮 Is AI immune to groupthink?

Rohit Krishnan, the researcher behind Strange Loop Cannon, decided to test something a lot of AI power-users take on faith: that stacking multiple language models into a council produces smarter answers than asking one model alone. Inspired by Andrej Karpathy's LLM Council setup and prior work like MarketBench, Krishnan built three flavors of committee — a simple blend where a fourth model merges everyone's answers, a peer-review setup where models critique each other before a chair summarizes, and a blunt "pick the best single answer" selector.

He then ran sixteen open-ended prompts, split evenly between strategy problems and writing tasks, and broke every answer into small idea "cards" using Claude's Sonnet model. Cards that clustered together across multiple models got tagged as shared ideas; cards that showed up in only one model's answer were tagged solo. Two blind judges then rated which of these clusters were actually good — useful, non-obvious, worth keeping.

The results echo something human committees have always struggled with. The blended council preserved only about a quarter of the good solo ideas that two judges had flagged as valuable — meaning roughly three-quarters of genuinely smart, single-model insights simply vanished once models got merged. Peer review didn't fix this: rare good ideas survived at almost the same rate, 24% versus 22% for plain blending. But when multiple models independently raised the same idea, peer review kept it about a third of the time, compared to a quarter for solo ideas — a modest but real bias toward consensus over originality.

Krishnan points to Stasser and Titus's 1980s research on "biased sampling of shared information," the finding that human groups gravitate toward what several members already know rather than surfacing what only one person has spotted — the so-called hidden-profile problem. His experiment suggests LLM councils inherit the same blind spot, even though nobody programmed it in. Multi-agent debate research has generally shown councils improve average answer quality, and Krishnan doesn't dispute that. His point is narrower and sharper: better on average doesn't mean better everywhere, and the specific ideas getting quietly dropped are often the sharpest ones — a status-symbol detail about scent cartridges in a squatted mall, a warning that logged-but-ignored risks are scarier than unknown ones, a clever crowdsourced data-recovery trick.

His fix isn't elegant, but it's concrete: explicitly extract, store, and rank the best ideas from each model's solo answer before any final synthesis happens, rather than trusting the merging process to preserve them automatically. It's slower and heavier, he admits, and he's not convinced it's optimal. But the core lesson is that council structure isn't a solved problem you can borrow from someone else's setup — it needs testing against your specific task, because the failure mode here is silent. You lose the good stuff and never notice it's gone.