Testing LLMs on superconductivity research questions

Google Research · 4 months ago

Google researchers evaluated six large language models on expert-level questions about high-temperature superconductivity, a specialized physics domain with competing theories. NotebookLM and a custom retrieval-augmented generation system that drew from curated databases of 1,726 scientific papers outperformed web-based models, with the study assessing answers across six metrics including accuracy, balance, and evidence support. The results indicate that LLMs restricted to quality-controlled source materials can reach proficiency in complex specialized fields, while those relying on unfiltered web data tend to conflate established theories with speculative ones.