LLM-powered Biographies

Eugene Yan · 3 years ago · 39

An author tested multiple large language models including GPT-4, GPT-3.5, and Claude to generate biographies of themselves, comparing accuracy and performance across different models. GPT-4 performed best with correct general themes but factual errors in education and career details, while older models like text-davinci-003 produced entirely fabricated information such as claiming the subject was a Canadian venture capitalist. The experiment demonstrates current limitations in LLM training data coverage and memorization patterns, with models showing varying degrees of hallucination regardless of their overall capability level.