Does Topic Sentiment Cause Perceived Ideology? Comparing Human and LLM Annotations in Political News Articles

arXiv cs.CL · 18 hours ago

Researchers compared how topic sentiment affects perceived political ideology in news articles, analyzing labels from human experts and large language models including GPT-4o-mini and Llama-3.3-70B using causal inference methods. Zero-shot LLMs inflated effect sizes compared to human annotations by an unspecified margin, though fine-tuning reduced this gap toward human-scale estimates. The findings suggest LLM annotations can identify whether sentiment affects ideology perception but may misestimate the magnitude of effects, affecting predictions of ideology from topics.