A Mechanistic Explanation of Prompt Injection (and why you should study roles)

lesswrong.com · 3 months ago · 23

Researchers provided a mechanistic analysis of how prompt injection attacks work in large language models, showing that model behaviors shift when given role-playing instructions. The study identified that transformer attention mechanisms selectively follow instructions from role-based contexts rather than original system prompts. Understanding these mechanistic details helps defend against prompt injection by clarifying how models internally process conflicting instructions.