The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions
OpenAI Blog
Researchers developed a training method that teaches language models to prioritize certain instructions over user-submitted ones, reducing vulnerability to prompt injection attacks. The technique assigns a hierarchy to instructions, with system prompts weighted to override conflicting user inputs during inference. This creates a technical safeguard that makes it harder for adversaries to manipulate model behavior through malicious prompts.
Why it matters
Today's LLMs are susceptible to prompt injections, jailbreaks, and other attacks that allow adversaries to overwrite a model's original instructions with their own malicious prompts.
