The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions

OpenAI · 2 years ago · 24

Researchers developed a training method that teaches language models to prioritize certain instructions over user-submitted ones, reducing vulnerability to prompt injection attacks. The technique assigns a hierarchy to instructions, with system prompts weighted to override conflicting user inputs during inference. This creates a technical safeguard that makes it harder for adversaries to manipulate model behavior through malicious prompts.