Defending against Prompt Injection with Structured Queries (StruQ) and Preference Optimization (SecAlign)

BAIR · 1 year ago

Researchers propose StruQ and SecAlign, two fine-tuning methods to defend LLMs against prompt injection attacks where untrusted data contains instructions that override legitimate system prompts. SecAlign reduces the attack success rate of optimization-based attacks to below 15%, a 4x improvement over previous approaches, while maintaining utility on general benchmarks. These defenses use special delimiter tokens to separate trusted instructions from untrusted data and train models to ignore injected instructions in the data portion.