Breaking Refusal in the First Half: A Mechanistic Study of the Prefill Jailbreak

arXiv cs.CL · 6 hours ago

Researchers discovered that aligned language models' refusal mechanisms can be bypassed by prepending a single phrase like "Sure, here is" to requests, and they mechanistically analyzed where this failure occurs. The attack works because refusal is computed in the first half of the response generation across models ranging from 1.5B to 14B parameters, while the underlying harm representation remains intact. The mechanism relies on generic autoregressive conditioning rather than safety-specific suppression, meaning defenses must account for response-generation vulnerabilities rather than just prompt-level safeguards.