Pigeonholing: how bad prompts hurt models, causing collapse and mistakes

arXiv cs.CL · 18 hours ago

Researchers identified a problem called "pigeonholing" where large language models experience performance degradation when given unhelpful context, such as incorrect math examples or buggy code, even without intentional jailbreaking attempts. Experiments across 10 models and 10 tasks showed that repeating incorrect answers from context caused 38-40% performance drops, and performance declined an additional 14% for every increase in conversation turns from 1 to 5. The team proposed RLVR with synthetic errors as a mitigation technique that improved model performance by 43-60% under bad contexts.