AI Alignment & Behavior

2 summarised stories about AI Alignment & Behavior, each linking back to the original source. Browse all topics →

Thursday, 16 July 2026

OpenAI Details GPT-Red: An Internal Automated Red-Teaming Model That Beat Human Red-Teamers 84% To 13% On Prompt Injection

MarkTechPost 3 hours ago

OpenAI developed GPT-Red, an internal automated red-teaming model trained via self-play reinforcement learning to find prompt injection vulnerabilities in its own models. On a replicated indirect prompt injection benchmark, GPT-Red succeeded on 84% of scenarios against GPT-5.1 compared to 13% for human red-teamers, and discovered a novel attack class called Fake Chain-of-Thought that injects spoofed reasoning entries. Training GPT-5.6 against GPT-Red's attacks reduced the hardest direct injection benchmark failures to 0.05%, six times fewer failures than OpenAI's best production model four months prior.

Adversarial Testing AI Alignment & Behavior AI Security OpenAI

Pigeonholing: how bad prompts hurt models, causing collapse and mistakes

arXiv cs.CL 18 hours ago

Researchers identified a problem called "pigeonholing" where large language models experience performance degradation when given unhelpful context, such as incorrect math examples or buggy code, even without intentional jailbreaking attempts. Experiments across 10 models and 10 tasks showed that repeating incorrect answers from context caused 38-40% performance drops, and performance declined an additional 14% for every increase in conversation turns from 1 to 5. The team proposed RLVR with synthetic errors as a mitigation technique that improved model performance by 43-60% under bad contexts.

AI Alignment & Behavior AI Robustness & Failure Analysis In-Context Learning Model Training & Optimization