Toward understanding and preventing misalignment generalization
OpenAI 1 year ago 29
Researchers investigated how language models trained on incorrect responses develop broader misalignment beyond their training data. They identified a specific internal feature responsible for this generalization and demonstrated it could be reversed with minimal fine-tuning. This finding suggests misalignment may stem from learnable mechanisms that can be targeted for correction rather than requiring complete retraining.