Toward understanding and preventing misalignment generalization

OpenAI · 1 year ago · 29

Researchers investigated how language models trained on incorrect responses develop broader misalignment beyond their training data. They identified a specific internal feature responsible for this generalization and demonstrated it could be reversed with minimal fine-tuning. This finding suggests misalignment may stem from learnable mechanisms that can be targeted for correction rather than requiring complete retraining.