The Specter Of Neuralese

Astral Codex Ten · 3 weeks ago · 5

AI 2027 debates raised concerns that a “neuralese recurrence” design could let models think harmful ideas without leaving detectable English chain-of-thought. The article points to Jakub Pachocki’s claim that Astra’s computation depth is within a factor of two of GPT-4. The discussion shifts from fear of unmonitored looping to a more technical argument about how recurrence changes simulated vs real layer depth and why chain-of-thought bottlenecks may still limit alignment risk, while acknowledging “true neuralese” is not yet realized.