Token Geometry

arXiv cs.AI · 18 hours ago

Researchers introduced Ember, an optimizer designed specifically for embedding and language model head matrices in transformers that reduces memory requirements from O(2VD) to O(V+D) compared to Adam. Ember uses only kilobytes of optimizer state while improving performance across supervised finetuning, reinforcement learning, and pretraining tasks. The work shows that token optimization follows a simple 1D trajectory rather than a heavily nonconvex landscape and is compatible with existing distributed training frameworks.