Parcae: Doing more with fewer parameters using stable looped models

Together AI · 3 months ago

Researchers introduced Parcae, a stable looped language model architecture that reuses layers multiple times to improve parameter efficiency. A 770M parameter Parcae model matches the quality of a 1.3B parameter Transformer while achieving 6.3% lower validation perplexity than previous looped models on equivalent data. The work establishes scaling laws for looped models and enables efficient training of smaller models for edge deployment by trading parameter count for increased computation within the same layer stack.