Parcae: Doing more with fewer parameters using stable looped models
Together AI
Researchers introduced Parcae, a stable looped language model architecture that reuses layers multiple times to improve parameter efficiency. A 770M parameter Parcae model matches the quality of a 1.3B parameter Transformer while achieving 6.3% lower validation perplexity than previous looped models on equivalent data. The work establishes scaling laws for looped models and enables efficient training of smaller models for edge deployment by trading parameter count for increased computation within the same layer stack.
Why it matters
Parcae is a stable looped language model that matches the quality of a Transformer twice its size — a 770M model reaching 1.3B-level performance. We introduce the first scaling laws for looping and show that increasing recurrence, not just data, is a compute-efficient path to bet
