DIVE: Embedding Compression via Self-Limiting Gradient Updates

arXiv cs.AI · 18 hours ago

Researchers introduced DIVE, a compression method that reduces the size of language model embeddings using a residual adapter combined with self-limiting gradient updates and geometry distillation. DIVE achieved the strongest performance across all five BEIR benchmarks when tested at 128d and 256d output dimensions against six baseline methods. The technique enables cheaper storage and faster search of embeddings without the overfitting problems that plague supervised compression when training labels are limited.