Beyond LoRA: Can you beat the most popular fine-tuning technique?
Hugging Face Blog
Hugging Face benchmarked over 40 parameter-efficient fine-tuning techniques to compare their performance against LoRA, which dominates 98.4% of fine-tuning implementations on their hub. On mathematical reasoning tasks, LoRA achieved 53.2% accuracy using 22.6 GB of memory, while on image generation, the OFT technique scored 0.708 similarity versus LoRA's 0.697 with lower memory (9.01 GB vs 9.97 GB). Users should evaluate multiple PEFT techniques on their own datasets rather than defaulting to LoRA, as different methods offer better tradeoffs depending on whether accuracy or memory efficiency is prioritized.
