Generalized Fisher-Weighted SVD: Scalable Kronecker-Factored Fisher Approximation for Compressing Large Language Models

arXiv cs.AI · 6 hours ago

Researchers proposed Generalized Fisher-Weighted SVD, a compression technique for large language models that uses Kronecker-factored approximations of the Fisher information matrix to account for parameter correlations beyond diagonal approximations. At 20x compression on MMLU, the method achieved 5 percentage point improvements over FWSVD and 6 percentage point improvements over ASVD. This approach enables more accurate parameter importance estimation during post-training compression, resulting in better downstream task performance than existing methods.