Pruning LLMs Like a Physicist: Block Removal as an Ising Optimization Problem

Hugging Face’s latest post takes a familiar compression trick and gives it a physics engine. Instead of ranking transformer blocks one by one, the team turns block removal into a constrained binary optimization problem and maps it to an Ising glass, where each block is either kept or removed and the interactions between choices matter.

That matters because block pruning is not a solo act. Removing block 20 can be harmless or disastrous depending on whether block 19 or block 24 is also gone, so the usual heuristics — magnitude, sensitivity, block influence — miss the couplings. The paper argues that this is exactly the kind of many-body problem statistical physics was made for.

The setup is neat. A second-order Taylor expansion of the loss gives a Hessian, with diagonal terms capturing how much each block matters alone and off-diagonal terms capturing pairwise interactions. Once that matrix is built from a small calibration set, candidate pruning layouts are cheap to score, and the same Hessian can be reused for different compression targets. For harder cases, the problem can be passed to classical, quantum, or quantum-inspired solvers in QUBO form.

The results are strongest when the pruning gets aggressive. On Llama-3.3-70B-Instruct, removing 32 of 80 blocks keeps MMLU at 76.6, while the block-influence baseline gets 59.3. At 40 of 80 blocks, CBO still lands at 76.9 on MMLU, with the baseline at 54.0. On Qwen3-14B, removing 12 of 40 blocks gives about a 10-point MMLU edge. And on Llama-3.1-8B-Instruct, the best answer was not the ground state at all, but an excited state that retrained better than the obvious pick.

The method also works on mixed architectures, not just neat stacks of identical transformer blocks. On NVIDIA-Nemotron-3-Nano-30B-A3B-FP8, which mixes Mamba2, attention, and MoE layers, the same approach found better cuts than block influence on AIME25 and GPQA without retraining. The code is open-sourced, which is the right move; if a method claims to make pruning more principled, people should be able to try it without filing a prayer request.