Identifying Interactions at Scale for LLMs
BAIR
Researchers developed SPEX and ProxySPEX, algorithms that efficiently identify influential feature interactions in large language models by leveraging sparsity and hierarchical properties to reduce computational costs from exponential to tractable levels. ProxySPEX achieves the same performance as SPEX with approximately 10 times fewer ablations required. The methods enable new applications in feature attribution, data attribution, and mechanistic interpretability across different scales of model analysis, with code made available in the SHAP-IQ repository.
Why it matters
Understanding the behavior of complex machine learning systems, particularly Large Language Models (LLMs), is a critical challenge in modern artificial intelligence. Interpretability research aims to make the decision-making process more transparent to model builders and impacted humans, a step toward safer and more trustworthy AI. To gain a comprehensive understanding, we can analyze these systems through different lenses: feature attribution, which isolates the specific input features driving a prediction (Lundberg & Lee, 2017; Ribeiro et al., 2022); data attribution, which links model behaviors to influential training examples (Koh & Liang, 2017; Ilyas et al., 2022); and mechanistic interpretability, which dissects the functions of internal components (Conmy et al., 2023; Sharkey et al., 2025). Across these perspectives, the same fundamental hurdle persists: complexity at scale. Model behavior is rarely the result of isolated components; rather, it emerges from complex dependencies
