Weight-Adjusted Gradients Reveal Parameter Importance and Failure Modes in LLMs

The paper introduces Weight-Adjusted Gradients (WAG), an importance metric that combines model weights with first-order gradients to identify parameters exerting disproportionate influence on LLM behavior. According to the abstract, WAG exposes a very small set of critical parameters whose modification can cause dramatic performance degradation, including parameters associated with collapse phenomena that existing metrics may miss. The authors apply the method to expert allocation in mixture-of-experts models, parameter-specific unlearning, mixed-precision quantization, and layer selection for knowledge editing. The work frames parameter importance as an interaction between zeroth-order weight information and first-order gradient information.