κ-LoRA: Condition Numbers Reveal Which LoRA Matrices Are Worth Updating

The paper argues that LoRA matrices do not contribute equally during adaptation. Matrices with larger condition numbers, defined as the ratio between their largest and smallest singular values, are claimed to contain richer underdeveloped directions and to drive most performance gains. κ-LoRA ranks matrices by condition number and updates only the top 50%. Across the reported benchmarks, this halves the trainable parameter count, reduces fine-tuning time by 16.2% on average, lowers memory cost by 4.5%, and matches standard LoRA accuracy. The authors also observe that selected matrices’ condition numbers decrease during training.