EPnG: Adaptive Expert Prune-and-Grow for Parameter-Efficient MoE Fine-tuning

EPnG adapts LoRA to the routing structure of mixture-of-experts models. It estimates expert importance from router gate probabilities, prunes under-utilized experts, and grows the rank of important experts with orthogonal initialization while keeping a fixed parameter budget. On OLMoE and Qwen1.5-MoE, the paper reports updating only 0.55%–0.72% of model parameters, or up to 140x–180x fewer than full fine-tuning, while outperforming LoRA under the same budget and approaching full fine-tuning performance. The supplied abstract does not provide detailed task, dataset, or variance information.