Data Scarcity and Model Sparsity: Mixtures-of-Experts Overfit More to Repeated Data

As pre-training exhausts unique human text, repeating data has become common, but sparse Mixture-of-Experts (MoE) architectures pay a steeper price than dense models. Across models up to 8.5B total parameters, MoEs overfit significantly faster under repetition—degrading at just 4 epochs compared to 8 or more for dense baselines, an effect dictated by total rather than active parameter count. Analysis reveals that MoE routing stabilizes too early, locking experts into memorization patterns. While masking-based regularization extends viability beyond 64 epochs, no current technique fully offsets the cost of running out of fresh data.