Data Scarcity and Model Sparsity: Mixtures-of-Experts Overfit More to Repeated Data
As pre-training exhausts unique human text, repeating data has become common, but sparse Mixture-of-Experts (MoE) architectures pay a steeper price than dense models. Across models up to 8.5B total parameters, MoEs overfit significantly faster under repetition—degrading at just 4 epochs compared to 8 or more for dense baselines, an effect dictated by total rather than active parameter count. Analysis reveals that MoE routing stabilizes too early, locking experts into memorization patterns. While masking-based regularization extends viability beyond 64 epochs, no current technique fully offsets the cost of running out of fresh data.
Why it's worth reading
It delivers crucial empirical evidence that MoE architectures overfit repeated data far faster than dense models, warning training teams against relying solely on active parameter counts when planning multi-epoch runs.