Towards Robust Reinforcement Learning for Small-Scale Language Model Agents

This paper investigates why reinforcement-learning alignment is unstable for 70M–500M parameter language models. Across 15 model–corpus configurations using PPO, it reports three reproducible failure modes: silently frozen LoRA parameters in common PEFT/TRL pipelines, importance-ratio overflow under bfloat16, and catastrophic policy collapse caused by reward-model errors. The authors address these issues with merge-and-reinitialize adapters, float32 PPO updates, and a safety stack combining reward whitening, importance-ratio guarding, and weight rollback. They propose that PPO success depends more on supervised fluency and reward discrimination than parameter count, and report stable convergence across experiments.