On-Policy Delta Distillation: Transferring Reasoning Capabilities Through Teacher–Base Model Differences

This paper introduces On-Policy Delta Distillation (OPD²), an on-policy post-training method that uses the token-level difference between an instruction-tuned reasoning teacher and its pre-tuning base model as the distillation reward. Instead of directly matching the teacher’s output distribution, the delta signal is intended to isolate changes introduced by reasoning tuning and provide a more targeted training signal. The authors report consistent improvements over conventional on-policy distillation across mathematics, science, and code-reasoning benchmarks, with strong performance after a short post-training period. Code is announced for release at the linked GitHub repository.