Weak-to-Strong Generalization via Direct On-Policy Distillation

The paper proposes Direct On-Policy Distillation (Direct-OPD), a weak-to-strong transfer method for reinforcement learning with verifiable rewards. RL is first run on a smaller teacher, then the log-ratio between the teacher's post-RL and pre-RL policies becomes a dense implicit reward. This signal is evaluated on the stronger student's own on-policy states, avoiding sparse-reward RL rollouts for the target model. The reported result improves Qwen3-1.7B on AIME 2024 from 48.3% to 58.3% using 8 A100 GPUs for 4 hours. The method also supports sequential composition of multiple policy shifts.