β-OPSD: Deriving with Policy Optimization, Training with Self-Distillation
AI Summary
The paper argues that on-policy self-distillation (OPSD) is exactly the β=1 case of a broader policy-optimization objective with a KL penalty anchoring the student to a reference policy. It introduces β-OPSD, where β controls the trade-off between reference-policy proximity and privileged teacher guidance. The method uses token-level logit mixing to construct a distillation target corresponding to the closed-form optimal policy, avoiding the cost and variance of direct reinforcement-learning optimization. Return-to-go credit assignment aligns token updates with sequence-level rewards. Experiments on mathematical reasoning benchmarks report improved optimization stability and downstream reasoning performance over vanilla OPSD.
Why it's worth reading
OPSD is increasingly used for reasoning-model training, and β-OPSD turns its fixed regularization into a controllable parameter. The key question now is whether the reported stability gains generalize beyond the paper’s mathematical-reasoning settings.
Deep Read
1. What happened
Original facts: The paper introduces β-OPSD and frames vanilla OPSD as the β=1 member of a broader policy-optimization family. The authors report that β-OPSD improves optimization stability and downstream reasoning performance over vanilla OPSD on mathematical reasoning benchmarks.
2. Core technology
Original facts: β weights the KL penalty that keeps the student close to a reference policy, controlling the trade-off between reference-policy proximity and privileged teacher guidance. The paper derives an optimal policy as a geometric interpolation between the reference and teacher policies, then constructs a distillation target by mixing their token-level logits.
Analysis: This converts an expensive policy-optimization problem into an inexpensive supervised-distillation approximation, while making β an explicit control over update strength and teacher influence.
3. Key evidence and numbers
Original facts: The abstract specifies that β=1 recovers vanilla OPSD and states that β-OPSD “consistently outperforms” it in mathematical-reasoning experiments, with gains in stability and downstream performance. It does not provide benchmark names, model sizes, numerical improvements, or the number of runs.
4. Why it matters
Analysis: OPSD can be difficult to engineer because teacher signals may destabilize training or move the student too far from its reference policy. If β reliably controls these effects, it turns an implicit fixed design choice into a tunable hyperparameter and clarifies the connection between self-distillation and policy optimization.
5. Practical impact
Original facts: Instead of directly optimizing a costly, high-variance RL objective, the method uses a distillation target derived from the closed-form solution. It also applies return-to-go credit assignment so token updates better reflect sequence-level objectives.
Analysis: Existing OPSD pipelines may be able to adopt the method by adding teacher/reference logit mixing, β tuning or scheduling, and return-weighted token updates, without replacing the entire system with online RL.
6. Limitations and uncertainty
Original facts: The available information is limited to the abstract. It does not establish the useful β range, added memory or compute cost, how the privileged teacher is obtained, or performance on coding, knowledge, or long-horizon planning tasks. Ablations over β, teacher quality, and reference-policy selection are also not available here.
Unverified inference: The effectiveness of token-level logit interpolation may depend on calibration, shared vocabulary, and temperature choices between the teacher and reference policy. These factors require verification from the full paper or reproduction.
7. Original sources
- arXiv abstract page
- Paper identifier:
arXiv:2607.28582
Statements marked “Original facts” come from the supplied paper abstract. “Analysis” and “Unverified inference” are interpretations based on that abstract.