Proxy Exploration and Reusable Guidance: A Modular LLM Post-Training Paradigm via Proxy-Guided Update Signals

The paper introduces Proxy-guided Update Signal Transfer (PUST), a post-training framework that separates behavior exploration from distribution alignment. A lightweight proxy model first explores high-reward behaviors; PUST then extracts the relative improvement between the proxy’s initial and optimized states and transfers that directional signal to a primary model. The design supports asynchronous signal generation, caching, reuse, and cross-model transfer. Evaluations on Qwen3-family models in mathematics and code reportedly show that signals from substantially weaker proxies can improve stronger primary models. The supplied abstract does not provide detailed metrics, baselines, or implementation specifics.