Poly-OPD: Heterogeneous Multi-Teacher On-Policy Distillation for Capability-Selectable Flow Models
Poly-OPD is an on-policy distillation framework for combining complementary capabilities from heterogeneous text-to-image teachers. It distills FLUX.1-dev and Z-Image into a 2.5B-parameter SD3.5-Medium student despite differences in autoencoders and noise schedules. A pixel bridge re-encodes student images with a selected teacher, while frozen DINOv2 features provide cross-latent supervision. Shared attention LoRA modules and teacher-specific feed-forward adapters are organized using gradient compatibility diagnostics, and a gap-aware curriculum targets weaker compositional categories. The paper reports GenEval improving from 67.3 to 73.3 and DrawBench HPSv3 from 9.34 to 11.35.
Why it's worth reading
Multi-teacher distillation is constrained by incompatible latent spaces and capability interference. Poly-OPD offers a concrete bridging and adapter strategy, with reported results showing a smaller student surpassing its larger teachers on the cited benchmarks.