BlockPilot: Instance-Adaptive Policy Learning for Diffusion-based Speculative Decoding

BlockPilot addresses a limitation in diffusion-based speculative decoding: using one fixed inference block size for every input is suboptimal. The paper reports that the best block size varies by sample but remains locally concentrated around the training block size. It introduces a lightweight policy that predicts the block size from the prefilling representation, only once after prefilling. According to the abstract, experiments on Qwen3-4B at temperature T=1 achieved an acceptance length of 5.92 and a 4.20x speedup, with minimal overhead and plug-and-play integration.