FVAttn: Adaptive Sparse Attention with Runtime Load Balancing for Video Generation

FVAttn targets workload imbalance in sparse attention for high-resolution video diffusion Transformers under multi-GPU sequence parallelism. Its frontend combines Top-p routing, a Top-k safety floor, and video-aware block organization. At runtime, it migrates a small number of heavy attention heads through peer-to-peer communication, fills slack on non-critical ranks with additional valuable blocks, and overlaps scheduling and migration with computation. The abstract reports that, on step-distilled Wan2.2 I2V, FVAttn reduces average load imbalance from 1.34 to 1.08, achieves a 4.41x attention speedup over FlashAttention, and improves DiT inference speed by 2.02–2.11x with competitive video quality.