The Illusion of High Utility in Safety Alignment of Text-to-Image Diffusion Models

This paper argues that safety-aligned text-to-image diffusion models can create an illusion of high utility when evaluated mainly with coarse metrics such as FID and CLIPScore. On TIFA, a structured text-to-image faithfulness benchmark, aligned models show substantial failures in object counts, attributes, and relationships. The authors associate this degradation with semantic collapse in the text-encoder embedding space: reduced embedding spread and distorted inter-prompt similarity structure. They propose StructureAware Geometric Regularization (SAGE), which preserves these geometric properties during alignment. According to the abstract, SAGE improves TIFA by 5.0% over the previous state of the art while retaining strong safety performance.