OmniVAE introduces a jointly trained audio-video VAE designed to align the latent spaces of two structurally different modalities. It combines reconstruction with a segment-level audio-video contrastive objective to capture temporal and semantic correspondence. It also distills features from pretrained modality-specific semantic encoders into each modality. According to the paper’s abstract, both objectives consistently improve latent-space learnability, leading to better downstream text-to-audio-video generation quality and more accurate cross-modal synchronization. The supplied abstract does not provide benchmark names, numerical gains, model scale, or computational costs.
No heat snapshots are available in the last 24 hours.