Scaling Native Multimodal Pre-Training From Scratch

This paper studies scaling laws for transformer-based vision-language models pretrained natively on multimodal inputs from scratch. Under a fixed compute budget, it analyzes how model size, token count, and data composition should be allocated. The authors report distinct scaling behavior for language and multimodal objectives: language allocation is relatively stable across mixtures, while multimodal allocation is highly sensitive to the text-to-multimodal ratio. They derive an efficiency frontier for choosing model size, tokens, and mixture, and report positive cross-modal transfer to text-only spatial reasoning and multimodal in-context learning.