奥古斯丁式 BabyLM：视觉指物初始化对小语言模型的效用边界

My Thoughts on Visual Initialization in Language Models

I began by considering that language models typically start with random word embeddings, meaning the model must learn the meaning of words like "banana" solely from training data. I implemented St. Augustine's concept of word learning, essentially "meaning by ostension," for a small masked language model (DeBERTa) trained on 10 million words. Before training, tokens visually grounded in images receive embeddings derived from their corresponding image regions; other tokens begin randomly. This visual initialization leaves a measurable imprint that persists throughout training. However, this effect remains largely invisible when evaluated using standard BabyLM benchmarks, which focus on abstract grammatical knowledge; visual initialization doesn't affect performance on those benchmarks. The only zero-shot exception I found was in object-property knowledge (COMPS, Misra et al. 2023), where seeding provides a consistent benefit. To further investigate this, I developed a corpus-specific version of the Visual-Property Swap benchmark (Lin et al., 2026), designed to test knowledge of color, material, size, and shape, considering per-item training frequency and whether the item was seeded. In this benchmark, models initialized with visual seeds exhibit a persistent and seed-replicated advantage, but only for the seeded words. To confirm this as a causal relationship, I demonstrated that applying synthetic grounding to previously unseeded words transfers this advantage to those specific words. I also observed that function words and abstract vocabulary benefit from strong visual seeds, retaining them throughout training, and that the training objective leverages these seeds, as evidenced by a reduction in held-out mask-prediction loss for these words across all seeds. Despite these findings, I've yet to find a benchmark that captures these effects. Determining an evaluation method capable of revealing this knowledge gain remains an open challenge.

语言模型通常以随机的词嵌入开始训练：无论“香蕉”是什么含义，都必须从训练语料库中学习。针对一个在1000万词规模上训练的小型掩码语言模型（DeBERTa），我实现了圣奥古斯丁关于词汇学习的构想（即通过指物示义）：在训练之前，具有视觉接地的词元会获得派生自其所标注图像区域的嵌入；其他词元则以随机状态开始。视觉初始化留下了一个可测量的印记，并持续存在到训练结束。与此同时，这种效应在大多数探究抽象语法知识的 BabyLM 基准测试中依然隐而不现：视觉初始化并未对这些测试的表现产生影响。唯一的零样本例外是物体属性知识（COMPS，Misra et al. 2023），在所有配置中，视觉种子植入都带来了改善。为了跟进这一结果，我构建了一个针对该语料库量身定制的“视觉-属性交换”（Visual-Property Swap）基准（Lin et al., 2026）版本，用于测试颜色、材质、尺寸和形状知识，并附带了每个条目的训练频率和种子状态。在该基准上，植入视觉种子的模型表现出持久且可在不同随机种子间复现的优势，并且这种优势仅限于植入了种子的词。作为因果检验，我证明对先前未植入种子的词进行合成式视觉接地，可以将这种优势精确转移到这些词上。虚词和抽象词汇也获得了较强的视觉种子，并在整个训练过程中保留了它们，训练目标亦利用了这些种子：在每一个随机种子设置下，这些词在留出集上的掩码预测损失均有所下降。然而，我运行的所有基准测试都未能体现出这一点。究竟何种评估手段能够捕捉到这一现象，仍然是一个悬而未决的问题。