Augustinian BabyLM: What Ostensive Definition Can and Cannot Teach a Small Language Model
Drawing on Augustine's view of word learning through pointing and naming, this paper investigates ostensive grounding in a small DeBERTa model trained on 10 million words. By initializing select token embeddings with visual features from labeled image regions, the author finds that visual imprints persist through pre-training, consistently improving zero-shot object-property knowledge such as color, size, and material. However, this grounding leaves standard grammatical benchmarks unaffected. Even as visual priors reduce prediction loss on abstract and function words, existing downstream evaluations fail to register the gain.
Why it's worth reading
It provides a clean, philosophically grounded test of multimodal token initialization, revealing clear boundaries between acquired physical property understanding and benchmark-blind representational shifts.