Do Speech Foundation Models Really Learn Words?

While self-supervised speech models excel at distinguishing words, strong discrimination often stems from acoustic word forms rather than genuine syntactic or semantic identities. By residualizing out phoneme information, this study demonstrates that later layers of HuBERT and wav2vec 2.0 genuinely encode form-independent word representations. This straightforward disentanglement also improves higher-order linguistic extraction in unsupervised word discovery tasks.