Wake up for Touch! Mask-isolated Tactile Alignment Learning in MLLMs

The paper introduces Splash, a mask-isolated tactile alignment framework for multimodal large language models. It estimates the significance of pretrained parameters and partitions them into dormant and critical subspaces. The critical subspace remains frozen as an anchor for existing visual knowledge, while the dormant subspace is updated to learn tactile alignment with the language model. According to the abstract, Splash enables tactile reasoning while avoiding catastrophic forgetting, adds no inference overhead in the LLM component, and achieves state-of-the-art results on visuo-tactile benchmarks including SSVTP, TVL, and TacQuad.