Which Modality Decides? Counterfactual Modality Attribution for Multimodal LLMs

The paper introduces Counterfactual Modality Attribution (CMA), a framework for estimating whether images, text, or their combination drives an MLLM prediction. It creates image-only, text-only, and joint multimodal counterfactuals with coupled diffusion priors, then derives modality contributions using Shapley values from cooperative game theory. On controlled synthetic benchmarks with known modality reliance and a real-world multimodal clinical dataset, the authors report 98% accuracy in identifying the decision-driving modality in controlled cases and consistent improvements over baselines. The work positions modality attribution as complementary to token- or region-level explanations, especially for auditing shortcut reasoning in safety-critical systems.