Loud or Silent? A Reusable Framework for Per-Modality Failure Analysis in Multimodal Clinical AI

This paper presents a model-agnostic, reusable framework for analyzing failures in multimodal clinical AI when modalities are missing. Given modality embeddings, a mask-aware probe, and labels, it produces a per-example failure taxonomy, a per-modality complementarity matrix, and a loud-versus-silent dropout profile. “Loud” failures are monitorable after modality removal, while “silent” failures pass unflagged, including cases far from the decision boundary. The authors report validation against planted ground truth across random seeds, with recovery of modality dominance, complementary subsets, and per-modality failure rates. The supplied abstract is truncated before its full scaling results.