Same Formulas, Different Semantics: Do Language Models Follow Modal Logic Specifications?

This paper tests whether language models follow explicitly stated modal semantics instead of defaulting to familiar logics. It constructs paired problems with identical premises and conjectures but different accessibility-frame or domain conditions, with automated reasoning verifying opposite labels. On a balanced core set, four of five recent models perform below a condition-only baseline under direct prompting. Enabling reasoning mode raises DeepSeek V4 Flash from 4.4% to 88.1% on unchanged prompts. The authors also release formulas, oracle artifacts, countermodels, and model responses.