LexFlip: A Dissociation Diagnostic for Legal Meaning Preservation Metrics
Standard semantic preservation metrics frequently confuse surface lexical overlap with legal validity. To probe this gap, researchers introduced LexFlip, a diagnostic suite of 373 minimal perturbations in Quebec statutory French where legal force is inverted while keeping 93% of tokens intact. Across seven embedding-based and BERTScore metrics, this inversion moved evaluation scores by less than 4% of their dynamic range. In contrast, bidirectional natural language inference captured the shift far more effectively, while simple token length outperformed every evaluated semantic metric against human annotations.
Why it's worth reading
It reveals a critical blind spot in legal text evaluation, demonstrating that conventional embedding metrics remain largely blind to reversed legal outcomes under high lexical overlap.