Standard semantic preservation metrics frequently confuse surface lexical overlap with legal validity. To probe this gap, researchers introduced LexFlip, a diagnostic suite of 373 minimal perturbations in Quebec statutory French where legal force is inverted while keeping 93% of tokens intact. Across seven embedding-based and BERTScore metrics, this inversion moved evaluation scores by less than 4% of their dynamic range. In contrast, bidirectional natural language inference captured the shift far more effectively, while simple token length outperformed every evaluated semantic metric against human annotations.
No heat snapshots are available in the last 24 hours.