Knowing the Form, Not the Function: Automatically Auditing Answer–Authority Decoupling in Legal Benchmarks

This paper tests whether correct answers can proxy for properly grounded legal authority in benchmark evaluation. Across 238 Taiwan bar-examination items, four LLMs spontaneously produced authority markers even when statutory citations were not requested. Joint auditing found answer correctness and authority grounding dissociated in both directions. In criminal-law items, 24.0–42.4% of valid responses were correct but missed the gold authority, while 15.2–21.7% were incorrect yet cited it. Retrieval probes and a permissive citation-abstention intervention also changed answer and citation behavior separately, motivating joint evaluation for statute-grounded legal benchmarks.