TLA+-Bench evaluates natural-language-to-TLA+ generation through execution: each gold specification includes a configuration that lets the TLA+ model checker explore the full reachable state space and determine whether the named properties hold. The dataset contains 403 model-checked gold specifications and 897 parse-only silver specifications from 13 public repositories, plus model-written descriptions in two styles. On one fixed output set, changing previously unstated grading choices moved the measured correct rate from 1.7% to 10.0%, or to 18.7% when interface names were supplied. The strongest model reached 16% correctness by default, while open models reached at most 1%.
No heat snapshots are available in the last 24 hours.