Good Benchmarks

The paper argues that high-quality benchmark tasks should be correct, solvable, verifiable, and well specified, while remaining difficult for interesting reasons. Tasks should represent real problems recognized by experienced practitioners and use language practitioners would naturally use. Evaluation should verify the outcome rather than prescribe or inspect the approach. The supplied abstract does not provide authors, experimental results, benchmark names, or a formal evaluation framework.