Codifying the Judge: Scalable Evaluation via Program Distillation

The paper introduces PAJAMA, which distills an LLM judge’s decision logic into a committee of inspectable and editable programs. These programs score candidates directly, while low-confidence cases can fall back to an LLM. According to the abstract, programmatic judges match a 13B LLM judge across five datasets and four model families, and improve accuracy and throughput when used as routing signals. On RewardBench, a reward model trained from program verdicts outperforms one trained on proprietary-LLM labels at two orders of magnitude lower API cost.