Cross-Model Agreement as a Deployment-Time Reliability Signal for Automatic Polyp Segmentation

Deploying polyp segmentation models during real-time colonoscopy carries the risk of silent inference failures due to the absence of ground-truth annotations. This work introduces Referee-Based Quality Estimation (RBQE), a reference-free framework that assesses segmentation confidence via agreement with an independently trained referee model. Evaluated on a 1,223-image multi-dataset benchmark, a lightweight SegFormer-B0 referee achieves an ROC-AUC of 0.960—outperforming test-time augmentation while requiring only a single extra deterministic forward pass. Even after filtering out trivial empty-mask cases, cross-model agreement reliably flags unreliable outputs and enhances the precision of selective prediction.