Few-Medoids: An Embarrassingly Simple Coreset Selection Method for Few-Shot Knowledge Distillation

This paper proposes Few-Medoids, a simple coreset selection strategy for few-shot knowledge distillation. For each class, it selects samples closest to the class centroid, treating them as representative medoids. The authors report extensive experiments across four datasets and three teacher-student model pairs involving convolutional and transformer architectures. According to the abstract, Few-Medoids consistently outperforms random selection and other coreset methods such as herding and k-center Greedy. The implementation is publicly released, making the method relatively easy to reproduce and evaluate in existing few-shot KD pipelines.