An Analysis of Self-supervised Pre-training with Dependent Samples

While self-supervised learning routinely pools multiple dependent augmentations of the same input—such as random masking or noise injection—theoretical analyses have typically avoided these dependencies by partitioning data into smaller, independent sets. This paper provides statistical error bounds showing that pooling dependent augmentations is never worse than independent partitioning. For masking or noise-based augmentations in shallow networks, pooling can even yield faster convergence rates, providing a formal mathematical grounding for a widespread empirical habit.