Predicting What You Already Know Helps: Provable Self-Supervised Learning

Predicting What You Already Know Helps: Provable Self-Supervised Learning

Self-supervised representation learning solves auxiliary prediction tasks (known as pretext tasks), that do not require labeled data, to learn semantic representations. These pretext tasks are created solely using the input features, such as predicting a missing image patch, recovering the color channels of an image from context, or predicting missing words, yet predicting this $known\ $information helps in learning representations effective for downstream prediction tasks.This paper posits a mechanism based on conditional independence to formalize how solving certain pretext tasks can learn representations that provably decreases the sample complexity of downstream supervised tasks. Formally, we quantify how approximate independence between the components of the pretext task (conditional on the label and latent variables) allows us to learn representations that can solve the downstream task with drastically reduced sample complexity by just training a linear layer on top of the learned representation.

预测您已经知道的内容会有所帮助：可行的自我监督学习

自我监督的表示学习解决了不需要语义数据的辅助预测任务（称为前文任务），以学习语义表示。这些前置任务仅使用输入功能创建，例如，预测丢失的图像补丁，从上下文中恢复图像的色彩通道，或预测丢失的单词，然后进行预测 ķñØwñ 信息有助于学习对下游预测任务有效的表示形式。.. 本文提出了一种基于条件独立性的机制，以规范化解决某些借口任务可以如何学习表示形式的方法，从而显着降低了下游监督任务的样本复杂度。形式上，我们量化前置任务各组成部分之间的近似独立性（以标签和潜在变量为条件）如何使我们能够通过仅在学习的知识之上训练线性层来学习能够极大降低样本复杂度的解决下游任务的表示形式表示。 （阅读更多）