In the early days of deep learning, we trained the models the hard way. But today, we have come up with so many sophistications. Once such thing is **Self-Supervised Learning (SSL). **SSL is a machine learning technique which allows you to build efficient models even with small amounts of labeled data. I’ve written a complete guide on Self-supervsied learning, do check it out.
A Beginner's Guide to Self-Supervised Learning
In this blog, we are about to explore Contrastive Learning — *“What if the data itself could tell the model what belongs together and what doesn’t — without any human labels at all?”. *By the end of this blog, you’ll see how contrastive learning fundamentally changed how we think about supervision, scale, and intelligence in AI.
Given an unlabeled dataset, the core idea of contrastive learning is to pull similar (positive) examples closer and push dissimilar (negative) examples farther in the representational or feature space. This can be done by defining a loss function called contrastive loss function. The work of this loss function is to maximize the similarity between the positive pairs and minimize it for the negative pairs.
The positive pairs are typically created through data augmentations of the same instance, whereas the negative pairs come from different instances in the batch. The basic contrastive framework consists of selecting a data sample, called “anchor”, a data point belonging to the same distribution as the anchor, called the “** positive**” sample, and another data point belonging to a different distribution called the “** negative**” sample. We try to minimize the distance between the anchor and the positive sample and maximize the distance between the anchor and the negative sample in the latent space.
The most common techniques to select the positive and negative samples with respect to the anchor are **Instance Discrimination Method **and Image Sampling/Patching Method.
In this method, the entirety of images undergoes a series of transformations and used as positive samples to an anchor image. For example, an instance can be chosen and converted into grayscale to use as the positive sample. The negative sample can be any other image which is irrelevant to the anchor.
The most popular methods for image augmentation are as follows:
Image patching breaks a single image into multiple patches of a fixed dimension. Instead of feeding the entire image pixel-by-pixel into a network, we divide the image into a grid of small, non-overlapping patches. We then flatten each patch into a vector, and treat these vectors as the input “tokens” to the transformer.
Patching became so dominant because transformers love sequences; Transformers were designed for 1D sequences. Patching typically turns the 2D images into 1D sequences. It also allows working with much lower memory and computing resources.
The most commonly used loss function is Noise-Contrastive Estimation (NCE) or its variant, InfoNCE.
Noise-Contrastive Estimation** (**NCE) is a statistical Method used to estimate probability distributions. It transforms the problem of estimating a complex distribution into a simpler classification problem. The primary idea is to distinguish the true samples from the artificially generated noise samples. NCE helps to learn the underlying data distribution without directly computing the partition function, which is often computationally expensive.
InfoNCE is a loss function designed for unsupervised learning, which helps to learn the general-purpose numerical representations from the raw data.
Here, The index probability depends solely on the ratio between conditional and marginal distribution. To approximate this ratio, soft-max is used for normalization.
It is one of the oldest loss proposed in contrastive learning. The core idea is to maximize the distance between samples if they do not belong to the same distribution. Similarly, the distance between the data points are minimized if they belong to the same distribution.
The goal is to enforce a margin (a safety buffer) between the similarity of positive pairs and negative pairs.
There are two dominant ways to write the max-margin contrastive loss, which are **Triplet loss **and Margin-based contrastive loss.
The core idea is to compare the distance between the data points and see if they lie under the margin. For every triplet (anchor, positive, negative), we want the distance between anchor and positive to be smaller than the distance between anchor and negative — by at least the fixed margin.
The margin typically lies in the range of 0.2–1.0 (depends on whether embeddings are normalized).
This is a variant of triplet loss — more modern and batch friendly. Instead of sampling explicit triplets, all positive and negative pairs within a batch are considered and applied with the margin constraint to every possible (anchor, negative) relative to each positive. This is essentially many soft triplet losses applied at once, without needing to sample triplets explicitly.
The N-pair loss generalized the triplet loss to use multiple positives per anchor.
N-pair uses one positive per anchor and was initially designed for supervised metric learning.
NT-Xent Loss is the most important contrastive loss in the model self-supervised learning. The core idea is to treat contrastive learning as a classification problem. For each image in a batch, we create two augmented views (positive pair). The model must learn to recognize that these two views belong together, while distinguishing them from all the other views in the batch (negatives). It is a Noise-Contrastive Estimation (NCE) loss.
Logistic loss is the classic loss used in binary classification.
Instead of one big softmax over all negatives, we treat every possible pair independently as a binary classification problem.
This is one of the most influential contrastive self-supervised learning mode available. It showed that with strong data augmentations, very large batch sizes, and a simple NT-Xent loss, the model could learn representations that could beat supervised pretraining on ImageNet — without any labels.
SimCLR takes one image, creates two random strong augmentations, pushes their representations (positives) close together in the embedding space, which pushing them away from all other images (negatives) in a very large batch. It uses **NT-Xent Loss **with a large batch size (4096).
Given N images, each image undergoes three random transformations applied twice, yielding 2N augmented samples. These augmented images are then fed into the encoder to obtain the representations.
A projection layer is then attached to the representations, which helps to reduce the dimensions. It squeezes the 2048-dimensional output down into 128 dimensions and calculate the contrastive loss on these smaller vectors.
MoCo solved one of the biggest practical problem of SimCLR — the need for extremely large batch sizes (4096+) to provide enough negatives. It introduces a **momentum encoder and a memory queue **to maintain a large and consistent set of negatives without requiring huge batches. The core idea is that it maintains a slowly-updating “teacher” encoder and a queue of past negative representations, allowing the model to have tens of thousands of negatives even with small batch sizes, while keeping the training stable.
SwAV combines contrastive learning with online clustering. This achieves strong performance while using much fewer negatives and avoiding the need for large batches or memory queues. It trains the model to assign two augmented views of the same image to the same cluster, but does so by swapping the assignment codes between the two views. This inherently forces the network to learn consistent and meaningful cluster assignment online. It uses **Sinkhorn-Knopp Normalization **to prevent all samples collapsing to one prototype, by enforcing that each prototype gets roughly equal share of assignments. This method has the semantic richness of clustering and the invariance power of contrastive learning, making it an efficient one. The core idea is instead of forcing both views to agree directly, make one view predict what the other view “thinks” the cluster should be — and do it symmetrically.
CLIP trains an image encoder and text decoder to map images and their natural language descriptions into a shared embedding space where matched pairs are close together and mismatched pairs are far apart. It is mainly used in MultiModal environments.
The unique features of CLIP are multimodal learning, zero-shot learning, and self-sueprvised learning.
Usage
$ conda install --yes -c pytorch pytorch=1.7.1 torchvision cudatoolkit=11.0$ pip install ftfy regex tqdm$ pip install git+https://github.com/openai/CLIP.git
python
import torchimport clipfrom PIL import Imagedevice = "cuda" if torch.cuda.is_available() else "cpu"model, preprocess = clip.load("ViT-B/32", device=device)image = preprocess(Image.open("CLIP.png")).unsqueeze(0).to(device)text = clip.tokenize(["a diagram", "a dog", "a cat"]).to(device)with torch.no_grad(): image_features = model.encode_image(image) text_features = model.encode_text(text) logits_per_image, logits_per_text = model(image, text) probs = logits_per_image.softmax(dim=-1).cpu().numpy()print("Label probs:", probs) # prints: [[0.9927937 0.00421068 0.00299572]]
The most important innovation of CLIP is it;s ability to perform “zero-shot learning”.
The code below performs zero-shot prediction using CLIP, as shown in Appendix B in the paper. This example takes an image from the CIFAR-100 dataset, and predicts the most likely labels among the 100 textual labels from the dataset.
import osimport clipimport torchfrom torchvision.datasets import CIFAR100# Load the modeldevice = "cuda" if torch.cuda.is_available() else "cpu"model, preprocess = clip.load('ViT-B/32', device)# Download the datasetcifar100 = CIFAR100(root=os.path.expanduser("~/.cache"), download=True, train=False)# Prepare the inputsimage, class_id = cifar100[3637]image_input = preprocess(image).unsqueeze(0).to(device)text_inputs = torch.cat([clip.tokenize(f"a photo of a {c}") for c in cifar100.classes]).to(device)# Calculate featureswith torch.no_grad(): image_features = model.encode_image(image_input) text_features = model.encode_text(text_inputs)# Pick the top 5 most similar labels for the imageimage_features /= image_features.norm(dim=-1, keepdim=True)text_features /= text_features.norm(dim=-1, keepdim=True)similarity = (100.0 * image_features @ text_features.T).softmax(dim=-1)values, indices = similarity[0].topk(5)# Print the resultprint("\nTop predictions:\n")for value, index in zip(values, indices): print(f"{cifar100.classes[index]:>16s}: {100 * value.item():.2f}%")
The output will look like the following (the exact numbers may be slightly different depending on the compute device):
Top predictions: snake: 65.31% turtle: 12.29% sweet_pepper: 3.83% lizard: 1.88% crocodile: 1.75%
As datasets grow to trillions of examples, as losses become more robust to noise (SigLIP, logistic variants), as architectures become more multimodal by default, and as we push contrastive principles into video, 3D, audio, graphs, and robotics — the complexity grows. But, thanks to contrastive learning, we can deal with them!
I hope you would have enjoyed reading this blog. Do let me know your feedback in the comments section. I’m happy to hear from you.
Stay curious!
Note: The content of this blog is curation of data from multiple sources. I have referred a lot of publicly available content and refined them into a single piece.
Contrastive Learning: The Secret Behind Today’s Smartest Vision Models was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.