SimCLR: Simple Contrastive Learning for Vision

A single sentence: contrastive learning doesn’t need momentum encoders, memory banks, or custom architectures — as long as you nail the data augmentation, add a nonlinear projection head, and use a large enough batch, a dead-simple framework lets a self-supervised ResNet-50 catch up to supervised ImageNet performance.

0. Notation#

1. Motivation: why contrastive learning needed simplifying#

1.1 The self-supervised landscape in 2019#

In NLP, BERT and T5 had already proven the power of “large-scale unlabeled pretraining + light fine-tuning.” But in vision, the methods were all over the place:

SimCLR’s core question was disarmingly simple: are all these fancy components actually necessary? Can you get the same results with nothing but data augmentation, a contrastive loss, and a standard network?

1.2 What self-supervised learning is really doing#

Self-supervised learning turns an unsupervised problem into a supervised one by constructing surrogate labels from unlabeled data. SimCLR picks the simplest surrogate labels possible: two randomly augmented views of the same image are a positive pair; views of different images are negatives.

This is instance discrimination — every image is its own class.

2. The core framework#

2.1 Four-step pipeline#

Original image x
    |
    ├── Augmentation t ~ T → view x̃ᵢ
    |                          ↓
    |                    ResNet f(·) → hᵢ
    |                          ↓
    |                    MLP g(·) → zᵢ
    |                                    ↘
    └── Augmentation t' ~ T → view x̃ⱼ       Contrastive loss
                                   ↓         ↗
                             ResNet f(·) → hⱼ
                                   ↓
                             MLP g(·) → zⱼ

Step 1: Random data augmentation. Sample twice from the same augmentation family $\mathcal{T}$ and apply to the same image, producing two views $\tilde{x}_i$ and $\tilde{x}_j$. This is the single most important step — the augmentation design directly determines what the model learns.

Step 2: Encoder. A ResNet (no modifications) encodes the representation $h_i = f(\tilde{x}_i) = \text{ResNet}(\tilde{x}_i)$. $h_i$ is the pooling layer output, typically 2048-dimensional.

Step 3: Projection head. A small MLP maps $h$ into the contrastive space: $z_i = g(h_i) = W^{(2)}\sigma(W^{(1)}h_i)$, where $\sigma$ is ReLU. $z_i$ is typically 128-dimensional.

Step 4: Contrastive loss. Compute the NT-Xent loss in $z$-space, maximizing agreement between $z_i$ and $z_j$ while pushing apart $z$ vectors from all other samples in the batch.

After pretraining: throw away $g$, use $h = f(x)$ as the representation for downstream tasks. This is one of SimCLR’s most important practical details — $h$ works better than $z$ because $z$ discards information that turns out to be useful downstream (see Finding 2).

2.2 Algorithm pseudocode (Algorithm 1)#

The paper’s pseudocode is cleanly indexed — the two views of the $k$-th sample in a batch are stored at positions $2k-1$ and $2k$:

Input: batch size N, temperature τ, encoder f,
       projection head g, augmentation family T

for minibatch {x_k} (k=1..N):
    for k in 1..N:
        sample two augmentations t ~ T, t' ~ T
        # First augmented view
        x̃_{2k-1} = t(x_k)
        h_{2k-1} = f(x̃_{2k-1})      # representation
        z_{2k-1} = g(h_{2k-1})      # projection
        # Second augmented view
        x̃_{2k} = t'(x_k)
        h_{2k} = f(x̃_{2k})
        z_{2k} = g(h_{2k})

    for i, j in 1..2N:
        s_{i,j} = z_i^⊤ z_j / (‖z_i‖ ‖z_j‖)   # cosine similarity

    ℓ(i,j) = -log( exp(s_{i,j}/τ) / Σ_{k≠i} exp(s_{i,k}/τ) )
    L = 1/(2N) · Σ_{k=1}^N [ℓ(2k-1, 2k) + ℓ(2k, 2k-1)]

    update f, g to minimize L

return encoder f(·), discard projection head g(·)

2.3 Global BN: how BatchNorm lets the model cheat, and how to fix it#

This is one of those implementation details that’s easy to overlook but will silently trash every experiment if you get it wrong. The core problem: BatchNorm in distributed training leaks information about which GPU a sample lives on, and the model learns to exploit this shortcut rather than learning semantics.

Step-by-step: how BN leaks information#

Step 1: Recall what BatchNorm does.

Given a batch of values $\{x_1, x_2, \dots, x_m\}$ along some feature dimension, BN computes:

$\hat{x}_i = \frac{x_i - \mu}{\sigma}$

where $\mu = \frac{1}{m}\sum x_i$ and $\sigma = \sqrt{\frac{1}{m}\sum (x_i-\mu)^2}$. Both $\mu$ and $\sigma$ are computed from the samples in the batch.

Step 2: The naive distributed training approach.

Suppose 8 GPUs and batch size 4096. Standard data parallelism splits the 4096 samples evenly across 8 GPUs (512 each). A naive implementation has each GPU independently compute $\mu_k$ and $\sigma_k$ from its own 512 samples.

So for the same raw feature value $x$, landing on GPU 0 produces $\hat{x} = (x - \mu_0)/\sigma_0$, while landing on GPU 1 produces $\hat{x} = (x - \mu_1)/\sigma_1$. $\mu_k$ and $\sigma_k$ depend on which samples happen to be on that GPU — in other words, the BN output embeds “which GPU am I on?” information.

Step 3: Why this leak is fatal for contrastive learning.

SimCLR’s data loading guarantees that both views of the same image always land on the same GPU (they’re generated in the same data loader worker). So:

• Positive pair (image A’s view1 and view2) → both on GPU 0 → normalized with the same $\mu_0, \sigma_0$ → output shares the same GPU-level offset
• Negative pair (image A’s view vs. image B’s view) → likely on different GPUs → normalized with different $\mu, \sigma$ → output has different GPU-level offsets

The model quickly discovers a shortcut: instead of looking at image content, just check whether two features share the same BN offset pattern. High contrastive accuracy → low loss → but the representations are garbage, because the model never learned image semantics.

Step 4: Why this isn’t a minor issue.

The decoupling between pretext task accuracy and downstream representation quality is a recurring theme in SimCLR (we’ll see it again in section 5.1 — no L2 normalization gives higher contrastive accuracy but worse linear evaluation). BN cheating makes the pretext task too easy, the model takes the shortcut, representation learning fails — but the training curve looks perfectly normal. Deeply deceptive.

Three solutions#

Why SimCLR’s large batch naturally mitigates this problem: when the per-GPU batch size is already large (e.g., all 4096 samples on one or two GPUs), the local $\mu$ and $\sigma$ are already close to the global statistics, and the “GPU fingerprint” naturally fades. SimCLR insists on large batches partly for this reason — so SimCLR needs large batches for at least two independent reasons: more negatives + BN statistic stability.

2.4 Architecture comparison with MoCo#

SimCLR relies on large batches to provide enough negatives; MoCo uses a queue to bypass the batch size dependency. Mathematically, both use the same InfoNCE loss — they only differ in the negative sampling strategy.

3. The loss function: NT-Xent#

3.1 Definition#

Given $N$ samples in a batch, each with two augmented views, we have $2N$ representations. For a positive pair $(i, j)$:

$\ell_{i,j} = -\log \frac{\exp(\text{sim}(z_i, z_j) / \tau)}{\sum_{k=1}^{2N} \mathbb{1}_{[k \neq i]} \exp(\text{sim}(z_i, z_k) / \tau)}$

The final loss averages over all positive pairs (each sample contributes two symmetric pairs):

$\mathcal{L} = \frac{1}{2N} \sum_{k=1}^{N} \big[ \ell_{2k-1, 2k} + \ell_{2k, 2k-1} \big]$

3.2 NT-Xent vs. other losses#

NT-Xent is just temperature-scaled InfoNCE — it’s essentially a $(2N-1)$-way cross-entropy classification problem. The paper compared three loss families:

Why NT-Xent wins:

1. Temperature $\tau$ controls the weight assigned to hard negatives. Smaller $\tau$ means harder negatives get proportionally larger gradients.
2. Negative competition: all negatives share the denominator — strong negatives “steal” probability mass, forcing stronger gradients. Sigmoid judges each pair independently, without this competitive dynamic.
3. Connection to mutual information: NT-Xent is a variational lower bound on $I(\tilde{x}_i; \tilde{x}_j)$. Minimizing the loss = maximizing the mutual information between two views.

3.3 The role of temperature $\tau$#

$\frac{\exp(\text{sim}/\tau)}{\sum \exp(\text{sim}/\tau)}$

The paper found $\tau$ significantly affects performance and needs validation-set tuning.

4. Three key findings#

Finding 1: Data augmentation composition is everything#

This is SimCLR’s most important and most underappreciated discovery. The paper systematically ablates every augmentation and their combinations:

Single augmentations are never enough. Figure 5’s asymmetric ablation (one branch fixed to crop, the other branch doing a single augmentation) shows no single augmentation alone produces good representations (diagonal values all below 33%).

The paper’s precise ablations under default settings (with flip, batch 4096, 100 epochs):

Random horizontal flip (50% probability) is already included in the default random cropping pipeline — it’s not an extra augmentation beyond the default set.

Why is color distortion so critical? If you only do random cropping (no color distortion), two crops from the same image share extremely similar color distributions. The model can cheat by matching color histograms — declaring patches with similar colors as positives, ignoring semantics entirely.

Color distortion is independently applied to each crop, destroying this shortcut. Once the two views have independently distorted colors, the model can’t rely on low-level color statistics to tell if they come from the same image — it’s forced to learn more general semantic features.

Color distortion strength matters too. The paper uses a combination of AutoContrast + color jittering + color dropping. Too weak, and the model still has color-cheating room; too strong, and the two views become unrecognizable, breaking the positive pair definition.

Finding 2: The nonlinear projection head is crucial#

This is SimCLR’s most counterintuitive finding:

The paper only tested three architectures: identity, linear, and nonlinear (1 hidden layer). Figure 8 shows the nonlinear head beats linear by ~3 points, and linear beats identity by ~8-9 points. SimCLR v2 later explored deeper projection heads (3 layers) — that’s not in v1.

The most counterintuitive part: the post-projection representation $z$ — the one you directly optimize in the contrastive space — performs worse on downstream tasks. You get better results using the pre-projection $h$.

First, let’s clarify: this is not a dimensionality bottleneck. Section 6.3 shows that whether $z$ is 32-dimensional or 4096-dimensional, $h$‘s quality is almost unaffected. The projection head’s value isn’t in “compressing dimensions” — it’s in “nonlinear gating.”

So what’s going on? The contrastive loss pushes the two views’ $z$ vectors to agree. But these two views underwent independent data augmentation — one might be reddish, the other bluish. The loss function has no constraint on “what information should be kept vs. discarded” — it only knows to make $z_i \approx z_j$. If you compute the loss directly on $h$, then $h$ will actively discard information that’s “inconsistent between the two views” — color, texture, orientation — just to lower the loss. But this discarded information could be useful for downstream classification (distinguishing a tomato from a green pepper depends on color).

The projection head is a nonlinear “sacrificial layer.” The ReLU nonlinearity in $g$ lets it selectively ignore the invariance pressure from the contrastive loss. Specifically, $g$ learns “color is unstable between two views → I won’t pass color information to $z$,” so color is lost in $z$ — but that’s fine, because color information in $h$ is fully preserved. $g$ absorbs the contrastive loss’s demand for augmentation invariance, freeing $h$ to retain everything that might be useful downstream.

The paper verified this directly with an experiment (Table 3): train an additional MLP on the pretrained model to predict which augmentation was applied:

$g(h)$ on rotation prediction drops precisely to random guessing (25.6%, with a 4-class random baseline of 25%) — $g$ actively and completely discards rotation information. Yet $h$ retains 67.6%, far above random. Color information through $g$ drops from 99.3% to 97.4% — also significantly weakened. Sobel filtering drops from 96.6% to 56.3%, original-vs-corrupted from 99.5% to 59.6%. Across all four prediction tasks, $g(h)$ accuracy is substantially lower than $h$. This directly proves: $g$ is a sacrificial layer that absorbs the contrastive loss’s demand for transformation invariance, freeing $h$ to retain information that’s useful downstream but “unstable” under augmentation.

Finding 3: Scale pays off — and differently than in supervised learning#

Self-supervised vs. supervised scaling: supervised learning on ImageNet largely saturates after 90-300 epochs (more epochs yield minimal gains). SimCLR’s linear eval accuracy is still climbing after 800 epochs.

This suggests self-supervised learning benefits from scale in a qualitatively different way — it’s not fine-tuning around the same solution (as in supervised learning) but exploring better representation spaces.

Training configuration: batch size 4096 for 100 epochs is the paper’s standard baseline; the optimal configuration uses batch size 8192 for 1000 epochs.

Learning rate scaling: the paper defaults to linear scaling (lr = 0.3 × BatchSize/256). But a footnote mentions that for smaller batch sizes, switching to square-root scaling improves performance. This means learning rate scheduling and batch size interact — linear scaling isn’t optimal for every batch size.

Why does self-supervised learning need more scale than supervised? Supervised gradients come from human labels — each sample gives a clear direction. Self-supervised gradients come from the pretext task defined by data augmentation — the signal is weaker and noisier. Large batches provide more negatives → tighter mutual information lower bound → cleaner gradients; long training lets the model slowly converge under the weaker signal. This also explains why performance still climbs at 800 epochs — weak signals need more iterations to accumulate useful information.

5. Experimental results#

5.1 ImageNet linear evaluation#

Frozen pretrained ResNet-50 representations + a single linear classifier:

ResNet-50 (1x) comparison (Table 6):

ResNet-50 (4x) comparison (Table 6):

SimCLR (4x) ties supervised ResNet-50 on linear evaluation. Note that 76.5% is the 4x model — the 1x model is 69.3%, which still handily beats contemporary methods (MoCo v1’s 60.6%).

5.2 Semi-supervised fine-tuning#

Fine-tuning with 1% or 10% of ImageNet labels:

All numbers above are Top-1 accuracy, from Table B.4 (Appendix). The main text’s Table 7 reports Top-5 (SimCLR 4x = 85.8% / 92.6%).

Supervised learning from random initialization with 1% labels reaches only 25.4%. SimCLR pretraining + 1% labels hits 63.0% — a 37.6-point gap. This is extremely strong evidence for the data efficiency of self-supervised pretraining.

5.3 Full fine-tuning#

Fine-tuning the entire network with 100% ImageNet labels:

Be careful with these numbers: SimCLR (1x) fine-tuned is 76.0%, not 76.5% (76.5% is the 4x model’s linear eval result). The supervised baseline 78.4% uses a 4x model, not 1x. Apples-to-apples: SimCLR (4x) fine-tuned 80.1% vs. Supervised (4x) 78.4%.

5.4 Transfer learning#

Transfer results across 12 downstream classification tasks:

SimCLR representations, across 12 datasets: 5 significantly better than supervised pretraining, 2 significantly worse, 5 statistically tied. Overall, contrastively learned features transfer at least as well as — and sometimes better than — supervised features.

6. Ablation highlights#

6.1 L2 normalization: you can train without it, but you shouldn’t#

Normalizing $z$ to unit length ($z \leftarrow z / \|z\|$) makes similarity reduce to cosine similarity $\text{sim}(u,v) = u^\top v$. The paper found this is a “works without it, works much better with it” choice. And — like the BN cheating and projection head stories — there’s a counterintuitive twist: without normalization, contrastive accuracy is higher, but downstream quality is worse.

What happens without L2 normalization#

Similarity uses the dot product $u^\top v$. The dot product has no upper bound — feature magnitudes $\|z\|$ can grow arbitrarily large.

Consider the softmax denominator:

$Z(u) = \sum_{k} \exp(u^\top v_k / \tau)$

If some negative samples have unusually large feature norms, $\exp(\text{large value}/\tau)$ explosively dominates the denominator, pushing those samples’ softmax probabilities to extremes. Nearly all the gradient concentrates on these few “outlier samples” — most negatives contribute near-zero gradient.

Meanwhile, positive pairs also have a shortcut: $z_i$ and $z_j$ can both grow their norms → the dot product $z_i^\top z_j$ grows → the positive pair’s softmax probability easily approaches 1 → contrastive accuracy looks great. But the model learned “grow the magnitude,” a semantically irrelevant trick, rather than meaningful feature alignment.

With L2 normalization#

All $z$ vectors are projected onto the unit hypersphere, $\text{sim}(u,v) \in [-1, 1]$. Temperature $\tau$ becomes the sole controller of distribution sharpness, affecting all samples equally.

From the gradient formula (see section 3.2), with L2 normalization, each negative’s gradient contribution is determined by its relative difficulty $\exp(\text{sim}/\tau)$ — hard negatives (sim near 1) get more gradient, easy negatives (sim far from 1) get less. Without L2 normalization, this weighting is scrambled by arbitrary differences in feature magnitude.

The numbers (Table 5)#

Without normalization, contrastive accuracy is 4.3 points higher — the pretext task is easier — but downstream performance drops by 7.4 points. Same story again: the easier the pretext task (because there are shortcuts), the worse the representations.

6.2 Crop scale in data augmentation#

• Global views (large crops) capture scene-level semantics
• Local views (small crops) capture object details
• Combining both works best

This insight was later amplified by SwAV and DINO into the multi-crop strategy.

6.3 Projection head output dimension#

The paper experimented with $z$ dimensions from 32 to 4096. The core finding: as long as you have a projection head (linear or nonlinear), the output dimension barely matters. Figure 8 shows that with a nonlinear head, performance is flat from 32 to 4096 dimensions. The paper defaults to 128, but there’s no special advantage — higher dimensions don’t help, and lower dimensions barely hurt. This means the projection head’s value is in existing at all (as a nonlinear sacrificial layer absorbing the contrastive loss’s invariance pressure), not in its specific dimensionality. The only exception is no projection head at all — performance crashes, as shown in Finding 2’s Table 3 evidence.

7. Deep comparison with MoCo#

SimCLR and MoCo are the two defining contrastive learning papers of 2020, released simultaneously with comparable impact — but they took different paths:

Who was right? MoCo v2 later adopted SimCLR’s data augmentation and projection head design, achieving higher performance with smaller batches. This shows both negative sampling strategies (large batch vs. queue) have pros and cons — but data augmentation and the projection head are the shared keys. SimCLR was the first to systematically prove both are indispensable.

8. How the line evolved#

SimCLR was a bombshell that triggered the 2020-2021 explosion in contrastive learning:

SimCLR (2020.02)
  |
  ├── SimCLR v2 (2020.06)
  |     └─ Bigger model + deeper projection head + distillation → SOTA
  |
  ├── MoCo v2 (2020.03)
  |     └─ Absorbed SimCLR's data augmentation + projection head
  |
  ├── MoCo v3 (2021.04)
  |     └─ ViT architecture + contrastive learning, discovered instability
  |
  ├── BYOL (2020.06)
  |     └─ "You don't even need negatives": predictor + stop-gradient
  |         Key insight: the core mechanism might not be contrast — it's collapse prevention
  |
  ├── SimSiam (2020.11)
  |     └─ Simplified BYOL further: stop-gradient alone is enough
  |         "Stop-gradient is the minimal sufficient condition to prevent collapse"
  |
  ├── SwAV (2020.06)
  |     └─ Clustering instead of instance discrimination, multi-crop augmentation
  |
  └── DINO (2021.04) / DINO v2 (2023)
        └─ Self-distillation + ViT, ViT features spontaneously exhibit semantic segmentation

8.1 What replaced SimCLR?#

SimCLR’s core weakness: it needs huge batch sizes (8192), which is hardware-unfriendly.

Subsequent methods addressed this from different angles:

• BYOL / SimSiam: eliminated negatives entirely — no more batch size dependency
• MoCo v2/v3: kept negatives but used a queue — small batches work fine
• MAE (2021): pivoted to reconstruction (Masked Autoencoder), regression on MSE, but with aggressive masking
• DINO v2: self-distillation + ViT, feature quality far beyond the ResNet era

SimCLR isn’t directly used today, but its three findings (augmentation composition, projection head, scaling) remain design fundamentals for all self-supervised visual methods.

9. Strengths and weaknesses#

Strengths#

1. Minimalist design: standard ResNet + standard augmentations + standard contrastive loss. No momentum encoders, memory banks, clustering, or custom architectures. The code is under 500 lines.

2. Systematic ablation: methodically verified the role of augmentations, projection head, batch size, training duration, and loss function choice. Every ablation includes quantitative results and qualitative explanation.

3. Clear theoretical intuition: color distortion → prevents color cheating; projection head → information bottleneck that protects useful features in $h$; large batch → more negatives → tighter mutual information lower bound.

4. A performance milestone: the first time a self-supervised ResNet-50 matched supervised learning on ImageNet linear evaluation. Semi-supervised fine-tuning substantially beat the supervised baseline.

5. Lasting influence: all three key findings were absorbed by MoCo v2, BYOL, SimSiam, SwAV, and every subsequent method.

Weaknesses#

1. Strong batch size dependence: needs 4096-8192 batch size, 32-128 TPU cores. Very unfriendly for hardware-constrained settings.

2. Within-batch negative coupling: samples in a batch serve as each other’s negatives — batch quality significantly affects training. Negative distributions can be uneven across batches.

3. BatchNorm information leakage was discovered concurrently by multiple groups: SimCLR, MoCo, and CPC v2 all identified the cross-sample BN information leak in contrastive learning almost simultaneously, each with their own fix: SimCLR used Global BN (cross-GPU μ/σ sync), MoCo used Shuffle BN, CPC v2 used Layer Norm. SimCLR’s large batch makes Global BN statistics more stable, but also means Global BN under small batches isn’t as clean as MoCo’s Shuffle BN — this is a tradeoff, not an unsolved problem.

4. Vision-specific: the core insight (color distortion prevents cheating) only applies to vision — unlike CPC, which works cross-modally.

5. Training efficiency: 800-1000 epochs far exceeds supervised learning’s 90-300. Representation quality is better, but compute cost is significantly higher.

10. Key formulas#

NT-Xent Loss (SimCLR’s form of InfoNCE)#

$\ell_{i,j} = -\log \frac{\exp(\text{sim}(z_i, z_j) / \tau)}{\sum_{k=1}^{2N} \mathbb{1}_{[k \neq i]} \exp(\text{sim}(z_i, z_k) / \tau)}$

where $\text{sim}(z_i, z_j) = \frac{z_i^\top z_j}{\|z_i\| \|z_j\|}$ (cosine similarity) and $\tau$ is the temperature.

Projection head#

$g(h) = W^{(2)} \cdot \text{ReLU}(W^{(1)}h)$

where $h \in \mathbb{R}^{2048}$ (ResNet-50 output), $W^{(1)} \in \mathbb{R}^{2048 \times 2048}$, $W^{(2)} \in \mathbb{R}^{2048 \times 128}$.

Core ablation conclusion (qualitative)#

$\text{Performance} \propto \text{Augmentation diversity} \times \text{Projection head nonlinearity} \times \text{Batch size} \times \text{Training duration}$

All four dimensions show near-monotonic positive returns, and they’re largely independent of each other.

11. Closing thought: SimCLR’s “three differences” from supervised learning#

The paper’s conclusion has a beautifully concise self-assessment:

“Our approach differs from standard supervised learning on ImageNet only in the choice of data augmentation, the use of a nonlinear head at the end of the network, and the loss function.” — SimCLR, Section 8 Conclusion

Another way to read this: take a standard ResNet-50, standard ImageNet data, and standard SGD. Swap out three components:

1. Labels → data augmentation (contrastive prediction task instead of classification)
2. Classification head → nonlinear projection head (MLP instead of fully-connected softmax)
3. Cross-entropy → NT-Xent (contrastive loss instead of classification loss)

— and you get a self-supervised model whose linear evaluation performance matches supervised learning.

The paper’s final sentence is worth sitting with:

“The strength of this simple framework suggests that, despite a recent surge in interest, self-supervised learning remains undervalued.”

Written in 2020. Looking back, the GPT family (autoregressive language modeling is, at its core, self-supervised), MAE, DINO v2, and today’s multimodal large models all keep proving this judgment right.

Related notes#

• [[NCE|Noise Contrastive Estimation]] — the mathematical roots of NT-Xent / InfoNCE
• [[CPC|Contrastive Predictive Coding]] — the origin paper of InfoNCE loss, direct predecessor of SimCLR’s loss function
• [[MoCo]] — contemporary contrastive learning method, momentum queue solves the batch size problem
• [[BYOL]] — eliminated negatives, proving contrastive loss may not be the core mechanism
• [[Vision Transformer]] — the backbone used by MoCo v3 / DINO and subsequent work
• [[SSL|Self-Supervised Learning]] — field overview

References#

• Chen, T., Kornblith, S., Norouzi, M., & Hinton, G. (2020). A Simple Framework for Contrastive Learning of Visual Representations. ICML 2020. arXiv:2002.05709.
• Google AI Blog (2020.04.08): Advancing Self-Supervised and Semi-Supervised Learning with SimCLR ↗ — by Ting Chen & Geoffrey Hinton
• Chen, T., Kornblith, S., Swersky, K., Norouzi, M., & Hinton, G. (2020). Big Self-Supervised Models are Strong Semi-Supervised Learners (SimCLR v2). NeurIPS 2020. arXiv:2006.10029.
• He, K., Fan, H., Wu, Y., Xie, S., & Girshick, R. (2020). Momentum Contrast for Unsupervised Visual Representation Learning (MoCo). CVPR 2020. arXiv:1911.05722.