-
What’s Hidden in a Randomly Weighted Neural Network?
Usually training a neural network means finding values for weights such that the network performs well on a given task, while keeping the network architecture fixed. The paper What’s Hidden in a Randomly Weighted Neural Network? investigates what happens if we swap the roles: we fix the values of weights and look for subnetworks in a given network that perform well on a given task; in other words, we optimise the network architecture while keeping weights fixed.
-
Representation Learning with Contrastive Predictive Coding
Contrastive Predictive Coding is an unsupervised learning approach for learning useful representations from high-dimensional data.
-
Going Deeper with Image Transformers
Our journey along the ImageNet leaderboard next takes us to 33rd place and the paper Going Deeper with Image Transformers by Touvron et al., 2021. In this paper they look at tweaks to the transformer architecture that allow them (a) to increase accuracy without needing external data beyond the ImageNet training set and (b) to train deeper transformer models.
-
Cross Modal Focal Loss for RGBD Face Anti-Spoofing
This is a summary and review of the recent paper on face anti-spoofing by George and Marcel 2021, presented at CVPR 2021.
-
Data-Efficient Image Transformers
This is the next post in the series on the ImageNet leaderboard and it takes us to place #71 – Training data-efficient image transformers & distillation through attention. The visual transformers paper showed that it is possible for transformers to surpass CNNs on visual tasks, but doing so takes hundreds of millions of images and hundreds if not thousands of compute-days on TPUs. This paper shows how to train transformers using only ImageNet data on GPUs in a few days.