You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
21ICCV # Emerging Properties in Self-Supervised Vision Transformers (DINO) #44
A new proposed self-supervised learning method with KD: a form of knowledge distillation with no labels. Especially, it uses a different way to avoid the collapse solution, that is use the momentum teacher encoder.
It encouraging "local-to-global" correspondences by feeding different sizes of views to student and teacher encoders.
SSL ViT features explicitly contain the scene layout and, in particular, object boundaries, as shown in the next figure.
They look at the self-attention of the [CLS] token on the heads of the last layer. This token is not attached to any label nor supervision. These maps show that the model automatically learns class-specific features leading to unsupervised object segmentations.
Activity
XFeiF commentedon Aug 30, 2021
They look at the self-attention of the [CLS] token on the heads of the last layer. This token is not attached to any label nor supervision. These maps show that the model automatically learns class-specific features leading to unsupervised object segmentations.
SSL segmentation magic -> SSL + Transformer's [CLS] token.