Skip to content

21ICCV # Emerging Properties in Self-Supervised Vision Transformers (DINO) #44

Open

Description

@XFeiF

Paper
Code

Authors:
Mathilde Caron, Hugo Touvron, etc.
FBAI.

Highlights:

  • A new proposed self-supervised learning method with KD: a form of knowledge distillation with no labels. Especially, it uses a different way to avoid the collapse solution, that is use the momentum teacher encoder.
  • It encouraging "local-to-global" correspondences by feeding different sizes of views to student and teacher encoders.
  • SSL ViT features explicitly contain the scene layout and, in particular, object boundaries, as shown in the next figure.

Activity

  1. added
    area/SSLself-supervised learning
    Summary/BriefA breif summary about the paper.
    CodeCode available.
    on Aug 30, 2021
  2. XFeiF commented on Aug 30, 2021

    @XFeiF
    OwnerAuthor

    They look at the self-attention of the [CLS] token on the heads of the last layer. This token is not attached to any label nor supervision. These maps show that the model automatically learns class-specific features leading to unsupervised object segmentations.

    SSL segmentation magic -> SSL + Transformer's [CLS] token.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    CodeCode available.Summary/BriefA breif summary about the paper.area/SSLself-supervised learningtrend/TransformerEvery paper uses transformer...

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions