Skip to content

About data leakage on zero-shot classification? #9

New issue
New issue
Open
Open
About data leakage on zero-shot classification?#9

Description

@LTEnjoy
LTEnjoy
opened on Nov 2, 2023

Hello!

Thanks for your great work! I have tested the zero-shot classification given your released checkpoint and it did a good performance. But I am confused that whether there exists some data leakage problem? Your model was fine-tuned on Swiss-Prot database and the DeepLoc dataset was also constructed from UniProt database. Did you do some filtering when you tested zero-shot performance?

Looking forward to your reply! Thanks in advance!

Activity

  1. KatarinaYuan commented on Nov 3, 2023

    @KatarinaYuan
    KatarinaYuan
    on Nov 3, 2023
    Collaborator

    Hi,
    Thank you being interested in our work!

    Please see the pre-training dataset

    ProtST/protst/dataset.py

    Line 22 in db53a76

    "swiss_prot": "https://miladeepgraphlearningproteindata.s3.us-east-2.amazonaws.com/uniprotdata/uniprot_sprot_filtered.tsv",
    . It does not expose test labeled data of each benchmark dataset that has not been observed during multimodal pre-training nor downstream fine-tuning.

  2. LTEnjoy commented on Nov 4, 2023

    @LTEnjoy
    LTEnjoy
    on Nov 4, 2023
    Author

    Hi,

    Thank you for the reply and I'll check it out!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions