Skip to content

Try out bayesian shrinkage #35

Open

Description

@sfluegel05

A problem we have, especially since introducing ChEBI25 models in the latest version: If a class only has 25 samples, this leaves 2-3 on average for validation (given a 10% validation split as done in Chebifier 2). The consequence: The validation F1 is a unreliable measure of test set performance since a single prediction can completely change the F1 score.

One remedy suggested by @jcapp4 and implemented for the MOSES paper (in submission): If a class gets an F1 score of 0, use the average F1 score of classes with <100 samples instead. This way, a model that (maybe due to being unlucky) fails on all validation samples, still gets a vote. This slightly improved performance for experiments with ChEBI50 and ELECTRA.

A reviewer of HyDI made a similar suggestion, naming bayesian shrinkage. The idea there is to pull uncertain validation F1 scores to a more stable prior (which could be the macro-F1 on similar sized classes). This is a generalisation of the above approach and worth evaluating for our ensemble.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions