You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
A problem we have, especially since introducing ChEBI25 models in the latest version: If a class only has 25 samples, this leaves 2-3 on average for validation (given a 10% validation split as done in Chebifier 2). The consequence: The validation F1 is a unreliable measure of test set performance since a single prediction can completely change the F1 score.
One remedy suggested by @jcapp4 and implemented for the MOSES paper (in submission): If a class gets an F1 score of 0, use the average F1 score of classes with <100 samples instead. This way, a model that (maybe due to being unlucky) fails on all validation samples, still gets a vote. This slightly improved performance for experiments with ChEBI50 and ELECTRA.
A reviewer of HyDI made a similar suggestion, naming bayesian shrinkage. The idea there is to pull uncertain validation F1 scores to a more stable prior (which could be the macro-F1 on similar sized classes). This is a generalisation of the above approach and worth evaluating for our ensemble.
Activity