Repository navigation
Open
Description
Question
Currently, the best model is selected based on val_micro-f1. Shouldn't the best model instead be selected based on val_macro-f1?
python-chebai/configs/training/default_callbacks.yml
Lines 1 to 12 in b0243f6
| - class_path: chebai.callbacks.model_checkpoint.CustomModelCheckpoint | |
| init_args: | |
| monitor: val_micro-f1 | |
| mode: 'max' | |
| filename: 'best_micro_f1_{epoch:02d}_{val_loss:.4f}_{val_macro-f1:.4f}_{val_micro-f1:.4f}' | |
| every_n_epochs: 1 | |
| save_top_k: 3 | |
| - class_path: chebai.callbacks.model_checkpoint.CustomModelCheckpoint | |
| init_args: | |
| filename: 'per_{epoch:02d}_{val_loss:.4f}_{val_macro-f1:.4f}_{val_micro-f1:.4f}' | |
| every_n_epochs: 25 | |
| save_top_k: -1 |
cc: @sfluegel05
Changes requested
- Change the best-model monitoring metric from
val_micro-f1toval_macro-f1, if this is the intended evaluation metric. - Save periodic checkpoints every 50 epochs.
- Ensure a checkpoint is also saved at the last/final epoch, even when the final epoch is not a multiple of 50.
- Keep all periodic checkpoints (
save_top_k: -1).
Expected behavior
The checkpointing should:
-
Select the top 3 best models based on
val_macro-f1. -
Save periodic checkpoints at:
- Epoch 50
- Epoch 100
- Epoch 150
- etc.
-
Always save a checkpoint at the final epoch.
Acceptance criteria
- Confirm whether
val_macro-f1should be used instead ofval_micro-f1for selecting the best model. - If confirmed, change
monitortoval_macro-f1. - Keep
mode: max. - Change periodic checkpointing from every 25 epochs to every 50 epochs.
- Ensure the final epoch is always saved.
- Keep
val_loss,val_macro-f1, andval_micro-f1in the checkpoint filename.
Metadata
Metadata
Assignees
Labels
No labels
Activity