You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
we need to make a few fixes to autorunner and ensembler
BUG: AutoRunner().set_num_fold() does not update the enemble method num_folds. So e.g. if the default method is AlgoEnsembleBestByFold (it'll be initialized with 5 folds by default), but when a user calls AutoRunner().set_num_fold(2), it only updates number of folds for training, and the ensembling will still look for 5 folds, which results in some algorithms being None. I believe this error AutoRunner small updates #5523 (comment) was due to this, and not only due to Auto3DSeg ensembler module doesn't work with Bundle Algorithms generated by the HPO Module #5558
change the default ensemble method to AlgoEnsembleBestByFold in AutoRunner().set_ensemble_method()
2b. improve AlgoEnsembleBestByFold to return N best from each fold. Currently it returns only 1 best result, but we need it to return for e.g. N=2, or N=3)
VoteEnsemble() currently does not work for (sigmoid==True) mode
remove pickling of algorithms class instances, and re-initialize them in standard way
in class AlgoEnsembleBestN() line 196 , raises error if I select only 1 algorithm and 1 fold for training. The default n_best is 2, since we trained only 1 fold/1alg, len(ranks)==1
Hi @myron , for the Item 3, I am wondering where the change should happen. Do you think it should be making monai.transforms.VoteEnsemble support one-hot encoding input?
classes = [prob2class(p, dim=0, keepdim=True, sigmoid=False) for p in preds]
return VoteEnsemble(num_classes=preds[0].shape[0])(classes)
it converts to classes fo sigmoid==False (hardcoded).
there should be an alternative for sigmoid==True
get classes as classes = [prob2class(p, dim=0, keepdim=True, sigmoid=True) for p in preds] (size PxCxHxWxD), then you can rearrange, for each channel , may be like this
for i in range(classes.shape[1]):
classes_i = [c[:,[i]] for c in range(len(classes))]
v_i=VoteEnsemble(...)(classes_i)
return concat/stack(v_i)...
@myron Thanks for the explanation. Before I ran into issues using VoteEnsemble with one-hot encoding. After checking the code details, I think the way to VoteEnsemble in my PR should address and give equivalent results.
@myron Thanks for the suggestion. When the saving should happen, right after infer_instance.predict()?
Also, item 2 is fixed, but this enhancement isn't implemented yet, just fyi
"improve AlgoEnsembleBestByFold to return N best from each fold. Currently it returns only 1 best result, but we need it to return for e.g. N=2, or N=3)"
@myron Thanks for the suggestion. When the saving should happen, right after infer_instance.predict()?
Also, item 2 is fixed, but this enhancement isn't implemented yet, just fyi "improve AlgoEnsembleBestByFold to return N best from each fold. Currently it returns only 1 best result, but we need it to return for e.g. N=2, or N=3)"
It will require some re-design to be for each file ( infer all models, ensemble, and save). Currently, we have for each file (infer all models, ensemble), then again for each file (save)
Activity
myron commentedon Dec 7, 2022
first issue is addressed in #5667
mingxin-zheng commentedon Dec 13, 2022
Hi @myron , for the Item 3, I am wondering where the change should happen. Do you think it should be making
monai.transforms.VoteEnsemblesupport one-hot encoding input?change the default ensemble method to AlgoEnsembleBestByFold in AutoR… (
myron commentedon Dec 13, 2022
I AlgoEnemble.ensemble_pred()
it converts to classes fo sigmoid==False (hardcoded).
there should be an alternative for sigmoid==True
classes = [prob2class(p, dim=0, keepdim=True, sigmoid=True) for p in preds](size PxCxHxWxD), then you can rearrange, for each channel , may be like thismyron commentedon Dec 13, 2022
PS: I also added issue 5 (above) @mingxin-zheng
mingxin-zheng commentedon Dec 14, 2022
@myron Thanks for the explanation. Before I ran into issues using VoteEnsemble with one-hot encoding. After checking the code details, I think the way to VoteEnsemble in my PR should address and give equivalent results.
mingxin-zheng commentedon Dec 14, 2022
Issue 5 was also addressed. Now it would post a warning instead of raising an error.
Remove hard-coded sigmoid for voting ensemble method in Auto3DSeg (#5734
myron commentedon Dec 14, 2022
@mingxin-zheng ok, thank you. I've added 6th item (it's non-urgent)
mingxin-zheng commentedon Dec 15, 2022
@myron Thanks for the suggestion. When the saving should happen, right after
infer_instance.predict()?Also, item 2 is fixed, but this enhancement isn't implemented yet, just fyi
"improve AlgoEnsembleBestByFold to return N best from each fold. Currently it returns only 1 best result, but we need it to return for e.g. N=2, or N=3)"
myron commentedon Dec 15, 2022
no, after the self.ensemble_pred()
https://github.com/Project-MONAI/MONAI/blob/dev/monai/apps/auto3dseg/ensemble_builder.py#L162
It will require some re-design to be for each file ( infer all models, ensemble, and save). Currently, we have for each file (infer all models, ensemble), then again for each file (save)