Skip to content

Autorunner AlgoEnsemble improvements #5564

Closed

Description

@myron

we need to make a few fixes to autorunner and ensembler

  1. BUG: AutoRunner().set_num_fold() does not update the enemble method num_folds. So e.g. if the default method is AlgoEnsembleBestByFold (it'll be initialized with 5 folds by default), but when a user calls AutoRunner().set_num_fold(2), it only updates number of folds for training, and the ensembling will still look for 5 folds, which results in some algorithms being None. I believe this error AutoRunner small updates #5523 (comment) was due to this, and not only due to Auto3DSeg ensembler module doesn't work with Bundle Algorithms generated by the HPO Module #5558

  2. change the default ensemble method to AlgoEnsembleBestByFold in AutoRunner().set_ensemble_method()

2b. improve AlgoEnsembleBestByFold to return N best from each fold. Currently it returns only 1 best result, but we need it to return for e.g. N=2, or N=3)

  1. VoteEnsemble() currently does not work for (sigmoid==True) mode

  2. remove pickling of algorithms class instances, and re-initialize them in standard way

  3. in class AlgoEnsembleBestN() line 196 , raises error if I select only 1 algorithm and 1 fold for training. The default n_best is 2, since we trained only 1 fold/1alg, len(ranks)==1

  4. in the ensembling step we accumulate all results in memory, and then save them to disk. this can create OOM if RAM is small. I think we should save to disk each result as soon as we get it. https://github.com/Project-MONAI/MONAI/blob/dev/monai/apps/auto3dseg/ensemble_builder.py#L162

@mingxin-zheng

Activity

  1. myron commented on Dec 7, 2022

    @myron
    CollaboratorAuthor

    first issue is addressed in #5667

  2. mingxin-zheng commented on Dec 13, 2022

    @mingxin-zheng
    Collaborator

    Hi @myron , for the Item 3, I am wondering where the change should happen. Do you think it should be making monai.transforms.VoteEnsemble support one-hot encoding input?

  3. myron commented on Dec 13, 2022

    @myron
    CollaboratorAuthor

    VoteEnsemble

    I AlgoEnemble.ensemble_pred()

                classes = [prob2class(p, dim=0, keepdim=True, sigmoid=False) for p in preds]
                return VoteEnsemble(num_classes=preds[0].shape[0])(classes)
    

    it converts to classes fo sigmoid==False (hardcoded).
    there should be an alternative for sigmoid==True

    • get classes as classes = [prob2class(p, dim=0, keepdim=True, sigmoid=True) for p in preds] (size PxCxHxWxD), then you can rearrange, for each channel , may be like this
     for i in range(classes.shape[1]):
            classes_i = [c[:,[i]] for c in range(len(classes))]
             v_i=VoteEnsemble(...)(classes_i)
     return concat/stack(v_i)...
    
  4. myron commented on Dec 13, 2022

    @myron
    CollaboratorAuthor

    PS: I also added issue 5 (above) @mingxin-zheng

  5. mingxin-zheng commented on Dec 14, 2022

    @mingxin-zheng
    Collaborator

    @myron Thanks for the explanation. Before I ran into issues using VoteEnsemble with one-hot encoding. After checking the code details, I think the way to VoteEnsemble in my PR should address and give equivalent results.

  6. mingxin-zheng commented on Dec 14, 2022

    @mingxin-zheng
    Collaborator

    PS: I also added issue 5 (above) @mingxin-zheng

    Issue 5 was also addressed. Now it would post a warning instead of raising an error.

  7. myron commented on Dec 14, 2022

    @myron
    CollaboratorAuthor

    @mingxin-zheng ok, thank you. I've added 6th item (it's non-urgent)

  8. mingxin-zheng commented on Dec 15, 2022

    @mingxin-zheng
    Collaborator

    @myron Thanks for the suggestion. When the saving should happen, right after infer_instance.predict()?

    Also, item 2 is fixed, but this enhancement isn't implemented yet, just fyi
    "improve AlgoEnsembleBestByFold to return N best from each fold. Currently it returns only 1 best result, but we need it to return for e.g. N=2, or N=3)"

  9. myron commented on Dec 15, 2022

    @myron
    CollaboratorAuthor

    @myron Thanks for the suggestion. When the saving should happen, right after infer_instance.predict()?

    Also, item 2 is fixed, but this enhancement isn't implemented yet, just fyi "improve AlgoEnsembleBestByFold to return N best from each fold. Currently it returns only 1 best result, but we need it to return for e.g. N=2, or N=3)"

    no, after the self.ensemble_pred()
    https://github.com/Project-MONAI/MONAI/blob/dev/monai/apps/auto3dseg/ensemble_builder.py#L162

    It will require some re-design to be for each file ( infer all models, ensemble, and save). Currently, we have for each file (infer all models, ensemble), then again for each file (save)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions