Skip to content

test_integration_autorunner.py" -> test_autorunner_hpo(self) , test for HPO. On my local machine it runs forever, and get stuck without any progress (no CPU or GPU usage) #5568

Closed

Description

@wyli

There is one integration test, I' unable to test in "test_integration_autorunner.py" -> test_autorunner_hpo(self) , test for HPO. On my local machine it runs forever, and get stuck without any progress (no CPU or GPU usage). Same behavior on old dev branch and on current (so it's unrelated to this PR). So I've test everything except for HPO test there.

Originally posted by @myron in #5562 (comment)

I also saw this issue multiple times with github runners, for example https://github.com/Project-MONAI/MONAI/actions/runs/3472091346/jobs/5802451135

Activity

  1. wyli commented on Nov 23, 2022

    @wyli
    ContributorAuthor

    hi @mingxin-zheng @Nic-Ma, do you have time to simplify test_autorunner_hpo? the way it creates nni server and monitors the file system changes is not reliable and difficult to debug.

  2. mingxin-zheng commented on Nov 23, 2022

    @mingxin-zheng
    Collaborator

    Hi @wyli , yes, the autorunner test needs improvements. I would like to find out what is not needed and should be fixed first, and here are a few feasible options for us to gauge:

    • avoid having a web service in the integration test
    • finding a way to get trial information and fail the test appropiately
    • avoid running hpo with training. For example, just checking if the trial command is correct
  3. wyli commented on Nov 23, 2022

    @wyli
    ContributorAuthor

    thanks, maybe it's good enough to go for option 3 for now?

    I have spent 5+ hours on option 2 but it's not easy, especially when we want to ensure the same testing behaviour across different testing environments

  4. myron commented on Nov 23, 2022

    @myron
    Collaborator

    I support option 3, to avoid running hpo, since it's rather slow (perhaps we could run it only once a month, or not running it at all)

    on top of being slow there is a small issue there, that on 8gpus in silently hangs (because the current toy dataset is small). so it's confirmed to work on 2gpus, but it's not clear if it works well on more than 2 gpus.

  5. mingxin-zheng commented on Dec 8, 2022

    @mingxin-zheng
    Collaborator

    @myron the 8-GPU is a known issue and will be fixed in another PR. But the fix won't work on a 16-GPU system until we update the algorithm templates.

  6. added a commit that references this issue on Dec 8, 2022
    b533a53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions