You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
test_integration_autorunner.py" -> test_autorunner_hpo(self) , test for HPO. On my local machine it runs forever, and get stuck without any progress (no CPU or GPU usage) #5568
There is one integration test, I' unable to test in "test_integration_autorunner.py" -> test_autorunner_hpo(self) , test for HPO. On my local machine it runs forever, and get stuck without any progress (no CPU or GPU usage). Same behavior on old dev branch and on current (so it's unrelated to this PR). So I've test everything except for HPO test there.
hi @mingxin-zheng@Nic-Ma, do you have time to simplify test_autorunner_hpo? the way it creates nni server and monitors the file system changes is not reliable and difficult to debug.
Hi @wyli , yes, the autorunner test needs improvements. I would like to find out what is not needed and should be fixed first, and here are a few feasible options for us to gauge:
avoid having a web service in the integration test
finding a way to get trial information and fail the test appropiately
avoid running hpo with training. For example, just checking if the trial command is correct
thanks, maybe it's good enough to go for option 3 for now?
I have spent 5+ hours on option 2 but it's not easy, especially when we want to ensure the same testing behaviour across different testing environments
I support option 3, to avoid running hpo, since it's rather slow (perhaps we could run it only once a month, or not running it at all)
on top of being slow there is a small issue there, that on 8gpus in silently hangs (because the current toy dataset is small). so it's confirmed to work on 2gpus, but it's not clear if it works well on more than 2 gpus.
Activity
wyli commentedon Nov 23, 2022
hi @mingxin-zheng @Nic-Ma, do you have time to simplify
test_autorunner_hpo? the way it creates nni server and monitors the file system changes is not reliable and difficult to debug.mingxin-zheng commentedon Nov 23, 2022
Hi @wyli , yes, the autorunner test needs improvements. I would like to find out what is not needed and should be fixed first, and here are a few feasible options for us to gauge:
wyli commentedon Nov 23, 2022
thanks, maybe it's good enough to go for option 3 for now?
I have spent 5+ hours on option 2 but it's not easy, especially when we want to ensure the same testing behaviour across different testing environments
myron commentedon Nov 23, 2022
I support option 3, to avoid running hpo, since it's rather slow (perhaps we could run it only once a month, or not running it at all)
on top of being slow there is a small issue there, that on 8gpus in silently hangs (because the current toy dataset is small). so it's confirmed to work on 2gpus, but it's not clear if it works well on more than 2 gpus.
mingxin-zheng commentedon Dec 8, 2022
@myron the 8-GPU is a known issue and will be fixed in another PR. But the fix won't work on a 16-GPU system until we update the algorithm templates.
Add dryrun mode for AutoRunner HPO to fix 5568 (#5681)