You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Output From Failed subprocess.run Calls Not Captured #6445
Describe the bug
In a few places such as heresubprocess.run is called without any checks that a non-zero result was returned. In these situations the stdout/stderr from the call is lost and so doesn't appear in the output logs when CI/CD tests are run. It's difficult to reproduce the environment the test is run under so having this output is important for debugging when PR tests fail.
The solution is to use subprocess.run with capture_output=True and stderr=subprocess.STDOUT so that the raise exception will have that information. The exception should perhaps be caught and the output value it stores printed to stderr before being reraised. Other places where run or Popen are used might need modification as well.
@myron@mingxin-zheng I think this is code that you've been involved in developing? This is related to an issue we're having with tests in #6257 I believe.
Hi @ericspod , capture_output=True was firstly introduced and then changed in #5523 . These subprocess calls are for network training, which takes a long time to run. When capture_output was set to true, the outputs/prints from the subprocess are only displayed after completion, and so users complained they can't see any progress for hours.
@mingxin-zheng I see, that could be a real bad thing for users. How long do the tests run? Perhaps we need some way of setting the capture on for tests only. Thanks!
@ericspod That's a good idea. The tests run pretty fast usually. I think it is less than 3 minutes for each related unit test. The integration tests could take a bit longer but it is no more than 10 minutes.
Hi @mingxin-zheng One of the tests that fails is this one so maybe adding a flag to run_algo to do capture would fix the issue for us. Also study.optimize has a similar error. These might be minor issues with the state of the CI/CD and something that'll be fixed in time but I can't tell without output.
Let's put this into a separate utility function we can use in multiple places in the general case. This allows setting capture_output explicitly if desired but will override it with the debug status from the variable:
Thanks @ericspod . I started a PR for the discussed solution. There were quite many changes and tests for 1.2 release. Since this is mostly for debugging purposes, I did not work on it until now.
Hi @mingxin-zheng Looking into this would be great thanks. We were able to sort the CICD issues relating to this for the PRs we had for 1.2 but it would be great to improve this now.
Activity
ericspod commentedon Apr 27, 2023
@myron @mingxin-zheng I think this is code that you've been involved in developing? This is related to an issue we're having with tests in #6257 I believe.
mingxin-zheng commentedon Apr 27, 2023
Hi @ericspod ,
capture_output=Truewas firstly introduced and then changed in #5523 . These subprocess calls are for network training, which takes a long time to run. Whencapture_outputwas set to true, the outputs/prints from the subprocess are only displayed after completion, and so users complained they can't see any progress for hours.ericspod commentedon Apr 27, 2023
@mingxin-zheng I see, that could be a real bad thing for users. How long do the tests run? Perhaps we need some way of setting the capture on for tests only. Thanks!
mingxin-zheng commentedon Apr 27, 2023
@ericspod That's a good idea. The tests run pretty fast usually. I think it is less than 3 minutes for each related unit test. The integration tests could take a bit longer but it is no more than 10 minutes.
ericspod commentedon Apr 27, 2023
Hi @mingxin-zheng One of the tests that fails is this one so maybe adding a flag to
run_algoto do capture would fix the issue for us. Also study.optimize has a similar error. These might be minor issues with the state of the CI/CD and something that'll be fixed in time but I can't tell without output.mingxin-zheng commentedon Apr 28, 2023
One implementation I have in mind now is to use the MONAI_DEBUG env var:
MONAI/monai/utils/misc.py
Line 519 in d23221f
Like other Auto3DSeg trainings, the chain of calls for
run_algois:run_algo-train-_run_cmd-subprocess.runIn my mind, it would be something like this in
_run_cmd:If this satisfy the need in this ticket @ericspod @myron @wyli , I can move forward to create a PR.
ericspod commentedon Apr 28, 2023
Let's put this into a separate utility function we can use in multiple places in the general case. This allows setting
capture_outputexplicitly if desired but will override it with the debug status from the variable:mingxin-zheng commentedon Jun 9, 2023
Thanks @ericspod . I started a PR for the discussed solution. There were quite many changes and tests for 1.2 release. Since this is mostly for debugging purposes, I did not work on it until now.
ericspod commentedon Jun 9, 2023
Hi @mingxin-zheng Looking into this would be great thanks. We were able to sort the CICD issues relating to this for the PRs we had for 1.2 but it would be great to improve this now.
Capture failures from auto3dseg related subprocess calls (#6596)