You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Auto3DSeg should separate bcprun as an unique execution method #6567
Is your feature request related to a problem? Please describe.
Currently, bcprun is the execution method of Auto3DSeg multi-node, I think it's not very straightforward: https://github.com/Project-MONAI/MONAI/blob/dev/monai/apps/auto3dseg/bundle_gen.py#L220-L222
Better to make it a dependent execution method of Auto3DSeg, supporting single GPU, multi-GPU, and multi-node.
Then it's in parallel with the default torchrun method, and we may also extend other executions like AWS, Azure, etc.
When we need to extend it to other platforms, such as AzureML, we can add:
# if we need to extend support to other frameworks, we can do something like this:fromazureml.coreimportWorkspace, Experiment, ScriptRunConfigdefazureml_prepare_fn(script, workspace=None):
workspace=Workspace.from_config() ifworkspaceisNoneelseworkspacereturnExperiment(workspace, '<experiment-name>')
defazureml_run_fn(experiment, config):
returnexperiment.submit(config)
ifmn_start_method=="azureml":
config=ScriptRunConfig(source_directory=<'<path/to/script>'>, script='train.py', ...)
azureml_job=FlexibleDistJob(azureml_run_fn, azureml_prepare_fn)
azureml_job.run(config=config)
Thanks for the proposal, I personally feel don't need to define the FlexibleDistJob class, I see no requirements for a "class" here (no big member variables, no state control <like time-consuming init, run, finalize>, no reusable data, etc.).
A bunch of XXX_prepare and XXX_run functions are enough.
What do you think?
The class is minimal now because we are using command line to run jobs. For azureml or sagemaker, the return variables of XXX_prepare and XXX_run are needed to monitor the jobs. So the FlexibleDistJob class can manage these together.
Activity
mingxin-zheng commentedon Jun 14, 2023
Hi @Nic-Ma @heyufan1995 , I want to propose a refractor as below:
When we need to use it for Auto3DSeg:
When we need to extend it to other platforms, such as AzureML, we can add:
Nic-Ma commentedon Jun 14, 2023
Hi @mingxin-zheng ,
Thanks for the proposal, I personally feel don't need to define the
FlexibleDistJobclass, I see no requirements for a "class" here (no big member variables, no state control <like time-consuming init, run, finalize>, no reusable data, etc.).A bunch of
XXX_prepareandXXX_runfunctions are enough.What do you think?
Thanks.
mingxin-zheng commentedon Jun 14, 2023
The class is minimal now because we are using command line to run jobs. For azureml or sagemaker, the return variables of
XXX_prepareandXXX_runare needed to monitor the jobs. So theFlexibleDistJobclass can manage these together.For example: https://azure.github.io/azureml-cheatsheets/docs/cheatsheets/python/v1/run/
wyli commentedon Jun 14, 2023
Agree that we can start with a set of simple functions and then make classes if needed, for example for monitoring the job status.
Adding this to the 1.3 feature discussion. btw the concept of
run()method has been used hereMONAI/monai/apps/auto3dseg/auto_runner.py
Lines 734 to 736 in a1c4371
mingxin-zheng commentedon Jun 18, 2023
Thanks @wyli @Nic-Ma . I will do it in the function way first.
Refactor multi-node running command into dedicated functions (#6623)