Skip to content

Auto3DSeg should separate bcprun as an unique execution method #6567

Closed

Description

@Nic-Ma

Is your feature request related to a problem? Please describe.
Currently, bcprun is the execution method of Auto3DSeg multi-node, I think it's not very straightforward:
https://github.com/Project-MONAI/MONAI/blob/dev/monai/apps/auto3dseg/bundle_gen.py#L220-L222
Better to make it a dependent execution method of Auto3DSeg, supporting single GPU, multi-GPU, and multi-node.
Then it's in parallel with the default torchrun method, and we may also extend other executions like AWS, Azure, etc.

Activity

  1. mingxin-zheng commented on Jun 14, 2023

    @mingxin-zheng
    Collaborator

    Hi @Nic-Ma @heyufan1995 , I want to propose a refractor as below:

    import subprocess
    def prepare_default(script, cmd_prefix=None, **kwargs):
            cmd = cmd_prefix
            if cmd is not None and not cmd.endswith(" "):
                cmd += " "
            cmd = "torchrun " if cmd is None else cmd
            # todo: add checks for num_nodes, nproc_per_node, etc.
            cmd += f"{script}"
            for k, v in kwargs.items():
                # todo: add check/support based on type(v)
                cmd += f" --{k} {str(v)}"
            return cmd
    
    def prepare_bcprun(script, cmd_prefix=None, **kwargs):
            bcprun_cmd = "bcprun "
            hyperparam = kwargs.copy()
            num_nodes = hyperparam.pop("n")
            n_devices = hyperparam.pop("p")
            bcprun_cmd += f"-n {num_nodes} -p {n_devices} "
            
            cmd_prefix = "python " if cmd_prefix is None else cmd_prefix
            if not cmd_prefix.endswith(" "):
                cmd_prefix += " "
    
            bcprun_cmd += cmd_prefix
            # todo: add checks for num_nodes, nproc_per_node, etc.
            cmd += f"{script}"
            for k, v in hyperparam.items():
                # todo: add check/support based on type(v)
                cmd += f" --{k} {str(v)}"
            return cmd
    
    def run_default(cmd):
        # todo: add checks for cmd
        # todo: add logging
        return subprocess.run(cmd, check=True, capture_output=True)
    
    
    class FlexibleDistJob:
        def __init__(self, run_fn=run_default, prepare_fn=prepare_default, **kwargs):
            self.run_fn = run_fn
            self._job = prepare_fn(**kwargs)
    
        def run(self, **kwargs):
            return self.run_fn(self._job, **kwargs)

    When we need to use it for Auto3DSeg:

    mn_start_method = ...
    num_nodes = ...
    nproc_per_node = ...
    
    if mn_start_method == "torchrun":
        dist_job = FlexibleDistJob(num_nodes=num_nodes, nproc_per_node=nproc_per_node)
        dist_job.run()
    elif mn_start_method == "bcprun":
        bcprun_job = FlexibleDistJob(prepare_fn=prepare_bcprun, n=num_nodes, p=nproc_per_node)
        bcprun_job.run()

    When we need to extend it to other platforms, such as AzureML, we can add:

    # if we need to extend support to other frameworks, we can do something like this:
    from azureml.core import Workspace, Experiment, ScriptRunConfig
    
    def azureml_prepare_fn(script, workspace=None):
        workspace = Workspace.from_config() if workspace is None else workspace
        return Experiment(workspace, '<experiment-name>')
    
    def azureml_run_fn(experiment, config):
        return experiment.submit(config)
    
    
    if mn_start_method == "azureml":
        config = ScriptRunConfig(source_directory=<'<path/to/script>'>, script='train.py', ...)
        azureml_job = FlexibleDistJob(azureml_run_fn, azureml_prepare_fn)
        azureml_job.run(config=config)
  2. Nic-Ma commented on Jun 14, 2023

    @Nic-Ma
    ContributorAuthor

    Hi @mingxin-zheng ,

    Thanks for the proposal, I personally feel don't need to define the FlexibleDistJob class, I see no requirements for a "class" here (no big member variables, no state control <like time-consuming init, run, finalize>, no reusable data, etc.).
    A bunch of XXX_prepare and XXX_run functions are enough.
    What do you think?

    Thanks.

  3. mingxin-zheng commented on Jun 14, 2023

    @mingxin-zheng
    Collaborator

    The class is minimal now because we are using command line to run jobs. For azureml or sagemaker, the return variables of XXX_prepare and XXX_run are needed to monitor the jobs. So the FlexibleDistJob class can manage these together.

    For example: https://azure.github.io/azureml-cheatsheets/docs/cheatsheets/python/v1/run/

  4. wyli commented on Jun 14, 2023

    @wyli
    Contributor

    Agree that we can start with a set of simple functions and then make classes if needed, for example for monitoring the job status.

    Adding this to the 1.3 feature discussion. btw the concept of run() method has been used here

    def run(self):
    """
    Run the AutoRunner pipeline
    I guess we need another name here. maybe job launcher or executor...

  5. mingxin-zheng commented on Jun 18, 2023

    @mingxin-zheng
    Collaborator

    Thanks @wyli @Nic-Ma . I will do it in the function way first.

  6. added a commit that references this issue on Jul 5, 2023
    c6c7ba9
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions