Skip to content

integration TestDynUNetWithInstanceNorm3dNVFuser #4235

Closed

Description

@wyli

Describe the bug
logs using pytorch image 22.04 https://github.com/Project-MONAI/MONAI/runs/6306011370?check_suite_focus=true

(| NVIDIA-SMI 495.29.05 Driver Version: 495.29.05 CUDA Version: 11.5 )

ERROR: test_consistency_0 (tests.test_dynunet.TestDynUNetWithInstanceNorm3dNVFuser)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/opt/conda/lib/python3.8/site-packages/parameterized/parameterized.py", line 533, in standalone_func
test_compute_0 (tests.test_prepare_batch_default_dist.DistributedPrepareBatchDefault) (10.5s)
test_compute_1 (tests.test_prepare_batch_default_dist.DistributedPrepareBatchDefault) (10.6s)
test_verify_0__tmp_tmp_HuvzKkaSgJ_tests_testing_data_metadata_json (tests.test_bundle_verify_net.TestVerifyNetwork) (11.2s)
    return func(*(a + p.args), **p.kwargs)
  File "/tmp/tmp.HuvzKkaSgJ/tests/test_dynunet.py", line 146, in test_consistency
    result_fuser = net_fuser(input_tensor)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl
    return forward_call(*input, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/monai/networks/nets/dynunet.py", line 268, in forward
    out = self.skip_layers(x)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl
    return forward_call(*input, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/monai/networks/nets/dynunet.py", line 46, in forward
    downout = self.downsample(x)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl
    return forward_call(*input, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/monai/networks/blocks/dynunet_block.py", line 80, in forward
    out = self.norm1(out)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl
    return forward_call(*input, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/apex/normalization/instance_norm.py", line 138, in forward
    out = InstanceNormNVFuserFunction.apply(
  File "/opt/conda/lib/python3.8/site-packages/apex/normalization/instance_norm.py", line 16, in forward
    instance_norm_nvfuser_cuda = importlib.import_module("instance_norm_nvfuser_cuda")
  File "/opt/conda/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 657, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 556, in module_from_spec
  File "<frozen importlib._bootstrap_external>", line 1166, in create_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
ImportError: /opt/conda/lib/python3.8/site-packages/instance_norm_nvfuser_cuda.cpython-38-x86_64-linux-gnu.so: undefined symbol: _ZTVN5torch3jit5fuser4cuda3kir6KernelE

----------------------------------------------------------------------

Activity

  1. Nic-Ma commented on May 5, 2022

    @Nic-Ma
    Contributor

    Hi @yiheng-wang-nv ,

    Could you please help investigate and fix this InstanceNorm3dNVFuser issue in PyTorch docker 22.04?

    Thanks in advance.

  2. yiheng-wang-nv commented on May 9, 2022

    @yiheng-wang-nv
    Contributor

    Hi @Nic-Ma and @wyli , my initial thought is that the issue is due to the driver version, with driver version 510.47.03, docker 22.04 does not raise the same issue. For the ci/cd machine, is it possible to test with different driver version?

  3. wyli commented on May 9, 2022

    @wyli
    ContributorAuthor

    thanks, I think blossom tests are with the different drivers. could you please discuss this with @pxLi? (there's no need to update the github CI because we aim to use the most compatible driver to cover as many cases as possible)

    at the same time it seems this condition is not good enough to detect a suitable environment, could you

    if not has_nvfuser:
    could you help improve?

  4. pxLi commented on May 10, 2022

    @pxLi
    Contributor

    FYI internal blossom hosts are mostly w/ driver 465.07. This depends on your guys what driver versions we should cover in tests. Please let me know if you have specific requirement, we could try find available machines in public pool to test before actually asking blossom guys to help modify drivers version on our entitled machines, thanks

  5. yiheng-wang-nv commented on May 11, 2022

    @yiheng-wang-nv
    Contributor

    Hi @Nic-Ma and @wyli , my initial thought is that the issue is due to the driver version, with driver version 510.47.03, docker 22.04 does not raise the same issue. For the ci/cd machine, is it possible to test with different driver version?

    Updates: I checked more about this issue, and with Peixin's help, we found that it may not the cuda/driver/torch version/docker issue, same environment is used for the cron-docker test (https://github.com/Project-MONAI/MONAI/runs/6306012243?check_suite_focus=true) and the docker image is based on 22.04.
    So far, the issue is avoided by adding more checks, but I'm not sure if there is any other issue within the env building part of cron-pip.

  6. wyli commented on May 11, 2022

    @wyli
    ContributorAuthor

    Updates: I checked more about this issue, and with Peixin's help, we found that it may not the cuda/driver/torch version/docker issue, same environment is used for the cron-docker test (https://github.com/Project-MONAI/MONAI/runs/6306012243?check_suite_focus=true) and the docker image is based on 22.04. So far, the issue is avoided by adding more checks, but I'm not sure if there is any other issue within the env building part of cron-pip.

    thanks for looking into this, now it's clear that
    reinstalling pytorch/torchvision in the docker breaks the apex customized module

    python -m pip uninstall -y torch torchvision

    python -m pip install $name[all]

    and Yiheng's PR makes the call robust in this case #4241

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions