Repository navigation
Closed
Description
Describe the bug
logs using pytorch image 22.04 https://github.com/Project-MONAI/MONAI/runs/6306011370?check_suite_focus=true
(| NVIDIA-SMI 495.29.05 Driver Version: 495.29.05 CUDA Version: 11.5 )
ERROR: test_consistency_0 (tests.test_dynunet.TestDynUNetWithInstanceNorm3dNVFuser)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/opt/conda/lib/python3.8/site-packages/parameterized/parameterized.py", line 533, in standalone_func
test_compute_0 (tests.test_prepare_batch_default_dist.DistributedPrepareBatchDefault) (10.5s)
test_compute_1 (tests.test_prepare_batch_default_dist.DistributedPrepareBatchDefault) (10.6s)
test_verify_0__tmp_tmp_HuvzKkaSgJ_tests_testing_data_metadata_json (tests.test_bundle_verify_net.TestVerifyNetwork) (11.2s)
return func(*(a + p.args), **p.kwargs)
File "/tmp/tmp.HuvzKkaSgJ/tests/test_dynunet.py", line 146, in test_consistency
result_fuser = net_fuser(input_tensor)
File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl
return forward_call(*input, **kwargs)
File "/opt/conda/lib/python3.8/site-packages/monai/networks/nets/dynunet.py", line 268, in forward
out = self.skip_layers(x)
File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl
return forward_call(*input, **kwargs)
File "/opt/conda/lib/python3.8/site-packages/monai/networks/nets/dynunet.py", line 46, in forward
downout = self.downsample(x)
File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl
return forward_call(*input, **kwargs)
File "/opt/conda/lib/python3.8/site-packages/monai/networks/blocks/dynunet_block.py", line 80, in forward
out = self.norm1(out)
File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl
return forward_call(*input, **kwargs)
File "/opt/conda/lib/python3.8/site-packages/apex/normalization/instance_norm.py", line 138, in forward
out = InstanceNormNVFuserFunction.apply(
File "/opt/conda/lib/python3.8/site-packages/apex/normalization/instance_norm.py", line 16, in forward
instance_norm_nvfuser_cuda = importlib.import_module("instance_norm_nvfuser_cuda")
File "/opt/conda/lib/python3.8/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
File "<frozen importlib._bootstrap>", line 991, in _find_and_load
File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 657, in _load_unlocked
File "<frozen importlib._bootstrap>", line 556, in module_from_spec
File "<frozen importlib._bootstrap_external>", line 1166, in create_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
ImportError: /opt/conda/lib/python3.8/site-packages/instance_norm_nvfuser_cuda.cpython-38-x86_64-linux-gnu.so: undefined symbol: _ZTVN5torch3jit5fuser4cuda3kir6KernelE
----------------------------------------------------------------------Metadata
Metadata
Assignees
Labels
No labels
Activity
Nic-Ma commentedon May 5, 2022
Hi @yiheng-wang-nv ,
Could you please help investigate and fix this
InstanceNorm3dNVFuserissue in PyTorch docker 22.04?Thanks in advance.
yiheng-wang-nv commentedon May 9, 2022
Hi @Nic-Ma and @wyli , my initial thought is that the issue is due to the driver version, with driver version
510.47.03, docker 22.04 does not raise the same issue. For the ci/cd machine, is it possible to test with different driver version?wyli commentedon May 9, 2022
thanks, I think blossom tests are with the different drivers. could you please discuss this with @pxLi? (there's no need to update the github CI because we aim to use the most compatible driver to cover as many cases as possible)
at the same time it seems this condition is not good enough to detect a suitable environment, could you
MONAI/monai/networks/layers/factories.py
Line 266 in 209db9e
pxLi commentedon May 10, 2022
FYI internal blossom hosts are mostly w/ driver 465.07. This depends on your guys what driver versions we should cover in tests. Please let me know if you have specific requirement, we could try find available machines in public pool to test before actually asking blossom guys to help modify drivers version on our entitled machines, thanks
yiheng-wang-nv commentedon May 11, 2022
Updates: I checked more about this issue, and with Peixin's help, we found that it may not the cuda/driver/torch version/docker issue, same environment is used for the
cron-dockertest (https://github.com/Project-MONAI/MONAI/runs/6306012243?check_suite_focus=true) and the docker image is based on 22.04.So far, the issue is avoided by adding more checks, but I'm not sure if there is any other issue within the env building part of
cron-pip.wyli commentedon May 11, 2022
thanks for looking into this, now it's clear that
reinstalling pytorch/torchvision in the docker breaks the apex customized module
MONAI/.github/workflows/cron.yml
Line 134 in 557c129
MONAI/.github/workflows/cron.yml
Line 149 in 557c129
and Yiheng's PR makes the call robust in this case #4241