Skip to content

test error auto3dseg/notebooks/auto3dseg_hello_world.ipynb #1059

Closed

Description

@wyli
[2022-11-24T02:39:19.982Z] Running ./auto3dseg/notebooks/auto3dseg_hello_world.ipynb
[2022-11-24T02:39:19.982Z] Checking PEP8 compliance...
[2022-11-24T02:39:20.908Z] Running notebook...
[2022-11-24T02:39:20.908Z] Before:
[2022-11-24T02:39:20.908Z]     "max_epochs = 2\n",
[2022-11-24T02:39:20.908Z] After:
[2022-11-24T02:39:20.908Z]     "max_epochs = 1\n",
[2022-11-24T02:39:25.067Z] MONAI version: 1.0.0+130.gdd178ea3
[2022-11-24T02:39:25.067Z] Numpy version: 1.22.2
[2022-11-24T02:39:25.067Z] Pytorch version: 1.10.2+cu102
[2022-11-24T02:39:25.067Z] MONAI flags: HAS_EXT = False, USE_COMPILED = False, USE_META_DICT = False
[2022-11-24T02:39:25.067Z] MONAI rev id: dd178ea3bcaadbe6e3da964acc3bb4c027383fbf
[2022-11-24T02:39:25.067Z] MONAI __file__: /home/jenkins/agent/workspace/Monai-notebooks/MONAI/monai/__init__.py
[2022-11-24T02:39:25.067Z] 
[2022-11-24T02:39:25.067Z] Optional dependencies:
[2022-11-24T02:39:25.067Z] Pytorch Ignite version: 0.4.10
[2022-11-24T02:39:25.067Z] Nibabel version: 4.0.2
[2022-11-24T02:39:25.067Z] scikit-image version: 0.19.3
[2022-11-24T02:39:25.067Z] Pillow version: 7.0.0
[2022-11-24T02:39:25.067Z] Tensorboard version: 2.11.0
[2022-11-24T02:39:25.067Z] gdown version: 4.5.3
[2022-11-24T02:39:25.067Z] TorchVision version: 0.11.3+cu102
[2022-11-24T02:39:25.067Z] tqdm version: 4.64.1
[2022-11-24T02:39:25.067Z] lmdb version: 1.3.0
[2022-11-24T02:39:25.067Z] psutil version: 5.9.2
[2022-11-24T02:39:25.067Z] pandas version: 1.1.5
[2022-11-24T02:39:25.067Z] einops version: 0.6.0
[2022-11-24T02:39:25.067Z] transformers version: 4.21.3
[2022-11-24T02:39:25.067Z] mlflow version: 2.0.1
[2022-11-24T02:39:25.067Z] pynrrd version: 1.0.0
[2022-11-24T02:39:25.067Z] 
[2022-11-24T02:39:25.067Z] For details about installing the optional dependencies, please visit:
[2022-11-24T02:39:25.067Z]     https://docs.monai.io/en/latest/installation.html#installing-the-recommended-dependencies
[2022-11-24T02:39:25.067Z] 
[2022-11-24T02:39:25.993Z] /opt/conda/lib/python3.8/site-packages/papermill/iorw.py:153: UserWarning: the file is not specified with any extension : -
[2022-11-24T02:39:25.993Z]   warnings.warn(
[2022-11-24T02:39:51.259Z] 
Executing:   0%|          | 0/20 [00:00<?, ?cell/s]
Executing:   5%|▌         | 1/20 [00:01<00:30,  1.60s/cell]
Executing:  15%|█▌        | 3/20 [00:06<00:36,  2.15s/cell]
Executing:  25%|██▌       | 5/20 [00:09<00:29,  1.94s/cell]
Executing:  45%|████▌     | 9/20 [00:10<00:09,  1.18cell/s]
Executing:  55%|█████▌    | 11/20 [00:10<00:06,  1.49cell/s]2022-11-24 02:39:50.821023: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F AVX512_VNNI FMA
[2022-11-24T02:39:51.259Z] To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
[2022-11-24T02:39:51.259Z] 2022-11-24 02:39:50.957900: I tensorflow/core/util/port.cc:104] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
[2022-11-24T02:39:51.821Z] 2022-11-24 02:39:51.622650: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/conda/lib/python3.8/site-packages/torch/lib:/opt/conda/lib/python3.8/site-packages/torch_tensorrt/lib:/usr/local/cuda/compat/lib:/usr/local/nvidia/lib:/usr/local/nvidia/lib64
[2022-11-24T02:39:51.821Z] 2022-11-24 02:39:51.622738: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/conda/lib/python3.8/site-packages/torch/lib:/opt/conda/lib/python3.8/site-packages/torch_tensorrt/lib:/usr/local/cuda/compat/lib:/usr/local/nvidia/lib:/usr/local/nvidia/lib64
[2022-11-24T02:39:51.821Z] 2022-11-24 02:39:51.622745: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
[2022-11-24T02:39:53.184Z] Default upsampling behavior when mode=trilinear is changed to align_corners=False since 0.4.0. Please specify align_corners=True if the old behavior is desired. See the documentation of nn.Upsample for details.
[2022-11-24T02:39:56.451Z] The default behavior for interpolate/upsample with float scale_factor changed in 1.6.0 to align with other frameworks/libraries, and now uses scale_factor directly, instead of relying on the computed output size. If you wish to restore the old behavior, please set recompute_scale_factor=True. See the documentation of nn.Upsample for details. 
[2022-11-24T02:39:56.451Z] Default upsampling behavior when mode=trilinear is changed to align_corners=False since 0.4.0. Please specify align_corners=True if the old behavior is desired. See the documentation of nn.Upsample for details.
[2022-11-24T02:40:14.493Z] <function compute_meandice at 0x7fd2c06d00d0>: Function `compute_meandice` has been deprecated since version 1.0.0. use `compute_dice` instead.
[2022-11-24T02:40:14.493Z] Default upsampling behavior when mode=trilinear is changed to align_corners=False since 0.4.0. Please specify align_corners=True if the old behavior is desired. See the documentation of nn.Upsample for details.
[2022-11-24T02:40:16.378Z] Default upsampling behavior when mode=trilinear is changed to align_corners=False since 0.4.0. Please specify align_corners=True if the old behavior is desired. See the documentation of nn.Upsample for details.
[2022-11-24T02:40:16.378Z] Default upsampling behavior when mode=trilinear is changed to align_corners=False since 0.4.0. Please specify align_corners=True if the old behavior is desired. See the documentation of nn.Upsample for details.
[2022-11-24T02:40:22.911Z] Default upsampling behavior when mode=trilinear is changed to align_corners=False since 0.4.0. Please specify align_corners=True if the old behavior is desired. See the documentation of nn.Upsample for details.
[2022-11-24T02:40:54.936Z] 2022-11-24 02:40:49.709860: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F AVX512_VNNI FMA
[2022-11-24T02:40:54.936Z] To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
[2022-11-24T02:40:54.936Z] 2022-11-24 02:40:49.857869: I tensorflow/core/util/port.cc:104] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
[2022-11-24T02:40:54.936Z] 2022-11-24 02:40:50.566805: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/conda/lib/python3.8/site-packages/torch/lib:/opt/conda/lib/python3.8/site-packages/torch_tensorrt/lib:/usr/local/cuda/compat/lib:/usr/local/nvidia/lib:/usr/local/nvidia/lib64
[2022-11-24T02:40:54.936Z] 2022-11-24 02:40:50.566890: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/conda/lib/python3.8/site-packages/torch/lib:/opt/conda/lib/python3.8/site-packages/torch_tensorrt/lib:/usr/local/cuda/compat/lib:/usr/local/nvidia/lib:/usr/local/nvidia/lib64
[2022-11-24T02:40:54.936Z] 2022-11-24 02:40:50.566898: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
[2022-11-24T02:40:56.824Z] The default behavior for interpolate/upsample with float scale_factor changed in 1.6.0 to align with other frameworks/libraries, and now uses scale_factor directly, instead of relying on the computed output size. If you wish to restore the old behavior, please set recompute_scale_factor=True. See the documentation of nn.Upsample for details. 
[2022-11-24T02:40:56.824Z] <function compute_meandice at 0x7feb8f633a60>: Function `compute_meandice` has been deprecated since version 1.0.0. use `compute_dice` instead.
[2022-11-24T02:41:00.985Z] Default upsampling behavior when mode=trilinear is changed to align_corners=False since 0.4.0. Please specify align_corners=True if the old behavior is desired. See the documentation of nn.Upsample for details.
[2022-11-24T02:41:00.985Z] Default upsampling behavior when mode=trilinear is changed to align_corners=False since 0.4.0. Please specify align_corners=True if the old behavior is desired. See the documentation of nn.Upsample for details.
[2022-11-24T02:41:13.153Z] 2022-11-24 02:41:11.934842: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F AVX512_VNNI FMA
[2022-11-24T02:41:13.153Z] To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
[2022-11-24T02:41:13.153Z] 2022-11-24 02:41:12.074795: I tensorflow/core/util/port.cc:104] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
[2022-11-24T02:41:13.153Z] 2022-11-24 02:41:12.767170: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/conda/lib/python3.8/site-packages/torch/lib:/opt/conda/lib/python3.8/site-packages/torch_tensorrt/lib:/usr/local/cuda/compat/lib:/usr/local/nvidia/lib:/usr/local/nvidia/lib64
[2022-11-24T02:41:13.153Z] 2022-11-24 02:41:12.767249: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/conda/lib/python3.8/site-packages/torch/lib:/opt/conda/lib/python3.8/site-packages/torch_tensorrt/lib:/usr/local/cuda/compat/lib:/usr/local/nvidia/lib:/usr/local/nvidia/lib64
[2022-11-24T02:41:13.154Z] 2022-11-24 02:41:12.767257: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
[2022-11-24T02:41:14.518Z] Default upsampling behavior when mode=trilinear is changed to align_corners=False since 0.4.0. Please specify align_corners=True if the old behavior is desired. See the documentation of nn.Upsample for details.
[2022-11-24T02:41:16.405Z] <function compute_meandice at 0x7f492bd26c10>: Function `compute_meandice` has been deprecated since version 1.0.0. use `compute_dice` instead.
[2022-11-24T02:41:17.331Z] Default upsampling behavior when mode=trilinear is changed to align_corners=False since 0.4.0. Please specify align_corners=True if the old behavior is desired. See the documentation of nn.Upsample for details.
[2022-11-24T02:41:17.331Z] Default upsampling behavior when mode=trilinear is changed to align_corners=False since 0.4.0. Please specify align_corners=True if the old behavior is desired. See the documentation of nn.Upsample for details.
[2022-11-24T02:41:29.493Z] 2022-11-24 02:41:28.419269: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F AVX512_VNNI FMA
[2022-11-24T02:41:29.493Z] To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
[2022-11-24T02:41:29.493Z] 2022-11-24 02:41:28.563346: I tensorflow/core/util/port.cc:104] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
[2022-11-24T02:41:29.493Z] 2022-11-24 02:41:29.248034: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/conda/lib/python3.8/site-packages/torch/lib:/opt/conda/lib/python3.8/site-packages/torch_tensorrt/lib:/usr/local/cuda/compat/lib:/usr/local/nvidia/lib:/usr/local/nvidia/lib64
[2022-11-24T02:41:29.493Z] 2022-11-24 02:41:29.248128: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/conda/lib/python3.8/site-packages/torch/lib:/opt/conda/lib/python3.8/site-packages/torch_tensorrt/lib:/usr/local/cuda/compat/lib:/usr/local/nvidia/lib:/usr/local/nvidia/lib64
[2022-11-24T02:41:29.493Z] 2022-11-24 02:41:29.248136: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
[2022-11-24T02:41:32.024Z] <function compute_meandice at 0x7f0bcfd58820>: Function `compute_meandice` has been deprecated since version 1.0.0. use `compute_dice` instead.
[2022-11-24T02:41:44.185Z] 2022-11-24 02:41:43.125331: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F AVX512_VNNI FMA
[2022-11-24T02:41:44.185Z] To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
[2022-11-24T02:41:44.185Z] 2022-11-24 02:41:43.264840: I tensorflow/core/util/port.cc:104] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
[2022-11-24T02:41:44.185Z] 2022-11-24 02:41:43.942032: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/conda/lib/python3.8/site-packages/torch/lib:/opt/conda/lib/python3.8/site-packages/torch_tensorrt/lib:/usr/local/cuda/compat/lib:/usr/local/nvidia/lib:/usr/local/nvidia/lib64
[2022-11-24T02:41:44.185Z] 2022-11-24 02:41:43.942113: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/conda/lib/python3.8/site-packages/torch/lib:/opt/conda/lib/python3.8/site-packages/torch_tensorrt/lib:/usr/local/cuda/compat/lib:/usr/local/nvidia/lib:/usr/local/nvidia/lib64
[2022-11-24T02:41:44.185Z] 2022-11-24 02:41:43.942121: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
[2022-11-24T02:41:47.447Z] <function compute_meandice at 0x7f782f2ea940>: Function `compute_meandice` has been deprecated since version 1.0.0. use `compute_dice` instead.
[2022-11-24T02:41:59.534Z] 
Executing:  85%|████████▌ | 17/20 [02:30<00:40, 13.39s/cell]
Executing:  95%|█████████▌| 19/20 [02:31<00:10, 10.69s/cell]
Executing:  95%|█████████▌| 19/20 [02:33<00:08,  8.08s/cell]
[2022-11-24T02:41:59.534Z] /opt/conda/lib/python3.8/site-packages/papermill/iorw.py:153: UserWarning: the file is not specified with any extension : -
[2022-11-24T02:41:59.534Z]   warnings.warn(
[2022-11-24T02:41:59.534Z] Traceback (most recent call last):
[2022-11-24T02:41:59.534Z]   File "/opt/conda/bin/papermill", line 8, in <module>
[2022-11-24T02:41:59.534Z]     sys.exit(papermill())
[2022-11-24T02:41:59.534Z]   File "/opt/conda/lib/python3.8/site-packages/click/core.py", line 1130, in __call__
[2022-11-24T02:41:59.534Z]     return self.main(*args, **kwargs)
[2022-11-24T02:41:59.534Z]   File "/opt/conda/lib/python3.8/site-packages/click/core.py", line 1055, in main
[2022-11-24T02:41:59.534Z]     rv = self.invoke(ctx)
[2022-11-24T02:41:59.534Z]   File "/opt/conda/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
[2022-11-24T02:41:59.534Z]     return ctx.invoke(self.callback, **ctx.params)
[2022-11-24T02:41:59.534Z]   File "/opt/conda/lib/python3.8/site-packages/click/core.py", line 760, in invoke
[2022-11-24T02:41:59.534Z]     return __callback(*args, **kwargs)
[2022-11-24T02:41:59.534Z]   File "/opt/conda/lib/python3.8/site-packages/click/decorators.py", line 26, in new_func
[2022-11-24T02:41:59.534Z]     return f(get_current_context(), *args, **kwargs)
[2022-11-24T02:41:59.534Z]   File "/opt/conda/lib/python3.8/site-packages/papermill/cli.py", line 250, in papermill
[2022-11-24T02:41:59.534Z]     execute_notebook(
[2022-11-24T02:41:59.534Z]   File "/opt/conda/lib/python3.8/site-packages/papermill/execute.py", line 128, in execute_notebook
[2022-11-24T02:41:59.534Z]     raise_for_execution_errors(nb, output_path)
[2022-11-24T02:41:59.534Z]   File "/opt/conda/lib/python3.8/site-packages/papermill/execute.py", line 232, in raise_for_execution_errors
[2022-11-24T02:41:59.534Z]     raise error
[2022-11-24T02:41:59.534Z] papermill.exceptions.PapermillExecutionError: 
[2022-11-24T02:41:59.534Z] ---------------------------------------------------------------------------
[2022-11-24T02:41:59.534Z] Exception encountered at "In [9]":
[2022-11-24T02:41:59.534Z] ---------------------------------------------------------------------------
[2022-11-24T02:41:59.534Z] IndexError                                Traceback (most recent call last)
[2022-11-24T02:41:59.534Z] Cell In [9], line 14
[2022-11-24T02:41:59.534Z]      12 cbar = plt.colorbar(shrink=0.8)
[2022-11-24T02:41:59.534Z]      13 plt.subplot(2, 2, 3)
[2022-11-24T02:41:59.534Z] ---> 14 plt.imshow(pred[32, :, :, 0])
[2022-11-24T02:41:59.534Z]      15 plt.title('background class prediction')
[2022-11-24T02:41:59.534Z]      16 cbar = plt.colorbar(shrink=0.8)
[2022-11-24T02:41:59.534Z] 
[2022-11-24T02:41:59.534Z] IndexError: too many indices for array: array is 3-dimensional, but 4 were indexed

Activity

  1. wyli commented on Nov 24, 2022

    @wyli
    ContributorAuthor

    is this introduced by Project-MONAI/MONAI#5562? could you please have a look @mingxin-zheng @myron

  2. myron commented on Nov 24, 2022

    @myron
    Contributor

    I think the issue that predictions are saved properly as 3D images now, and not 4D.
    so the plot should be
    plt.imshow(pred[32, :, :]==0) instead of plt.imshow(pred[32, :, :, 0])
    and
    plt.imshow(pred[32, :, :]==1) instead of plt.imshow(pred[32, :, :, 1])
    @mingxin-zheng can you test it please, I don't have an easy way to test at the moment

  3. mingxin-zheng commented on Nov 24, 2022

    @mingxin-zheng
    Contributor

    @wyli @myron Since the output is not default to one-hot encoding, it breaks the imshow lines. My fix should work in either way (1.0.1 and the dev branch).

  4. added a commit that references this issue on Nov 24, 2022
    2046080
  5. added a commit that references this issue on Apr 21, 2024
    d21eb41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions