Repository navigation
Closed
Description
[2022-11-24T02:39:19.982Z] Running ./auto3dseg/notebooks/auto3dseg_hello_world.ipynb
[2022-11-24T02:39:19.982Z] Checking PEP8 compliance...
[2022-11-24T02:39:20.908Z] Running notebook...
[2022-11-24T02:39:20.908Z] Before:
[2022-11-24T02:39:20.908Z] "max_epochs = 2\n",
[2022-11-24T02:39:20.908Z] After:
[2022-11-24T02:39:20.908Z] "max_epochs = 1\n",
[2022-11-24T02:39:25.067Z] MONAI version: 1.0.0+130.gdd178ea3
[2022-11-24T02:39:25.067Z] Numpy version: 1.22.2
[2022-11-24T02:39:25.067Z] Pytorch version: 1.10.2+cu102
[2022-11-24T02:39:25.067Z] MONAI flags: HAS_EXT = False, USE_COMPILED = False, USE_META_DICT = False
[2022-11-24T02:39:25.067Z] MONAI rev id: dd178ea3bcaadbe6e3da964acc3bb4c027383fbf
[2022-11-24T02:39:25.067Z] MONAI __file__: /home/jenkins/agent/workspace/Monai-notebooks/MONAI/monai/__init__.py
[2022-11-24T02:39:25.067Z]
[2022-11-24T02:39:25.067Z] Optional dependencies:
[2022-11-24T02:39:25.067Z] Pytorch Ignite version: 0.4.10
[2022-11-24T02:39:25.067Z] Nibabel version: 4.0.2
[2022-11-24T02:39:25.067Z] scikit-image version: 0.19.3
[2022-11-24T02:39:25.067Z] Pillow version: 7.0.0
[2022-11-24T02:39:25.067Z] Tensorboard version: 2.11.0
[2022-11-24T02:39:25.067Z] gdown version: 4.5.3
[2022-11-24T02:39:25.067Z] TorchVision version: 0.11.3+cu102
[2022-11-24T02:39:25.067Z] tqdm version: 4.64.1
[2022-11-24T02:39:25.067Z] lmdb version: 1.3.0
[2022-11-24T02:39:25.067Z] psutil version: 5.9.2
[2022-11-24T02:39:25.067Z] pandas version: 1.1.5
[2022-11-24T02:39:25.067Z] einops version: 0.6.0
[2022-11-24T02:39:25.067Z] transformers version: 4.21.3
[2022-11-24T02:39:25.067Z] mlflow version: 2.0.1
[2022-11-24T02:39:25.067Z] pynrrd version: 1.0.0
[2022-11-24T02:39:25.067Z]
[2022-11-24T02:39:25.067Z] For details about installing the optional dependencies, please visit:
[2022-11-24T02:39:25.067Z] https://docs.monai.io/en/latest/installation.html#installing-the-recommended-dependencies
[2022-11-24T02:39:25.067Z]
[2022-11-24T02:39:25.993Z] /opt/conda/lib/python3.8/site-packages/papermill/iorw.py:153: UserWarning: the file is not specified with any extension : -
[2022-11-24T02:39:25.993Z] warnings.warn(
[2022-11-24T02:39:51.259Z]
Executing: 0%| | 0/20 [00:00<?, ?cell/s]
Executing: 5%|▌ | 1/20 [00:01<00:30, 1.60s/cell]
Executing: 15%|█▌ | 3/20 [00:06<00:36, 2.15s/cell]
Executing: 25%|██▌ | 5/20 [00:09<00:29, 1.94s/cell]
Executing: 45%|████▌ | 9/20 [00:10<00:09, 1.18cell/s]
Executing: 55%|█████▌ | 11/20 [00:10<00:06, 1.49cell/s]2022-11-24 02:39:50.821023: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 AVX512F AVX512_VNNI FMA
[2022-11-24T02:39:51.259Z] To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
[2022-11-24T02:39:51.259Z] 2022-11-24 02:39:50.957900: I tensorflow/core/util/port.cc:104] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
[2022-11-24T02:39:51.821Z] 2022-11-24 02:39:51.622650: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/conda/lib/python3.8/site-packages/torch/lib:/opt/conda/lib/python3.8/site-packages/torch_tensorrt/lib:/usr/local/cuda/compat/lib:/usr/local/nvidia/lib:/usr/local/nvidia/lib64
[2022-11-24T02:39:51.821Z] 2022-11-24 02:39:51.622738: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/conda/lib/python3.8/site-packages/torch/lib:/opt/conda/lib/python3.8/site-packages/torch_tensorrt/lib:/usr/local/cuda/compat/lib:/usr/local/nvidia/lib:/usr/local/nvidia/lib64
[2022-11-24T02:39:51.821Z] 2022-11-24 02:39:51.622745: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
[2022-11-24T02:39:53.184Z] Default upsampling behavior when mode=trilinear is changed to align_corners=False since 0.4.0. Please specify align_corners=True if the old behavior is desired. See the documentation of nn.Upsample for details.
[2022-11-24T02:39:56.451Z] The default behavior for interpolate/upsample with float scale_factor changed in 1.6.0 to align with other frameworks/libraries, and now uses scale_factor directly, instead of relying on the computed output size. If you wish to restore the old behavior, please set recompute_scale_factor=True. See the documentation of nn.Upsample for details.
[2022-11-24T02:39:56.451Z] Default upsampling behavior when mode=trilinear is changed to align_corners=False since 0.4.0. Please specify align_corners=True if the old behavior is desired. See the documentation of nn.Upsample for details.
[2022-11-24T02:40:14.493Z] <function compute_meandice at 0x7fd2c06d00d0>: Function `compute_meandice` has been deprecated since version 1.0.0. use `compute_dice` instead.
[2022-11-24T02:40:14.493Z] Default upsampling behavior when mode=trilinear is changed to align_corners=False since 0.4.0. Please specify align_corners=True if the old behavior is desired. See the documentation of nn.Upsample for details.
[2022-11-24T02:40:16.378Z] Default upsampling behavior when mode=trilinear is changed to align_corners=False since 0.4.0. Please specify align_corners=True if the old behavior is desired. See the documentation of nn.Upsample for details.
[2022-11-24T02:40:16.378Z] Default upsampling behavior when mode=trilinear is changed to align_corners=False since 0.4.0. Please specify align_corners=True if the old behavior is desired. See the documentation of nn.Upsample for details.
[2022-11-24T02:40:22.911Z] Default upsampling behavior when mode=trilinear is changed to align_corners=False since 0.4.0. Please specify align_corners=True if the old behavior is desired. See the documentation of nn.Upsample for details.
[2022-11-24T02:40:54.936Z] 2022-11-24 02:40:49.709860: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 AVX512F AVX512_VNNI FMA
[2022-11-24T02:40:54.936Z] To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
[2022-11-24T02:40:54.936Z] 2022-11-24 02:40:49.857869: I tensorflow/core/util/port.cc:104] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
[2022-11-24T02:40:54.936Z] 2022-11-24 02:40:50.566805: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/conda/lib/python3.8/site-packages/torch/lib:/opt/conda/lib/python3.8/site-packages/torch_tensorrt/lib:/usr/local/cuda/compat/lib:/usr/local/nvidia/lib:/usr/local/nvidia/lib64
[2022-11-24T02:40:54.936Z] 2022-11-24 02:40:50.566890: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/conda/lib/python3.8/site-packages/torch/lib:/opt/conda/lib/python3.8/site-packages/torch_tensorrt/lib:/usr/local/cuda/compat/lib:/usr/local/nvidia/lib:/usr/local/nvidia/lib64
[2022-11-24T02:40:54.936Z] 2022-11-24 02:40:50.566898: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
[2022-11-24T02:40:56.824Z] The default behavior for interpolate/upsample with float scale_factor changed in 1.6.0 to align with other frameworks/libraries, and now uses scale_factor directly, instead of relying on the computed output size. If you wish to restore the old behavior, please set recompute_scale_factor=True. See the documentation of nn.Upsample for details.
[2022-11-24T02:40:56.824Z] <function compute_meandice at 0x7feb8f633a60>: Function `compute_meandice` has been deprecated since version 1.0.0. use `compute_dice` instead.
[2022-11-24T02:41:00.985Z] Default upsampling behavior when mode=trilinear is changed to align_corners=False since 0.4.0. Please specify align_corners=True if the old behavior is desired. See the documentation of nn.Upsample for details.
[2022-11-24T02:41:00.985Z] Default upsampling behavior when mode=trilinear is changed to align_corners=False since 0.4.0. Please specify align_corners=True if the old behavior is desired. See the documentation of nn.Upsample for details.
[2022-11-24T02:41:13.153Z] 2022-11-24 02:41:11.934842: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 AVX512F AVX512_VNNI FMA
[2022-11-24T02:41:13.153Z] To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
[2022-11-24T02:41:13.153Z] 2022-11-24 02:41:12.074795: I tensorflow/core/util/port.cc:104] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
[2022-11-24T02:41:13.153Z] 2022-11-24 02:41:12.767170: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/conda/lib/python3.8/site-packages/torch/lib:/opt/conda/lib/python3.8/site-packages/torch_tensorrt/lib:/usr/local/cuda/compat/lib:/usr/local/nvidia/lib:/usr/local/nvidia/lib64
[2022-11-24T02:41:13.153Z] 2022-11-24 02:41:12.767249: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/conda/lib/python3.8/site-packages/torch/lib:/opt/conda/lib/python3.8/site-packages/torch_tensorrt/lib:/usr/local/cuda/compat/lib:/usr/local/nvidia/lib:/usr/local/nvidia/lib64
[2022-11-24T02:41:13.154Z] 2022-11-24 02:41:12.767257: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
[2022-11-24T02:41:14.518Z] Default upsampling behavior when mode=trilinear is changed to align_corners=False since 0.4.0. Please specify align_corners=True if the old behavior is desired. See the documentation of nn.Upsample for details.
[2022-11-24T02:41:16.405Z] <function compute_meandice at 0x7f492bd26c10>: Function `compute_meandice` has been deprecated since version 1.0.0. use `compute_dice` instead.
[2022-11-24T02:41:17.331Z] Default upsampling behavior when mode=trilinear is changed to align_corners=False since 0.4.0. Please specify align_corners=True if the old behavior is desired. See the documentation of nn.Upsample for details.
[2022-11-24T02:41:17.331Z] Default upsampling behavior when mode=trilinear is changed to align_corners=False since 0.4.0. Please specify align_corners=True if the old behavior is desired. See the documentation of nn.Upsample for details.
[2022-11-24T02:41:29.493Z] 2022-11-24 02:41:28.419269: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 AVX512F AVX512_VNNI FMA
[2022-11-24T02:41:29.493Z] To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
[2022-11-24T02:41:29.493Z] 2022-11-24 02:41:28.563346: I tensorflow/core/util/port.cc:104] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
[2022-11-24T02:41:29.493Z] 2022-11-24 02:41:29.248034: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/conda/lib/python3.8/site-packages/torch/lib:/opt/conda/lib/python3.8/site-packages/torch_tensorrt/lib:/usr/local/cuda/compat/lib:/usr/local/nvidia/lib:/usr/local/nvidia/lib64
[2022-11-24T02:41:29.493Z] 2022-11-24 02:41:29.248128: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/conda/lib/python3.8/site-packages/torch/lib:/opt/conda/lib/python3.8/site-packages/torch_tensorrt/lib:/usr/local/cuda/compat/lib:/usr/local/nvidia/lib:/usr/local/nvidia/lib64
[2022-11-24T02:41:29.493Z] 2022-11-24 02:41:29.248136: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
[2022-11-24T02:41:32.024Z] <function compute_meandice at 0x7f0bcfd58820>: Function `compute_meandice` has been deprecated since version 1.0.0. use `compute_dice` instead.
[2022-11-24T02:41:44.185Z] 2022-11-24 02:41:43.125331: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 AVX512F AVX512_VNNI FMA
[2022-11-24T02:41:44.185Z] To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
[2022-11-24T02:41:44.185Z] 2022-11-24 02:41:43.264840: I tensorflow/core/util/port.cc:104] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
[2022-11-24T02:41:44.185Z] 2022-11-24 02:41:43.942032: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/conda/lib/python3.8/site-packages/torch/lib:/opt/conda/lib/python3.8/site-packages/torch_tensorrt/lib:/usr/local/cuda/compat/lib:/usr/local/nvidia/lib:/usr/local/nvidia/lib64
[2022-11-24T02:41:44.185Z] 2022-11-24 02:41:43.942113: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/conda/lib/python3.8/site-packages/torch/lib:/opt/conda/lib/python3.8/site-packages/torch_tensorrt/lib:/usr/local/cuda/compat/lib:/usr/local/nvidia/lib:/usr/local/nvidia/lib64
[2022-11-24T02:41:44.185Z] 2022-11-24 02:41:43.942121: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
[2022-11-24T02:41:47.447Z] <function compute_meandice at 0x7f782f2ea940>: Function `compute_meandice` has been deprecated since version 1.0.0. use `compute_dice` instead.
[2022-11-24T02:41:59.534Z]
Executing: 85%|████████▌ | 17/20 [02:30<00:40, 13.39s/cell]
Executing: 95%|█████████▌| 19/20 [02:31<00:10, 10.69s/cell]
Executing: 95%|█████████▌| 19/20 [02:33<00:08, 8.08s/cell]
[2022-11-24T02:41:59.534Z] /opt/conda/lib/python3.8/site-packages/papermill/iorw.py:153: UserWarning: the file is not specified with any extension : -
[2022-11-24T02:41:59.534Z] warnings.warn(
[2022-11-24T02:41:59.534Z] Traceback (most recent call last):
[2022-11-24T02:41:59.534Z] File "/opt/conda/bin/papermill", line 8, in <module>
[2022-11-24T02:41:59.534Z] sys.exit(papermill())
[2022-11-24T02:41:59.534Z] File "/opt/conda/lib/python3.8/site-packages/click/core.py", line 1130, in __call__
[2022-11-24T02:41:59.534Z] return self.main(*args, **kwargs)
[2022-11-24T02:41:59.534Z] File "/opt/conda/lib/python3.8/site-packages/click/core.py", line 1055, in main
[2022-11-24T02:41:59.534Z] rv = self.invoke(ctx)
[2022-11-24T02:41:59.534Z] File "/opt/conda/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
[2022-11-24T02:41:59.534Z] return ctx.invoke(self.callback, **ctx.params)
[2022-11-24T02:41:59.534Z] File "/opt/conda/lib/python3.8/site-packages/click/core.py", line 760, in invoke
[2022-11-24T02:41:59.534Z] return __callback(*args, **kwargs)
[2022-11-24T02:41:59.534Z] File "/opt/conda/lib/python3.8/site-packages/click/decorators.py", line 26, in new_func
[2022-11-24T02:41:59.534Z] return f(get_current_context(), *args, **kwargs)
[2022-11-24T02:41:59.534Z] File "/opt/conda/lib/python3.8/site-packages/papermill/cli.py", line 250, in papermill
[2022-11-24T02:41:59.534Z] execute_notebook(
[2022-11-24T02:41:59.534Z] File "/opt/conda/lib/python3.8/site-packages/papermill/execute.py", line 128, in execute_notebook
[2022-11-24T02:41:59.534Z] raise_for_execution_errors(nb, output_path)
[2022-11-24T02:41:59.534Z] File "/opt/conda/lib/python3.8/site-packages/papermill/execute.py", line 232, in raise_for_execution_errors
[2022-11-24T02:41:59.534Z] raise error
[2022-11-24T02:41:59.534Z] papermill.exceptions.PapermillExecutionError:
[2022-11-24T02:41:59.534Z] ---------------------------------------------------------------------------
[2022-11-24T02:41:59.534Z] Exception encountered at "In [9]":
[2022-11-24T02:41:59.534Z] ---------------------------------------------------------------------------
[2022-11-24T02:41:59.534Z] IndexError Traceback (most recent call last)
[2022-11-24T02:41:59.534Z] Cell In [9], line 14
[2022-11-24T02:41:59.534Z] 12 cbar = plt.colorbar(shrink=0.8)
[2022-11-24T02:41:59.534Z] 13 plt.subplot(2, 2, 3)
[2022-11-24T02:41:59.534Z] ---> 14 plt.imshow(pred[32, :, :, 0])
[2022-11-24T02:41:59.534Z] 15 plt.title('background class prediction')
[2022-11-24T02:41:59.534Z] 16 cbar = plt.colorbar(shrink=0.8)
[2022-11-24T02:41:59.534Z]
[2022-11-24T02:41:59.534Z] IndexError: too many indices for array: array is 3-dimensional, but 4 were indexed
Metadata
Metadata
Assignees
Labels
No labels
Activity
wyli commentedon Nov 24, 2022
is this introduced by Project-MONAI/MONAI#5562? could you please have a look @mingxin-zheng @myron
myron commentedon Nov 24, 2022
I think the issue that predictions are saved properly as 3D images now, and not 4D.
so the plot should be
plt.imshow(pred[32, :, :]==0)instead ofplt.imshow(pred[32, :, :, 0])and
plt.imshow(pred[32, :, :]==1)instead ofplt.imshow(pred[32, :, :, 1])@mingxin-zheng can you test it please, I don't have an easy way to test at the moment
mingxin-zheng commentedon Nov 24, 2022
@wyli @myron Since the output is not default to one-hot encoding, it breaks the imshow lines. My fix should work in either way (1.0.1 and the dev branch).
fix plotting and correct typos (#1060)
fix plotting and correct typos (Project-MONAI#1060)