Skip to content

MetaTensor conflict with Tensorboard #6326

Closed

Description

@ChenglongWang

Describe the bug
TL,DR: Error occurred when using SummaryWriter.add_graph().

To Reproduce
Steps to reproduce the behavior:

  1. The bug can be reproduced on 'segmentation2d' tutorial.
  2. Just simply add writer.add_graph(model, inputs, verbose=True) in the training loop
  3. SummaryWriter from either torch.utils.tensorboard or tensorboardX failed.

Environment (please complete the following information):

  • OS: Ubuntu 18.04
  • Python version: 3.10
  • MONAI version: 1.1.0
  • CUDA/cuDNN version: 11.6
  • More details:
MONAI version: 1.1.0
Numpy version: 1.23.5
Pytorch version: 1.12.1
MONAI flags: HAS_EXT = False, USE_COMPILED = False, USE_META_DICT = False
MONAI rev id: a2ec3752f54bfc3b40e7952234fbeb5452ed63e3

Optional dependencies:
Pytorch Ignite version: 0.4.10
Nibabel version: 5.1.0
scikit-image version: 0.20.0
Pillow version: 9.4.0
Tensorboard version: 2.12.1
gdown version: 4.7.1
TorchVision version: 0.13.1
tqdm version: 4.65.0
lmdb version: 1.4.1
psutil version: 5.9.4
pandas version: 2.0.0
einops version: 0.6.0
transformers version: 4.21.3
mlflow version: 2.2.2

Exception messages:

generating synthetic data to /tmp/tmp9810tcm9 (this may take a while)
torch.Size([10, 1, 96, 96]) torch.Size([10, 1, 96, 96])
----------
epoch 1/10
1/5, train_loss: 0.3956
Traceback (most recent call last):
  File "/homes/clwang/Code/tests/add_graph_test.py", line 174, in <module>
    main(tempdir)
  File "/homes/clwang/Code/tests/add_graph_test.py", line 127, in main
    writer.add_graph(model, inputs, verbose=True)
  File "/homes/clwang/.miniconda3/envs/monai/lib/python3.10/site-packages/tensorboardX/writer.py", line 937, in add_graph
    self._get_file_writer().add_graph(graph(model, input_to_model, verbose))
  File "/homes/clwang/.miniconda3/envs/monai/lib/python3.10/site-packages/torch/utils/tensorboard/_pytorch_graph.py", line 338, in graph
    trace = torch.jit.trace(model, args, strict=use_strict_trace)
  File "/homes/clwang/.miniconda3/envs/monai/lib/python3.10/site-packages/torch/jit/_trace.py", line 750, in trace
    return trace_module(
  File "/homes/clwang/.miniconda3/envs/monai/lib/python3.10/site-packages/torch/jit/_trace.py", line 992, in trace_module
    _check_trace(
  File "/homes/clwang/.miniconda3/envs/monai/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "/homes/clwang/.miniconda3/envs/monai/lib/python3.10/site-packages/torch/jit/_trace.py", line 329, in _check_trace
    copied_dict[name] = _clone_inputs(data)
  File "/homes/clwang/.miniconda3/envs/monai/lib/python3.10/site-packages/torch/jit/_trace.py", line 160, in _clone_inputs
    return function._nested_map(
  File "/homes/clwang/.miniconda3/envs/monai/lib/python3.10/site-packages/torch/autograd/function.py", line 463, in _map
    return fn(obj)
  File "/homes/clwang/.miniconda3/envs/monai/lib/python3.10/site-packages/torch/jit/_trace.py", line 150, in clone_input
    a.detach()
TypeError: MetaTensor.clone() got an unexpected keyword argument 'memory_format'

Activity

  1. KumoLiu commented on Apr 10, 2023

    @KumoLiu
    Contributor

    Hi @ChenglongWang, a workaround can be writer.add_graph(model, tensor(inputs), verbose=True).
    Thanks.

  2. ChenglongWang commented on Apr 10, 2023

    @ChenglongWang
    Author

    Hi @ChenglongWang, a workaround can be writer.add_graph(model, tensor(inputs), verbose=True).
    Thanks.

    Thanks! It really helps!

  3. added 3 commits that reference this issue on Apr 11, 2023
    a3aaaab
    2b6d0de
    9ef42ff
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions