Skip to content

Update base image to 2410 - #8164

Merged
KumoLiu merged 5 commits into
Project-MONAI:devfrom
KumoLiu:update-base-image
Nov 11, 2024
Merged

KumoLiu merged 5 commits into
Project-MONAI:devfrom
KumoLiu:update-base-image

Conversation

@KumoLiu

@KumoLiu KumoLiu commented Oct 22, 2024

Copy link
Copy Markdown
Contributor

Fixes # .

Description

A few sentences describing the changes proposed in this pull request.

Types of changes

  • Non-breaking change (fix or new feature that would not break existing functionality).
  • Breaking change (fix or new feature that would cause existing functionality to change).
  • New tests added to cover the changes.
  • Integration tests passed locally by running ./runtests.sh -f -u --net --coverage.
  • Quick tests passed locally by running ./runtests.sh --quick --unittests --disttests.
  • In-line docstrings updated.
  • Documentation updated, tested make html command in the docs/ folder.

Signed-off-by: YunLiu <55491388+KumoLiu@users.noreply.github.com>
@KumoLiu

KumoLiu commented Oct 22, 2024

Copy link
Copy Markdown
Contributor Author

Hi @borisfom, could you please take a look at this KeyError that occurred when I upgraded the base image to version 24.09?


[2024-10-22T07:05:01.446Z] ======================================================================

[2024-10-22T07:05:01.446Z] ERROR: test_handler (tests.test_trt_compile.TestTRTCompile)

[2024-10-22T07:05:01.446Z] ----------------------------------------------------------------------

[2024-10-22T07:05:01.446Z] Traceback (most recent call last):

[2024-10-22T07:05:01.446Z]   File "/tmp/tmp.xs4H7I9CC3/tests/test_trt_compile.py", line 66, in test_handler

[2024-10-22T07:05:01.446Z]     net1.forward(torch.tensor([[0.0, 1.0], [1.0, 2.0]], device="cuda"))

[2024-10-22T07:05:01.446Z]   File "/usr/local/lib/python3.10/dist-packages/monai/networks/trt_compiler.py", line 496, in trt_forward

[2024-10-22T07:05:01.446Z]     return self._trt_compiler.forward(self, argv, kwargs)

[2024-10-22T07:05:01.446Z]   File "/usr/local/lib/python3.10/dist-packages/monai/networks/trt_compiler.py", line 351, in forward

[2024-10-22T07:05:01.446Z]     raise e

[2024-10-22T07:05:01.446Z]   File "/usr/local/lib/python3.10/dist-packages/monai/networks/trt_compiler.py", line 342, in forward

[2024-10-22T07:05:01.446Z]     self._build_and_save(model, build_args)

[2024-10-22T07:05:01.446Z]   File "/usr/local/lib/python3.10/dist-packages/monai/networks/trt_compiler.py", line 447, in _build_and_save

[2024-10-22T07:05:01.446Z]     engine_bytes = torch_tensorrt.convert_method_to_trt_engine(

[2024-10-22T07:05:01.446Z]   File "/usr/local/lib/python3.10/dist-packages/torch_tensorrt/_compile.py", line 355, in convert_method_to_trt_engine

[2024-10-22T07:05:01.446Z]     serialized_engine: bytes = ts_convert_method_to_trt_engine(

[2024-10-22T07:05:01.446Z]   File "/usr/local/lib/python3.10/dist-packages/torch_tensorrt/ts/_compiler.py", line 270, in convert_method_to_trt_engine

[2024-10-22T07:05:01.446Z]     module._c, method_name, _parse_compile_spec(compile_spec)

[2024-10-22T07:05:01.446Z]   File "/usr/local/lib/python3.10/dist-packages/torch_tensorrt/ts/_compile_spec.py", line 225, in _parse_compile_spec

[2024-10-22T07:05:01.446Z]     elif compile_spec["input_signature"] is not None:

[2024-10-22T07:05:01.446Z] KeyError: 'input_signature'

@KumoLiu KumoLiu mentioned this pull request Oct 22, 2024
1 task done
@borisfom

Copy link
Copy Markdown
Contributor

@KumoLiu : Unfortunately, this looks like internal Torch-TensorRT error I don't see an easy workaround for. Please just disable the failing test (test_handler) for now.

@KumoLiu

KumoLiu commented Oct 22, 2024

Copy link
Copy Markdown
Contributor Author

@KumoLiu : Unfortunately, this looks like internal Torch-TensorRT error I don't see an easy workaround for. Please just disable the failing test (test_handler) for now.

Yes, the error originates from the internal code in Torch. However, I'm wondering if we need to update the base image. As you mentioned here, if not, we can simply wait for the 24.10 release. Thanks.

@KumoLiu

KumoLiu commented Oct 23, 2024

Copy link
Copy Markdown
Contributor Author

[2024-10-22T07:05:01.446Z] KeyError: 'input_signature'

report a bug here: pytorch/pytorch#138674

@borisfom

Copy link
Copy Markdown
Contributor

@KumoLiu : yes we should just wait until 24.10 release

KumoLiu added a commit to KumoLiu/MONAI that referenced this pull request Oct 23, 2024
Signed-off-by: YunLiu <55491388+KumoLiu@users.noreply.github.com>
@KumoLiu
KumoLiu marked this pull request as draft October 23, 2024 08:35
Signed-off-by: YunLiu <55491388+KumoLiu@users.noreply.github.com>

@borisfom borisfom left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. The title should say '2410' though.

@KumoLiu KumoLiu changed the title Update base image to 2409 Update base image to 2410 Nov 8, 2024
@KumoLiu

KumoLiu commented Nov 8, 2024

Copy link
Copy Markdown
Contributor Author

/build

Signed-off-by: YunLiu <55491388+KumoLiu@users.noreply.github.com>
@KumoLiu

KumoLiu commented Nov 8, 2024

Copy link
Copy Markdown
Contributor Author

/build

@KumoLiu
KumoLiu marked this pull request as ready for review November 8, 2024 11:23
@KumoLiu
KumoLiu merged commit 0bb20a8 into Project-MONAI:dev Nov 11, 2024
@KumoLiu
KumoLiu deleted the update-base-image branch November 11, 2024 02:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants