Repository navigation
Tf32 warnings - #6816
Tf32 warnings#6816
Conversation
Signed-off-by: Qingpeng Li <qingpeng9802@gmail.com>
Signed-off-by: Qingpeng Li <qingpeng9802@gmail.com>
when a function/class is imported Signed-off-by: Qingpeng Li <qingpeng9802@gmail.com>
Signed-off-by: Qingpeng Li <qingpeng9802@gmail.com>
for more information, see https://pre-commit.ci
Signed-off-by: Qingpeng Li <qingpeng9802@gmail.com>
…into tf32-warnings
Signed-off-by: Qingpeng Li <qingpeng9802@gmail.com>
|
/black |
Signed-off-by: monai-bot <monai.miccai2019@gmail.com>
wyli
left a comment
There was a problem hiding this comment.
thanks! please see some minor inline comments
Signed-off-by: Qingpeng Li <qingpeng9802@gmail.com>
Signed-off-by: Qingpeng Li <qingpeng9802@gmail.com>
Signed-off-by: monai-bot <monai.miccai2019@gmail.com>
|
/black |
wyli
left a comment
There was a problem hiding this comment.
Thanks, it looks good to me, testing with more environments
|
The failed test is caused by #2161 (also see kornia/kornia#1951). Pytorch only initializes |
Signed-off-by: Qingpeng Li <qingpeng9802@gmail.com>
…into tf32-warnings
|
/black |
for more information, see https://pre-commit.ci
Signed-off-by: monai-bot <monai.miccai2019@gmail.com>
ok... in this case how about only calling MONAI/monai/config/deviceconfig.py Line 195 in 65cf5fe currently the |
|
/build |
|
This failed case is kind of interesting🤔 My test env is |
|
The key of Actually, I have an ugly but safe solution: |
|
As the disscussion above, we prefer to use alt 2. Feel free to add any comment on alt 2. If alt 2. is okay, I can push an alt 2. commit. Looks like PyNVML is a safe choice. There are some similar usages in PyTorch, such as https://github.com/pytorch/pytorch/blob/v2.0.1/torch/cuda/__init__.py#L771-L794 |
|
sure, I think we can go for option 2 if it works fine across the test platforms and the import timing doesn't increase too much |
Signed-off-by: Qingpeng Li <qingpeng9802@gmail.com>
|
/black |
for more information, see https://pre-commit.ci
Signed-off-by: monai-bot <monai.miccai2019@gmail.com>
|
/build |
Signed-off-by: Qingpeng Li <qingpeng9802@gmail.com>
…into tf32-warnings
for more information, see https://pre-commit.ci
Signed-off-by: monai-bot <monai.miccai2019@gmail.com>
|
/build |
Following #6816 ### Description make `is_tf32_env()` safer. check `cuda` to prevent fallthrough case when `pynvml` is not found ### Types of changes <!--- Put an `x` in all the boxes that apply, and remove the not applicable items --> - [x] Non-breaking change (fix or new feature that would not break existing functionality). - [ ] Breaking change (fix or new feature that would cause existing functionality to change). - [ ] New tests added to cover the changes. - [ ] Integration tests passed locally by running `./runtests.sh -f -u --net --coverage`. - [ ] Quick tests passed locally by running `./runtests.sh --quick --unittests --disttests`. - [ ] In-line docstrings updated. - [ ] Documentation updated, tested `make html` command in the `docs/` folder. Signed-off-by: Qingpeng Li <qingpeng9802@gmail.com>
about #6754 .
Description
show a warning if any thing may enable tf32 is detected
Types of changes
./runtests.sh -f -u --net --coverage../runtests.sh --quick --unittests --disttests.make htmlcommand in thedocs/folder.