Repository navigation
Description
** Edit ** Feature request:
- 3D support for PanopticQualityMetric
- provide option to return confusion matrix
Describe the bug
Not sure whether one should report bugs for weekly versions?
I am playing around with PanopticQualityMetric.
The metric_name is ignored, I always get the same output, no matter if I use:
metric_name=("pq")
metric_name=("sq")
metric_name=("rq")
metric_name=("pq", "sq", "rq")
Also there is no option to turn on the confusion matrix output already, correct?
When I turn it on in the code I get a tensor like:
score1: tensor([153.0000, 24.0000, 21.0000, 139.7191])
If I am not mistaken these values represent the following?
tp, fp, fn, iou_sum = f[..., 0], f[..., 1], f[..., 2], f[..., 3]
feature request: I believe there should also be an option to directly compute mean iou as it is done in the manuscript.
To Reproduce
Compute PanopticQualityMetric with different metric_name parameters.
Environment
python -c 'import monai; monai.config.print_debug_info()'
python -c 'import monai; monai.config.print_debug_info()'
================================
Printing MONAI config...
MONAI version: 1.1.dev2250
Numpy version: 1.23.5
Pytorch version: 1.13.0+cu117
MONAI flags: HAS_EXT = False, USE_COMPILED = False, USE_META_DICT = False
MONAI rev id: 1d25ea145532319feb57647797ed3a8c2e7e9eb4
MONAI file: /home/florian/miniconda3/envs/emcaps/lib/python3.10/site-packages/monai/init.py
Optional dependencies:
Pytorch Ignite version: NOT INSTALLED or UNKNOWN VERSION.
Nibabel version: 4.0.2
scikit-image version: 0.19.3
Pillow version: 9.3.0
Tensorboard version: 2.11.0
gdown version: 4.5.4
TorchVision version: NOT INSTALLED or UNKNOWN VERSION.
tqdm version: 4.64.1
lmdb version: NOT INSTALLED or UNKNOWN VERSION.
psutil version: 5.9.4
pandas version: 1.5.1
einops version: NOT INSTALLED or UNKNOWN VERSION.
transformers version: NOT INSTALLED or UNKNOWN VERSION.
mlflow version: NOT INSTALLED or UNKNOWN VERSION.
pynrrd version: NOT INSTALLED or UNKNOWN VERSION.
For details about installing the optional dependencies, please visit:
https://docs.monai.io/en/latest/installation.html#installing-the-recommended-dependencies
================================
Printing system config...
System: Linux
Linux version: Ubuntu 20.04.5 LTS
Platform: Linux-5.15.0-50-generic-x86_64-with-glibc2.31
Processor: x86_64
Machine: x86_64
Python version: 3.10.8
Process name: python
Command: ['python', '-c', 'import monai; monai.config.print_debug_info()']
Open files: [popenfile(path='/home/florian/.vscode-server/data/logs/20221211T175743/remoteagent.log', fd=19, position=20132, mode='a', flags=33793), popenfile(path='/home/florian/.vscode-server/data/logs/20221211T175743/ptyhost.log', fd=20, position=4094, mode='a', flags=33793), popenfile(path='/home/florian/.vscode-server/bin/5235c6bb189b60b01b1f49062f4ffa42384f8c91/vscode-remote-lock.florian.5235c6bb189b60b01b1f49062f4ffa42384f8c91', fd=99, position=0, mode='w', flags=32769)]
Num physical CPUs: 16
Num logical CPUs: 32
Num usable CPUs: 32
CPU usage (%): [32.1, 20.7, 16.2, 17.0, 45.5, 30.8, 36.7, 36.4, 40.6, 17.9, 40.6, 23.4, 18.9, 15.2, 16.0, 86.0, 16.8, 30.8, 18.9, 18.1, 22.6, 34.3, 32.1, 34.0, 56.5, 35.5, 15.2, 27.4, 18.5, 16.8, 16.0, 30.8]
CPU freq. (MHz): 2008
Load avg. in last 1, 5, 15 mins (%): [11.3, 14.0, 14.8]
Disk usage (%): 35.1
Avg. sensor temp. (Celsius): UNKNOWN for given OS
Total physical memory (GB): 125.7
Available memory (GB): 114.8
Used memory (GB): 9.5
================================
Printing GPU config...
Num GPUs: 2
Has CUDA: True
CUDA version: 11.7
cuDNN enabled: True
cuDNN version: 8500
Current device: 0
Library compiled for CUDA architectures: ['sm_37', 'sm_50', 'sm_60', 'sm_70', 'sm_75', 'sm_80', 'sm_86']
GPU 0 Name: NVIDIA RTX A5000
GPU 0 Is integrated: False
GPU 0 Is multi GPU board: False
GPU 0 Multi processor count: 64
GPU 0 Total memory (GB): 23.7
GPU 0 CUDA capability (maj.min): 8.6
GPU 1 Name: Quadro RTX 8000
GPU 1 Is integrated: False
GPU 1 Is multi GPU board: False
GPU 1 Multi processor count: 72
GPU 1 Total memory (GB): 47.5
GPU 1 CUDA capability (maj.min): 7.5
Activity
yiheng-wang-nv commentedon Dec 12, 2022
Hi @neuronflow , could you post the full code you used to calculate
PanopticQualityMetric?Different metric_name should correspond to different outputs, the following is the code I just used to check:
neuronflow commentedon Dec 13, 2022
Hi @yiheng-wang-nv
thanks, your example works!
I realize I used the
aggregate()method in a wrong way, sorry. Is there a way to return the confusion matrix without changing the class?Do you plan to extend this to 3D? I would highly appreciate a 3D version!
wyli commentedon Dec 14, 2022
do you know why there's a constraint for 2d here @yiheng-wang-nv?
MONAI/monai/metrics/panoptic_quality.py
Lines 93 to 100 in 5200cc6
looking at the rest of the logic I don't find anything requiring 2D
yiheng-wang-nv commentedon Dec 15, 2022
Hi @wyli , the initial purpose to implement this metric is for the HoverNet pipeline. Considering that the reference repo uses only 2d logic, I decided to follow their logic. I think we can enhance it for 3d if needed after 1.1 release.
neuronflow commentedon Dec 15, 2022
Thanks, I would highly appreciate a 3D implementation, I have multiple projects where I would use it! :)
yiheng-wang-nv commentedon Dec 16, 2022
Hi @neuronflow , could you please re-edit this ticket to make it more clear? I think it is a feature request of making PanopticQualityMetric support 3d, rather than a bug. Thanks
[-][Bug/Question/Feature Request] PanopticQualityMetric - metric_name is ignored [monai-weekly][/-][+][Feature Request] 3D support for PanopticQualityMetric + provide option to return confusion matrix[/+]neuronflow commentedon Feb 13, 2023
Hi @yiheng-wang-nv any more edits needed? :)
yiheng-wang-nv commentedon Feb 13, 2023
Hi @neuronflow no progress so far because of other more urgent tasks.
Add 3D support and confusion matrix output to PanopticQualityMetric (P…
Add 3D support and confusion matrix output to PanopticQualityMetric (P…
Add 3D support and confusion matrix output to PanopticQualityMetric (P…
Add 3D support and confusion matrix output to PanopticQualityMetric (#…
Add 3D support and confusion matrix output to PanopticQualityMetric (P…