Skip to content

[Feature Request] 3D support for PanopticQualityMetric + provide option to return confusion matrix #5702

Closed

Description

@neuronflow

** Edit ** Feature request:

  • 3D support for PanopticQualityMetric
  • provide option to return confusion matrix

Describe the bug
Not sure whether one should report bugs for weekly versions?

I am playing around with PanopticQualityMetric.

The metric_name is ignored, I always get the same output, no matter if I use:

    metric_name=("pq")
    metric_name=("sq")
    metric_name=("rq")
    metric_name=("pq", "sq", "rq")

Also there is no option to turn on the confusion matrix output already, correct?

When I turn it on in the code I get a tensor like:

score1: tensor([153.0000,  24.0000,  21.0000, 139.7191])

If I am not mistaken these values represent the following?

tp, fp, fn, iou_sum = f[..., 0], f[..., 1], f[..., 2], f[..., 3]

feature request: I believe there should also be an option to directly compute mean iou as it is done in the manuscript.

To Reproduce
Compute PanopticQualityMetric with different metric_name parameters.
Environment

python -c 'import monai; monai.config.print_debug_info()'

python -c 'import monai; monai.config.print_debug_info()'

================================
Printing MONAI config...

MONAI version: 1.1.dev2250
Numpy version: 1.23.5
Pytorch version: 1.13.0+cu117
MONAI flags: HAS_EXT = False, USE_COMPILED = False, USE_META_DICT = False
MONAI rev id: 1d25ea145532319feb57647797ed3a8c2e7e9eb4
MONAI file: /home/florian/miniconda3/envs/emcaps/lib/python3.10/site-packages/monai/init.py

Optional dependencies:
Pytorch Ignite version: NOT INSTALLED or UNKNOWN VERSION.
Nibabel version: 4.0.2
scikit-image version: 0.19.3
Pillow version: 9.3.0
Tensorboard version: 2.11.0
gdown version: 4.5.4
TorchVision version: NOT INSTALLED or UNKNOWN VERSION.
tqdm version: 4.64.1
lmdb version: NOT INSTALLED or UNKNOWN VERSION.
psutil version: 5.9.4
pandas version: 1.5.1
einops version: NOT INSTALLED or UNKNOWN VERSION.
transformers version: NOT INSTALLED or UNKNOWN VERSION.
mlflow version: NOT INSTALLED or UNKNOWN VERSION.
pynrrd version: NOT INSTALLED or UNKNOWN VERSION.

For details about installing the optional dependencies, please visit:
https://docs.monai.io/en/latest/installation.html#installing-the-recommended-dependencies

================================
Printing system config...

System: Linux
Linux version: Ubuntu 20.04.5 LTS
Platform: Linux-5.15.0-50-generic-x86_64-with-glibc2.31
Processor: x86_64
Machine: x86_64
Python version: 3.10.8
Process name: python
Command: ['python', '-c', 'import monai; monai.config.print_debug_info()']
Open files: [popenfile(path='/home/florian/.vscode-server/data/logs/20221211T175743/remoteagent.log', fd=19, position=20132, mode='a', flags=33793), popenfile(path='/home/florian/.vscode-server/data/logs/20221211T175743/ptyhost.log', fd=20, position=4094, mode='a', flags=33793), popenfile(path='/home/florian/.vscode-server/bin/5235c6bb189b60b01b1f49062f4ffa42384f8c91/vscode-remote-lock.florian.5235c6bb189b60b01b1f49062f4ffa42384f8c91', fd=99, position=0, mode='w', flags=32769)]
Num physical CPUs: 16
Num logical CPUs: 32
Num usable CPUs: 32
CPU usage (%): [32.1, 20.7, 16.2, 17.0, 45.5, 30.8, 36.7, 36.4, 40.6, 17.9, 40.6, 23.4, 18.9, 15.2, 16.0, 86.0, 16.8, 30.8, 18.9, 18.1, 22.6, 34.3, 32.1, 34.0, 56.5, 35.5, 15.2, 27.4, 18.5, 16.8, 16.0, 30.8]
CPU freq. (MHz): 2008
Load avg. in last 1, 5, 15 mins (%): [11.3, 14.0, 14.8]
Disk usage (%): 35.1
Avg. sensor temp. (Celsius): UNKNOWN for given OS
Total physical memory (GB): 125.7
Available memory (GB): 114.8
Used memory (GB): 9.5

================================
Printing GPU config...

Num GPUs: 2
Has CUDA: True
CUDA version: 11.7
cuDNN enabled: True
cuDNN version: 8500
Current device: 0
Library compiled for CUDA architectures: ['sm_37', 'sm_50', 'sm_60', 'sm_70', 'sm_75', 'sm_80', 'sm_86']
GPU 0 Name: NVIDIA RTX A5000
GPU 0 Is integrated: False
GPU 0 Is multi GPU board: False
GPU 0 Multi processor count: 64
GPU 0 Total memory (GB): 23.7
GPU 0 CUDA capability (maj.min): 8.6
GPU 1 Name: Quadro RTX 8000
GPU 1 Is integrated: False
GPU 1 Is multi GPU board: False
GPU 1 Multi processor count: 72
GPU 1 Total memory (GB): 47.5
GPU 1 CUDA capability (maj.min): 7.5

Activity

  1. yiheng-wang-nv commented on Dec 12, 2022

    @yiheng-wang-nv
    Contributor

    Hi @neuronflow , could you post the full code you used to calculate PanopticQualityMetric ?
    Different metric_name should correspond to different outputs, the following is the code I just used to check:

    from monai.metrics import compute_panoptic_quality, PanopticQualityMetric
    import torch
    
    sample_pred = torch.as_tensor(
        [
            [[[2, 0, 1], [2, 1, 1], [0, 1, 1]], [[0, 1, 3], [0, 0, 0], [1, 2, 1]]],
            [[[1, 1, 1], [3, 2, 0], [3, 2, 1]], [[1, 1, 3], [3, 1, 1], [0, 3, 0]]],
        ]
    )
    
    sample_gt = torch.as_tensor(
        [
            [[[2, 0, 0], [2, 0, 0], [2, 2, 3]], [[3, 3, 3], [3, 2, 1], [2, 2, 3]]],
            [[[1, 1, 1], [0, 0, 3], [0, 0, 3]], [[0, 1, 3], [2, 1, 0], [3, 0, 3]]],
        ]
    )
    
    metric = PanopticQualityMetric(num_classes=3, metric_name=["sq", "pq"])
    metric(sample_pred, sample_gt)
    outputs = metric.aggregate()
    print(outputs)
    
    metric = PanopticQualityMetric(num_classes=3, match_iou_threshold=0.4, reduction="none", metric_name=["pq"])
    metric(sample_pred, sample_gt)
    outputs = metric.aggregate()
    print(outputs)
    
  2. neuronflow commented on Dec 13, 2022

    @neuronflow
    ContributorAuthor

    Hi @yiheng-wang-nv
    thanks, your example works!
    I realize I used the aggregate() method in a wrong way, sorry. Is there a way to return the confusion matrix without changing the class?

    Do you plan to extend this to 3D? I would highly appreciate a 3D version!

  3. wyli commented on Dec 14, 2022

    @wyli
    Contributor

    do you know why there's a constraint for 2d here @yiheng-wang-nv?

    if y_pred.shape[1] != 2:
    raise ValueError(
    f"for panoptic quality calculation, only 2 channels input is supported, got {y_pred.shape[1]}."
    )
    dims = y_pred.ndimension()
    if dims != 4:
    raise ValueError(f"y_pred should have 4 dimensions (batch, 2, h, w), got {dims}.")

    looking at the rest of the logic I don't find anything requiring 2D

  4. yiheng-wang-nv commented on Dec 15, 2022

    @yiheng-wang-nv
    Contributor

    do you know why there's a constraint for 2d here @yiheng-wang-nv?

    if y_pred.shape[1] != 2:
    raise ValueError(
    f"for panoptic quality calculation, only 2 channels input is supported, got {y_pred.shape[1]}."
    )
    dims = y_pred.ndimension()
    if dims != 4:
    raise ValueError(f"y_pred should have 4 dimensions (batch, 2, h, w), got {dims}.")

    looking at the rest of the logic I don't find anything requiring 2D

    Hi @wyli , the initial purpose to implement this metric is for the HoverNet pipeline. Considering that the reference repo uses only 2d logic, I decided to follow their logic. I think we can enhance it for 3d if needed after 1.1 release.

  5. neuronflow commented on Dec 15, 2022

    @neuronflow
    ContributorAuthor

    Thanks, I would highly appreciate a 3D implementation, I have multiple projects where I would use it! :)

  6. yiheng-wang-nv commented on Dec 16, 2022

    @yiheng-wang-nv
    Contributor

    Hi @neuronflow , could you please re-edit this ticket to make it more clear? I think it is a feature request of making PanopticQualityMetric support 3d, rather than a bug. Thanks

  7. changed the title [-][Bug/Question/Feature Request] PanopticQualityMetric - metric_name is ignored [monai-weekly][/-] [+][Feature Request] 3D support for PanopticQualityMetric + provide option to return confusion matrix[/+] on Dec 16, 2022
  8. neuronflow commented on Feb 13, 2023

    @neuronflow
    ContributorAuthor

    Hi @yiheng-wang-nv any more edits needed? :)

  9. yiheng-wang-nv commented on Feb 13, 2023

    @yiheng-wang-nv
    Contributor

    Hi @yiheng-wang-nv any more edits needed? :)

    Hi @neuronflow no progress so far because of other more urgent tasks.

  10. added a commit that references this issue on Feb 26, 2026
    7d3674e
  11. added a commit that references this issue on May 25, 2026
    373f986
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions