Skip to content

Clarification required for SurfaceDiceMetric #5906

Closed

Description

@tvercaut

Describe the bug
The documentation of the SurfaceDiceMetric is confusing and would benefit from being clarified:
https://github.com/Project-MONAI/MONAI/blob/dev/monai/metrics/surface_dice.py

It indeed says it computes the "Normalized Surface Distance (NSD)" intially and later it says it computes the "(Normalized) Surface Dice (NSD)". It then refers to https://github.com/deepmind/surface-distance which does not provide a metric called "Normalized Surface Distance" or "Normalized Surface Dice".

The DeepMind repo however provides a Surface Dice (with compute_surface_dice_at_tolerance) which may be what SurfaceDiceMetric implements but compute_surface_dice_at_tolerance requires a tolerance as input whereas SurfaceDiceMetric does not. It's thus unclear how these can be the same.

The referenced paper in MONAI's documentation, https://arxiv.org/abs/1809.04430, also does not mention a metric which accronym would be NSD. The reference for the 2D extension does though: https://arxiv.org/abs/2111.05408

Additional context
Similar confusions can be found in biomedical challenges:
https://doi.org/10.1038/s41467-022-30695-9
https://doi.org/10.1016/j.media.2020.101920

It's thus not only MONAI being confusing but it would be helpful to clarify nonetheless.

Activity

  1. Nic-Ma commented on Jan 27, 2023

    @Nic-Ma
    Contributor

    Hi @yiheng-wang-nv ,

    Could you please help take a look at this ticket?

    Thanks in advance.

  2. tvercaut commented on Jan 27, 2023

    @tvercaut
    MemberAuthor

    I had a quick discussion with @mjorgecardoso and Michela about NSD in the medical segmentation decathlon (https://doi.org/10.1038/s41467-022-30695-9). They confirmed that what they refer to as NSD in the challenge is the same as the Surface Dice in the Deepmind repo (compute_surface_dice_at_tolerance) and the Surface Dice Similarity Coefficient in section 2.2 of https://arxiv.org/abs/1809.04430

    Screenshot 2023-01-27 at 13 36 31

    I further assume that what MONAI refers to as class_thresholds is actually similar to what the other source call "tolerance". However class_thresholds is given in pixels/voxels rather than milimetres. This makes is harder to relate. Also, it's unclear to me how anisotropix voxels would be handled. If this is a limitation of MONAI's implementation, it would be worth highlighting.

    Given that the NSD acronym has been used in a few existing challenges, it may be worth keeping it in MONAI as well but maybe expand it to "Normalized Surface Dice" to improve clarity.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions