Skip to content

Resize transform produces aliasing artifacts #4020

Closed

Description

@lemairecarl

Describe the bug
The Resize transform produces aliasing artifacts. It uses the F.interpolate function from PyTorch, which has an antialiasing option, but that does not support 3D downsampling of volumes (5D tensors). The Resize transforms does not use the antialiasing option at all:

resized = torch.nn.functional.interpolate(

This line in PyTorch source code confirms that the antialiasing option is not available for 3D volumes (trilinear interpolation):
https://github.com/pytorch/pytorch/blob/8ed6cb42ba2635319b67417b9ee2f1a0142d2c25/torch/nn/functional.py#L3851

The issue of ignoring aliasing in deep learning research is covered in this paper: https://arxiv.org/abs/2104.11222

To Reproduce
Steps to reproduce the behavior:

  1. Resize an image or volume down to a small size, e.g. using a factor of 8
  2. Observe the aliasing artifacts in the resized tensor

Expected behavior
The downsampled image should look like a smaller version of the original (no aliasing).

Activity

  1. Nic-Ma commented on Mar 29, 2022

    @Nic-Ma
    Contributor

    Hi @lemairecarl ,

    Thanks for your finding and feedback!
    @wyli , I think maybe it's necessary to add the antialias arg to Resize, Zoom, RandZoom transforms based on:
    https://github.com/pytorch/pytorch/blob/v1.11.0/torch/nn/functional.py#L3780
    What do you think?

    Thanks.

  2. wyli commented on Mar 29, 2022

    @wyli
    Contributor

    Yes there's also a related p1 milestone but I don't have the bandwidth to implement it https://github.com/Project-MONAI/MONAI/milestone/44

  3. Nic-Ma commented on Mar 29, 2022

    @Nic-Ma
    Contributor

    Hi @wyli ,

    OK, as you and @yiheng-wang-nv are busy with other bugs, let me self-assign this ticket later.

    Thanks.

  4. self-assigned this
    on Mar 29, 2022
  5. removed their assignment
    on Mar 29, 2022
  6. lemairecarl commented on Mar 29, 2022

    @lemairecarl
    Author

    Hi, thanks for taking this seriously.

    I see that you added this to the "label downsampling with antialiasing" milestone. Maybe you should rename the milestone as just "downsampling with antialiasing" ?

  7. dongyang0122 commented on Mar 31, 2022

    @dongyang0122
    Collaborator

    hi @lemairecarl, thanks for reporting the issue! Do you have sampling images (before and after the down-sampling) showing the undesired artifacts? That would be helpful for us to understand and resolve the issue.

  8. wyli commented on Mar 31, 2022

    @wyli
    Contributor

    hi @lemairecarl, thanks for reporting the issue! Do you have sampling images (before and after the down-sampling) showing the undesired artifacts? That would be helpful for us to understand and resolve the issue.

    Hi @dongyang0122 the appendix section 6.4 figure 12 of the paper is a good demo https://arxiv.org/pdf/2104.11222.pdf

  9. 3 remaining items

  10. dongyang0122 commented on Apr 4, 2022

    @dongyang0122
    Collaborator

    @lemairecarl thank you for the reply! I understand the effect of aliasing in the image appearance. But it would be better to have a way to directly/indirectly validate the effect of anti-aliasing and describe it in the tutorial or README.

  11. lemairecarl commented on Apr 8, 2022

    @lemairecarl
    Author

    Here is an example:

    downsample_compare4

    As you can see, the F.interpolate looks similar to Nearest. We can see lots of jagged edges. The bottom of the image is pretty crappy in F.interpolate. The structure is very "fragmented", and we can observe the same phenomenon elsewhere in the image. On the other hand, in the filtered version ("PIL Bilinear"), each output pixel summarizes a small region in the input image. It's more faithful.

  12. lemairecarl commented on Apr 8, 2022

    @lemairecarl
    Author

    It seems that in Pillow, the filtering is done at the same time as the resampling. Here is where a pixel value is multiplied with a filter value (k[x]):
    https://github.com/python-pillow/Pillow/blob/63f64de8b5cc6d065f19cc73c865633ab4940a52/src/libImaging/Resample.c#L493

    The filter used for bilinear filtering is defined here:
    https://github.com/python-pillow/Pillow/blob/63f64de8b5cc6d065f19cc73c865633ab4940a52/src/libImaging/Resample.c#L21

    Now, if we want to avoid implementing something in C++, one approach I see is the following:

    1. Build a low-pass filter
    2. Apply the low pass filter to the volume using F.conv3d
    3. Call F.interpolate

    Another option which I've just discovered is to use scipy's affine_transform (https://docs.scipy.org/doc/scipy/reference/generated/scipy.ndimage.affine_transform.html). By default, a spline filtering is applied. The downside of this function is that we have to build a transformation matrix.

    We could borrow from the following functions from the scilpy package. resample_volume takes a Nifti and a target resolution, and calls reslice, which in turns calls scipy's affine_transform.

    https://github.com/scilus/scilpy/blob/be6944a16ed078f53984e0d3c4908696bc2c87c2/scilpy/image/resample_volume.py#L15

    https://github.com/scilus/scilpy/blob/be6944a16ed078f53984e0d3c4908696bc2c87c2/scilpy/image/reslice.py#L18

  13. miccai-tutorials commented on Apr 12, 2022

    @miccai-tutorials

    hi @lemairecarl, thank you for the example! We are currently working on solutions for the anti-aliasing issue. Adding low-pass filter is one of options. And such filtering will be implemented with torch tensors.

  14. lemairecarl commented on Apr 12, 2022

    @lemairecarl
    Author

    @miccai-tutorials Awesome, thanks!

  15. moved this to In Progress in MONAI 0.9on Apr 18, 2022
  16. Repository owner moved this from In Progress to Done in MONAI 0.9on May 10, 2022
  17. vfdev-5 commented on Jun 30, 2022

    @vfdev-5
    Member

    @wyli by the way, is there any interest in enabling antialias option from pytorch side for 5D ?
    I was implementing it for pytorch and main blocker was in backward pass that can't be easily generalized to Nd case.

  18. wyli commented on Jun 30, 2022

    @wyli
    Contributor

    yes, I think spatially 2d and 3d will be very helpful already. is the backward pass some non-trainable deconvolution?

  19. Can-Zhao commented on Jun 30, 2022

    @Can-Zhao
    Collaborator

    @wyli by the way, is there any interest in enabling antialias option from pytorch side for 5D ? I was implementing it for pytorch and main blocker was in backward pass that can't be easily generalized to Nd case.

    Thanks. In

    if anti_aliasing and any(x < y for x, y in zip(spatial_size_, img_.shape[1:])):
    factors = torch.div(torch.Tensor(list(img_.shape[1:])), torch.Tensor(spatial_size_))
    if anti_aliasing_sigma is None:
    # if sigma is not given, use the default sigma in skimage.transform.resize
    anti_aliasing_sigma = torch.maximum(torch.zeros(factors.shape), (factors - 1) / 2).tolist()
    else:
    # if sigma is given, use the given value for downsampling axis
    anti_aliasing_sigma = list(ensure_tuple_rep(anti_aliasing_sigma, len(spatial_size_)))
    for axis in range(len(spatial_size_)):
    anti_aliasing_sigma[axis] = anti_aliasing_sigma[axis] * int(factors[axis] > 1)
    anti_aliasing_filter = GaussianSmooth(sigma=anti_aliasing_sigma)
    img_ = anti_aliasing_filter(img_)
    ,
    you can see that there is no constraint in the spatial dimension of images for the anti-aliasing part...

  20. vfdev-5 commented on Jun 30, 2022

    @vfdev-5
    Member

    is the backward pass some non-trainable deconvolution?

    sorry, when I was saying about "can't be easily generalized to Nd case." it was purely c++ implementation I wanted to say.
    https://github.com/pytorch/pytorch/blob/4e12300e4ee9ee4dadebd42aa2dc8ed28f836291/aten/src/ATen/native/cpu/UpSampleKernel.cpp#L1413

  21. vfdev-5 commented on Jun 30, 2022

    @vfdev-5
    Member

    @Can-Zhao, yes, applying gaussian blur can be also an option. It is a two steps resizing, it would be nice to benchmark these two implementations.

  22. Can-Zhao commented on Jun 30, 2022

    @Can-Zhao
    Collaborator

    @Can-Zhao, yes, applying gaussian blur can be also an option. It is a two steps resizing, it would be nice to benchmark these two implementations.

    Sorry I did not realize it goes to c++ level. I might misunderstand your concern...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type

Projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions