You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Describe the bug
The Resize transform produces aliasing artifacts. It uses the F.interpolate function from PyTorch, which has an antialiasing option, but that does not support 3D downsampling of volumes (5D tensors). The Resize transforms does not use the antialiasing option at all:
I see that you added this to the "label downsampling with antialiasing" milestone. Maybe you should rename the milestone as just "downsampling with antialiasing" ?
hi @lemairecarl, thanks for reporting the issue! Do you have sampling images (before and after the down-sampling) showing the undesired artifacts? That would be helpful for us to understand and resolve the issue.
hi @lemairecarl, thanks for reporting the issue! Do you have sampling images (before and after the down-sampling) showing the undesired artifacts? That would be helpful for us to understand and resolve the issue.
@lemairecarl thank you for the reply! I understand the effect of aliasing in the image appearance. But it would be better to have a way to directly/indirectly validate the effect of anti-aliasing and describe it in the tutorial or README.
As you can see, the F.interpolate looks similar to Nearest. We can see lots of jagged edges. The bottom of the image is pretty crappy in F.interpolate. The structure is very "fragmented", and we can observe the same phenomenon elsewhere in the image. On the other hand, in the filtered version ("PIL Bilinear"), each output pixel summarizes a small region in the input image. It's more faithful.
We could borrow from the following functions from the scilpy package. resample_volume takes a Nifti and a target resolution, and calls reslice, which in turns calls scipy's affine_transform.
hi @lemairecarl, thank you for the example! We are currently working on solutions for the anti-aliasing issue. Adding low-pass filter is one of options. And such filtering will be implemented with torch tensors.
@wyli by the way, is there any interest in enabling antialias option from pytorch side for 5D ?
I was implementing it for pytorch and main blocker was in backward pass that can't be easily generalized to Nd case.
@wyli by the way, is there any interest in enabling antialias option from pytorch side for 5D ? I was implementing it for pytorch and main blocker was in backward pass that can't be easily generalized to Nd case.
Activity
Nic-Ma commentedon Mar 29, 2022
Hi @lemairecarl ,
Thanks for your finding and feedback!
@wyli , I think maybe it's necessary to add the
antialiasarg toResize,Zoom,RandZoomtransforms based on:https://github.com/pytorch/pytorch/blob/v1.11.0/torch/nn/functional.py#L3780
What do you think?
Thanks.
wyli commentedon Mar 29, 2022
Yes there's also a related p1 milestone but I don't have the bandwidth to implement it https://github.com/Project-MONAI/MONAI/milestone/44
Nic-Ma commentedon Mar 29, 2022
Hi @wyli ,
OK, as you and @yiheng-wang-nv are busy with other bugs, let me self-assign this ticket later.
Thanks.
lemairecarl commentedon Mar 29, 2022
Hi, thanks for taking this seriously.
I see that you added this to the "label downsampling with antialiasing" milestone. Maybe you should rename the milestone as just "downsampling with antialiasing" ?
dongyang0122 commentedon Mar 31, 2022
hi @lemairecarl, thanks for reporting the issue! Do you have sampling images (before and after the down-sampling) showing the undesired artifacts? That would be helpful for us to understand and resolve the issue.
wyli commentedon Mar 31, 2022
Hi @dongyang0122 the appendix section 6.4 figure 12 of the paper is a good demo https://arxiv.org/pdf/2104.11222.pdf
3 remaining items
dongyang0122 commentedon Apr 4, 2022
@lemairecarl thank you for the reply! I understand the effect of aliasing in the image appearance. But it would be better to have a way to directly/indirectly validate the effect of anti-aliasing and describe it in the tutorial or README.
lemairecarl commentedon Apr 8, 2022
Here is an example:
As you can see, the
F.interpolatelooks similar toNearest. We can see lots of jagged edges. The bottom of the image is pretty crappy inF.interpolate. The structure is very "fragmented", and we can observe the same phenomenon elsewhere in the image. On the other hand, in the filtered version ("PIL Bilinear"), each output pixel summarizes a small region in the input image. It's more faithful.lemairecarl commentedon Apr 8, 2022
It seems that in Pillow, the filtering is done at the same time as the resampling. Here is where a pixel value is multiplied with a filter value (
k[x]):https://github.com/python-pillow/Pillow/blob/63f64de8b5cc6d065f19cc73c865633ab4940a52/src/libImaging/Resample.c#L493
The filter used for bilinear filtering is defined here:
https://github.com/python-pillow/Pillow/blob/63f64de8b5cc6d065f19cc73c865633ab4940a52/src/libImaging/Resample.c#L21
Now, if we want to avoid implementing something in C++, one approach I see is the following:
F.conv3dF.interpolateAnother option which I've just discovered is to use scipy's
affine_transform(https://docs.scipy.org/doc/scipy/reference/generated/scipy.ndimage.affine_transform.html). By default, a spline filtering is applied. The downside of this function is that we have to build a transformation matrix.We could borrow from the following functions from the
scilpypackage.resample_volumetakes a Nifti and a target resolution, and callsreslice, which in turns calls scipy'saffine_transform.https://github.com/scilus/scilpy/blob/be6944a16ed078f53984e0d3c4908696bc2c87c2/scilpy/image/resample_volume.py#L15
https://github.com/scilus/scilpy/blob/be6944a16ed078f53984e0d3c4908696bc2c87c2/scilpy/image/reslice.py#L18
miccai-tutorials commentedon Apr 12, 2022
hi @lemairecarl, thank you for the example! We are currently working on solutions for the anti-aliasing issue. Adding low-pass filter is one of options. And such filtering will be implemented with torch tensors.
lemairecarl commentedon Apr 12, 2022
@miccai-tutorials Awesome, thanks!
vfdev-5 commentedon Jun 30, 2022
@wyli by the way, is there any interest in enabling antialias option from pytorch side for 5D ?
I was implementing it for pytorch and main blocker was in backward pass that can't be easily generalized to Nd case.
wyli commentedon Jun 30, 2022
yes, I think spatially 2d and 3d will be very helpful already. is the backward pass some non-trainable deconvolution?
Can-Zhao commentedon Jun 30, 2022
Thanks. In
MONAI/monai/transforms/spatial/array.py
Lines 760 to 771 in 7f80288
you can see that there is no constraint in the spatial dimension of images for the anti-aliasing part...
vfdev-5 commentedon Jun 30, 2022
sorry, when I was saying about "can't be easily generalized to Nd case." it was purely c++ implementation I wanted to say.
https://github.com/pytorch/pytorch/blob/4e12300e4ee9ee4dadebd42aa2dc8ed28f836291/aten/src/ATen/native/cpu/UpSampleKernel.cpp#L1413
vfdev-5 commentedon Jun 30, 2022
@Can-Zhao, yes, applying gaussian blur can be also an option. It is a two steps resizing, it would be nice to benchmark these two implementations.
Can-Zhao commentedon Jun 30, 2022
Sorry I did not realize it goes to c++ level. I might misunderstand your concern...