You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Tutorial deep_atlas shows a huge negative loss in PyTorch 22.09-23.03 on some GPUs #1336
Hi @wyli ,
When I ran this tutorial on the 3090Ti, I got some negative values in tensor t2_sum - t_avg*t_sum in this line as shown below. I think this should not be correct and the values should always be positive. What do you think about this?
agreed, looks like some precision issue, I think this should be reported back to the framework team if we can narrow it down to a minimal reproducible example... perhaps you can try to save the inputs to the lncc_loss with torch.save
It seems that the NGC pytorch 23.03 container on the 3090Ti just rounds the output number somehow. And this leads to a large round-off error, since this logic is recursively used in the separable_filtering function of the loss function.
thanks, maybe you should fix the random seed or try to use exactly the same input to make it easier to see the differences? and perhaps set the datatypes and the print options to make sure they are not just printing differences https://pytorch.org/docs/stable/generated/torch.set_printoptions.html
thanks, maybe you should fix the random seed or try to use exactly the same input to make it easier to see the differences? and perhaps set the datatypes and the print options to make sure they are not just printing differences https://pytorch.org/docs/stable/generated/torch.set_printoptions.html
Hi @wyli ,
Thank you for the advices! I have updated the comment accordingly.
One more thing. I think this relates to the tf32 precision of the cuda computation. If I set the tf32 precision off in the first experiment mentioned above, the output becomes normal and the tutorial could run successfully.
These two lines need to be added to the program to turn off the tf32 precision.
Sorry, I forgot to add the tutorial result. I ran the deep atlas tutorial on the 3090Ti NGC pytorch docker with allow_tf32=False. The output turned out normally to me. Haven't got a chance to run it on L40 and 3080Ti.
ok thanks, we can add allow_tf32=False to the notebook as a solution for this ticket, what do you think?
Agreed. Should we add allow_tf32=False to the beginning of the whole tutorial or just add it to the loss cell and reset it to the previous value at the end of that cell?
Activity
binliunls commentedon May 12, 2023
Hi @wyli ,
When I ran this tutorial on the 3090Ti, I got some negative values in tensor
t2_sum - t_avg*t_sumin this line as shown below. I think this should not be correct and the values should always be positive. What do you think about this?Thanks,
Bin
wyli commentedon May 12, 2023
agreed, looks like some precision issue, I think this should be reported back to the framework team if we can narrow it down to a minimal reproducible example... perhaps you can try to save the inputs to the lncc_loss with
torch.savebinliunls commentedon May 15, 2023
I think it is a NGC pytorch docker related precision problem. Here is the code to show the bug.
Here are three experiments.
It seems that the NGC pytorch 23.03 container on the 3090Ti just rounds the output number somehow. And this leads to a large round-off error, since this logic is recursively used in the separable_filtering function of the loss function.
wyli commentedon May 15, 2023
thanks, maybe you should fix the random seed or try to use exactly the same input to make it easier to see the differences? and perhaps set the datatypes and the print options to make sure they are not just printing differences https://pytorch.org/docs/stable/generated/torch.set_printoptions.html
binliunls commentedon May 16, 2023
Hi @wyli ,
Thank you for the advices! I have updated the comment accordingly.
Thanks,
Bin
binliunls commentedon May 16, 2023
One more thing. I think this relates to the
tf32precision of the cuda computation. If I set thetf32precision off in the first experiment mentioned above, the output becomes normal and the tutorial could run successfully.These two lines need to be added to the program to turn off the
tf32precision.wyli commentedon May 16, 2023
so do you think settingsorry I didn't read your comments properlyallow_tf32=Falsewill address the original issue?binliunls commentedon May 16, 2023
Sorry, I forgot to add the tutorial result. I ran the deep atlas tutorial on the 3090Ti NGC pytorch docker with
allow_tf32=False. The output turned out normally to me. Haven't got a chance to run it on L40 and 3080Ti.wyli commentedon May 16, 2023
ok thanks, we can add
allow_tf32=Falseto the notebook as a solution for this ticket, what do you think?binliunls commentedon May 16, 2023
Agreed. Should we add
allow_tf32=Falseto the beginning of the whole tutorial or just add it to the loss cell and reset it to the previous value at the end of that cell?wyli commentedon May 17, 2023
created a ticket to track the issue Project-MONAI/MONAI#6525, feel free to create a PR if you have a solution for it...
1366-fix-deepatlas-ssim-loss-issue (#1397)
1366-fix-deepatlas-ssim-loss-issue (Project-MONAI#1397)