You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
CUDA_VISIBLE_DEVICES environ setting not work in ver. 0.5.0 #2161
Describe the bug
I'm using os.environ["CUDA_VISIBLE_DEVICES"] = '0' to set available GPU instead of specifing device like cuda:0.
Everthing works fine until updating to ver. 0.5.0. Maybe some GPU-related operations were added in the latest version during import phase.
Hi @ChenglongWang what metric are you using to determine which GPU is being used? Are you simply using nvidia-smi and observing that both GPUs are being used?
Also I think it's unlikely that MONAI is the cause of your problem, since MONAI doesn't (and I don't think ever has) use the variable CUDA_VISIBLE_DEVICES. It can be set with monai.config.deviceconfig.set_visible_devices(0), but this will just call os.environ["CUDA_VISIBLE_DEVICES"] = '0'. The rest gets handled by pytorch.
If you think I've missed something, would it be possible to create a minimum example that highlights your point? Preferably using all the same versions for the dependencies and simply switching between 0.4.x and 0.5.x? Thanks.
and then have the rest of your script as normal will work. If you set CUDA_VISIBLE_DEVICES after torch has been imported (and/or monai, since monai imports torch), your changes won't be used.
Hi @rijobro . Thank you for your kind help. Put the os.environ["CUDA_VISIBLE_DEVICES"] = x to the top will definitely fix this problem. However, in my case, the gpu is specified online, which means the gpu id is one argument of the program, and cannot be hard-coded on the top of script.
and then have the rest of your script as normal will work. If you set CUDA_VISIBLE_DEVICES after torch has been imported (and/or monai, since monai imports torch), your changes won't be used.
BTW, put the os.environ["CUDA_VISIBLE_DEVICES"] = x after the import torch would not cause the problem. Only GPU-related imports will make the cuda environ setting invalid. For example, you can simply move the monai-related imports after the os.environ["CUDA_VISIBLE_DEVICES"] = x, left torch-related imports on the top, like:
import torch
from PIL import Image
from torch.utils.data import DataLoader
from torch.utils.tensorboard import SummaryWriter
def main(tempdir):
os.environ["CUDA_VISIBLE_DEVICES"] = '0'
import monai
from monai.data import ArrayDataset, create_test_image_2d
from monai.inferers import sliding_window_inference
from monai.metrics import DiceMetric
from monai.transforms import (
Activations,
AddChannel,
AsDiscrete,
Compose,
LoadImage,
RandRotate90,
RandSpatialCrop,
ScaleIntensity,
ToTensor,
)
This will also fix the problem in the v0.5.0. So I'm considering the problem is caused by MONAI.
Activity
rijobro commentedon May 10, 2021
Hi @ChenglongWang what metric are you using to determine which GPU is being used? Are you simply using
nvidia-smiand observing that both GPUs are being used?rijobro commentedon May 10, 2021
Also I think it's unlikely that MONAI is the cause of your problem, since MONAI doesn't (and I don't think ever has) use the variable
CUDA_VISIBLE_DEVICES. It can be set withmonai.config.deviceconfig.set_visible_devices(0), but this will just callos.environ["CUDA_VISIBLE_DEVICES"] = '0'. The rest gets handled by pytorch.If you think I've missed something, would it be possible to create a minimum example that highlights your point? Preferably using all the same versions for the dependencies and simply switching between
0.4.xand0.5.x? Thanks.rijobro commentedon May 10, 2021
See this slightly related issue: #2167.
I think if you add to the top of your script:
and then have the rest of your script as normal will work. If you set
CUDA_VISIBLE_DEVICESafter torch has been imported (and/or monai, since monai imports torch), your changes won't be used.ChenglongWang commentedon May 11, 2021
Hi @rijobro . Thank you for your kind help. Put the
os.environ["CUDA_VISIBLE_DEVICES"] = xto the top will definitely fix this problem. However, in my case, the gpu is specified online, which means the gpu id is one argument of the program, and cannot be hard-coded on the top of script.For reproduction, you can simply test on one tutorial (ex. https://github.com/Project-MONAI/tutorials/blob/master/2d_segmentation/torch/unet_training_array.py).
BTW, put the
os.environ["CUDA_VISIBLE_DEVICES"] = xafter theimport torchwould not cause the problem. Only GPU-related imports will make the cuda environ setting invalid. For example, you can simply move the monai-related imports after theos.environ["CUDA_VISIBLE_DEVICES"] = x, left torch-related imports on the top, like:This will also fix the problem in the v0.5.0. So I'm considering the problem is caused by MONAI.
rijobro commentedon May 11, 2021
Think i found the bug and fixed it, see #2174.