You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Training seems fine for half a day with decent progress on all loss metrics, but then halts:
File "./train_pt.py", line 727, in <module>
main(sys.argv[1:])
File "./train_pt.py", line 612, in main
optimizer.step()
File "/home/morbo/git/sjeng/train/venv19/lib/python3.8/site-packages/torch/optim/optimizer.py", line 88, in wrapper
return func(*args, **kwargs)
File "/home/morbo/git/sjeng/train/venv19/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 28, in decorate_context
return func(*args, **kwargs)
File "/home/morbo/git/Ranger21/ranger21/ranger21.py", line 714, in step
raise RuntimeError("hit nan for variance_normalized")
RuntimeError: hit nan for variance_normalized
To be fair I'm also seeing this on Facebook's MADGRAD now, so I wonder if Adam/madgrad are just more likely to trigger this kind of divergence or if a bug slipped into the training data.
Basically one of the loss values NaN's and this causes the optimizer to instantly fail (I guess SGD just recovers if that happens).
I've had the same issue. Reducing the learning rate did help, but I'm at 1e-5 with default parameters and 1e-6 with madgrad still gave NaN on loss values. Curious if there's something else I can do.
changing lr, eps, weight_decay, use_adaptive_gradient_clipping, use_warmup appears to have no effect. The NaN comes from the forward pass in the second step, so some weights become NaN. Adam and AdaBelief cores work fine.
Activity
swarmt commentedon Sep 12, 2021
Am also seeing this.
gcp commentedon Sep 13, 2021
To be fair I'm also seeing this on Facebook's MADGRAD now, so I wonder if Adam/madgrad are just more likely to trigger this kind of divergence or if a bug slipped into the training data.
Basically one of the loss values NaN's and this causes the optimizer to instantly fail (I guess SGD just recovers if that happens).
swarmt commentedon Sep 13, 2021
Reducing my learning rate solved it.
TomStarshak commentedon Sep 20, 2021
I've had the same issue. Reducing the learning rate did help, but I'm at 1e-5 with default parameters and 1e-6 with madgrad still gave NaN on loss values. Curious if there's something else I can do.
dnhkng commentedon Sep 24, 2021
I've just hit it too :(
swarmt commentedon Sep 29, 2021
I found my error. I had some training data with values way outside me expected range of 0-1 which I found by adding an assert in my dataloader.
Sopel97 commentedon Mar 17, 2022
I integrated ranger21 into https://github.com/glinscott/nnue-pytorch and exploring different parameters. I'm hitting this issue always after first step of training.
This is what I'm using:
changing lr, eps, weight_decay, use_adaptive_gradient_clipping, use_warmup appears to have no effect. The NaN comes from the forward pass in the second step, so some weights become NaN. Adam and AdaBelief cores work fine.