RuntimeError: probability tensor contains either inf, nan or element < 0  #86

Activity

1. DavidFarago commented on May 24, 2023

   

   DavidFarago

   on May 24, 2023

   More actions

   I have the same problem. inputs are all valid (not nan, not inf, and >0), but probs seems to be all nan:

   (Pdb) probs[0,-100:]
   tensor([nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
           nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
           nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
           nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
           nan, nan, nan, nan], device='cuda:0', dtype=torch.float16)
   
2. ChengYen-Tang commented on Aug 7, 2023

   

   ChengYen-Tang

   on Aug 7, 2023

   More actions

   This problem seems to be caused by incorrect configuration of the Converting weights to Hugging Face format step

   --n-stages can be found from checkpoint folder, in my environment --n-stages 2
   

   --n-layer-per-stage must refer to the model configuration
   --n-layer-per-stage = model layers / --n-stages, in my environment --n-layer-per-stage 16
   https://huggingface.co/EleutherAI/pythia-6.9b-deduped