You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Hi @neuronflow,
Thanks for opening the issue!
Ranger21 does maintain a basic state dict but for sure we need to update it with some additional data to ensure a clean restart if training is stopped.
Let me use this issue to track it and I'll test and fix it ideally in the next few days as this has been on my todo list.
Activity
lessw2020 commentedon Jul 8, 2021
Hi @neuronflow,
Thanks for opening the issue!
Ranger21 does maintain a basic state dict but for sure we need to update it with some additional data to ensure a clean restart if training is stopped.
Let me use this issue to track it and I'll test and fix it ideally in the next few days as this has been on my todo list.
neuronflow commentedon Nov 2, 2021
any updates on this one? :) I lost multiple GPU days of training because the trainings are non resumable :/
Elevory commentedon Jul 4, 2022
Seconding the need for this feature!