Skip to content

resuming training with ranger21? #18

Open

Description

@neuronflow

As I learned ranger21 does internal lr scheduling etc.

How should training be resumed? Is there a state dict to be loaded etc.?

Activity

  1. lessw2020 commented on Jul 8, 2021

    @lessw2020
    Owner

    Hi @neuronflow,
    Thanks for opening the issue!
    Ranger21 does maintain a basic state dict but for sure we need to update it with some additional data to ensure a clean restart if training is stopped.
    Let me use this issue to track it and I'll test and fix it ideally in the next few days as this has been on my todo list.

  2. neuronflow commented on Nov 2, 2021

    @neuronflow
    Author

    any updates on this one? :) I lost multiple GPU days of training because the trainings are non resumable :/

  3. Elevory commented on Jul 4, 2022

    @Elevory

    Seconding the need for this feature!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions