Skip to content

decouple the lr scheduler and optimizer? #36

Open

Description

@hiyyg

Hi @lessw2020, thanks for the very nice work!
I noticed that in this Ranger21, the optimizer is tightly coupled with the lr scheduler, could you guide me how I can decouple them?

Activity

  1. neuronflow commented on Nov 2, 2021

    @neuronflow

    I would like to second this. A split in ranger optimizer and ranger scheduler would be really cool.

  2. lessw2020 commented on Nov 2, 2021

    @lessw2020
    Owner

    Hi @hiyyg and @neuronflow,
    Right now you can turn off the built in lr scheduling by turning off both warmup and warmdown:
    use_warmup=False warmdown_active=False
    that should simply pass through the input lr and not touch it.
    Is that what you mean by decouple? Or do you mean having the scheduler seperately programmable (i.e. cosine decay vs we use linear etc).

  3. neuronflow commented on Nov 2, 2021

    @neuronflow
    Or do you mean having the scheduler seperately programmable (i.e. cosine decay vs we use linear etc).
    

    This is what I initially had in mind. Maybe, just maybe Ranger optimizer should go hand in hand with Ranger scheduler following the standard pytorch conventions?

  4. felipemello1 commented on Mar 10, 2022

    @felipemello1

    Hi @lessw2020, apparently in this current implementation there is no way to have different parameters learn using different learning rates. Did I get it right?

    If this were available, I would love to use it. Two use cases are the following:

    1. Fine tuning a network where layers closer to the head have a higher lr;
    2. My case: I train a graph neural network, and I need the embeddings to have 100x learning rate of the model, but in this current script I cant use the standard pytorch way of doing it:
    model_params = [params for name, params in self.model.named_parameters() if name.startswith('emb.') == False]
    emb_params = [params for name, params in self.model.named_parameters() if name.startswith('emb.') == True]
    optimizer_model = madgrad_wd([{'params': emb_params, 'lr': self.model_config['emb_max_lr']},
                              {'params': model_params, 'lr': self.model_config['model_max_lr']}], weight_decay=self.model_config['wd'])
    
  5. lessw2020 commented on Mar 11, 2022

    @lessw2020
    Owner

    Hi @fmellomascarenhas, @neuronflow and @hiyyg - fully agree with all the points above (decoupled scheduler and parameter groups.
    This split between scheduler and optimizer will happen for Ranger22 (the 2022 edition lol).
    Should have more info and updates shortly, as we just agreed last night to go ahead with the Ranger22 version.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions