Skip to content

Multi GPU problem #17

Open
Open

Description

@zsgj-Xxx

Hi I think I'm having a new problem I've compared Ranger with Ranger 21 on a fine-grained dataset, but Ranger 21's results are much worse than Ranger's. I do get exciting results on my own computer, but the results on a multi-card server are poor. Do you know why?

Ranger
net_top11

Ranger21
net_top1

Activity

  1. zsgj-Xxx commented on Jul 6, 2021

    @zsgj-Xxx
    Author

    All the settings are the same except for the optimizer

  2. lessw2020 commented on Jul 6, 2021

    @lessw2020
    Owner

    Hi @zsgj-Xxx,
    Thanks for opening the issue. We have not had a chance to test Ranger21 out on multi-gpu yet, but for sure some aspects of it need to be adjusted in order to run properly (primarily b/c Ranger21 includes an lr scheduler internally), so performance will not be good on multi-gpu vs single gpu.
    Ranger (original Ranger) should operate w/o issue on multi-gpu and that's reflected in part by the better perf for multi-gpu here.
    I'm finishing up some testing on a new feature for Ranger21 today and then will try to setup a multi-gpu scenario to get it optimized for handling this.
    I'll leave this issue open for now to track it.
    Thanks!

  3. ryanstout commented on Jul 14, 2021

    @ryanstout

    @lessw2020 Great project, thanks for all of the hard work on it! I'm seeing similar issues. Interestingly, when switching to multiple gpu's (even multiplying the LR by the number of GPU's), the loss doesn't drop any faster. (Steps or wall time). Any ideas on why that would be? Thanks!

  4. rsomani95 commented on Aug 11, 2021

    @rsomani95

    Hey @lessw2020, Curious if you had a chance to work on this further? :)

    @ryanstout This discussion is probably of interest to you: #4 (reply in thread)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions