Skip to content

Improve performance on large datasets #1

Open

Description

@parser-algebra-0m

Hello,

Do you have suggestions on how to improve performance on large datasets (100K+ cells)?
I have been stuck running step2 for several days.

Activity

  1. huBioinfo commented on Jan 21, 2024

    @huBioinfo
    Collaborator

    Hi,

    There are several choices to accelerate the learning process for samples containing more than 100,000 cells .
    (1) Increase Learning_Rate=1E-3, but you must check the resulting loss file to make sure the training loss decreases normally when you increase the learning rate.
    (2) Increase Loss_Cutoff=-0.2. This modification would impact the accuracy of TCNs.
    (3) Decrease Num_Epoch=200 or a value which can guarantee the final loss <= -0.2. This modification could also impact the accuracy.
    (4) Decrease Num_Run=10, or parallelize these runs and then execute Step3 after obtaining all results of independent runs.

  2. huBioinfo commented on Apr 10, 2025

    @huBioinfo
    Collaborator

    Hi, if you are still interested, we just released an extended version CytoCommunity+, which can handle large datasets with millions of cells while using limited (graphics) memory. Thanks! https://github.com/huBioinfo/CytoCommunity-plus

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions