Skip to content

IntegrationClassification2D timeout #6311

Closed

Description

@wyli

Describe the bug

[2023-04-06T04:23:52.950Z] ======================================================================
[2023-04-06T04:23:52.950Z] ERROR: test_timing (__main__.IntegrationClassification2D)
[2023-04-06T04:23:52.950Z] ----------------------------------------------------------------------
[2023-04-06T04:23:52.950Z] Traceback (most recent call last):
[2023-04-06T04:23:52.950Z]   File "/home/jenkins/agent/workspace/MONAI-postmerge/monai_github/tests/utils.py", line 633, in _wrapper
[2023-04-06T04:23:52.950Z]     raise timeout_error
[2023-04-06T04:23:52.950Z] multiprocessing.context.TimeoutError: 'test_timing' in '__main__' did not finish in 1000s.
[2023-04-06T04:23:52.950Z] 
[2023-04-06T04:23:52.950Z] ----------------------------------------------------------------------

(this happens randomly on A100 postmerge, works fine v100)

Activity

  1. self-assigned this
    on Apr 6, 2023
  2. wyli commented on Apr 6, 2023

    @wyli
    ContributorAuthor

    seems to be addressed by limiting num_threads

    export OMP_NUM_THREADS=8
    export MKL_NUM_THREADS=8
    
  3. wyli commented on Apr 11, 2023

    @wyli
    ContributorAuthor

    Still an issue

  4. reopened this on Apr 11, 2023
  5. Nic-Ma commented on Apr 11, 2023

    @Nic-Ma
    Contributor

    @KumoLiu is testing on A100.

    Thanks.

  6. added 2 commits that reference this issue on Apr 11, 2023
  7. reopened this on Apr 11, 2023
  8. wyli commented on Aug 10, 2023

    @wyli
    ContributorAuthor

    no issue after the blossom migration

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions