Skip to content

[CI failure] CI: Nightly optional-deps scheduled runs #3211

Open
#3263
Open
[CI failure] CI: Nightly optional-deps scheduled runs#3211
#3263
Labels
CI/CDCI/CD infrastructurebugSomething isn't workingci-workflow-healthAssigned to rolling issues managed by the ci-workflow-health.yml workflow.triageNeeds the team's attention

Description

@github-actions
github-actions
bot
opened on Oct 24, 2026

This issue tracks an unhealthy unattended CI workflow.
Subsequent failures are recorded in comments; a successful run in the
same trigger scope closes the issue automatically.

First unhealthy run

CI: Nightly optional-deps run #227, attempt 1 concluded with failure.

  • Trigger: schedule
  • Branch: main
  • Commit: 2da9c65
  • Completed: 2026-10-24T07:20:01Z

Created automatically by the CI workflow-health monitor.

Activity

  1. github-actions
    added
    bugSomething isn't working
    triageNeeds the team's attention
    CI/CDCI/CD infrastructure
    ci-workflow-healthAssigned to rolling issues managed by the ci-workflow-health.yml workflow.
    on Oct 24, 2026

  2. github-actions commented on Oct 25, 2026

    @github-actions
    github-actionsBot
    on Oct 25, 2026 – with GitHub Actions
    ContributorAuthor

    Another unhealthy run

    CI: Nightly optional-deps run #228, attempt 1 concluded with failure.

    • Trigger: schedule
    • Branch: main
    • Commit: a2fb804
    • Completed: 2026-10-25T03:21:30Z

  3. github-actions commented on Oct 26, 2026

    @github-actions
    github-actionsBot
    on Oct 26, 2026 – with GitHub Actions
    ContributorAuthor

    Another unhealthy run

    CI: Nightly optional-deps run #229, attempt 1 concluded with failure.

    • Trigger: schedule
    • Branch: main
    • Commit: a2fb804
    • Completed: 2026-10-26T03:08:44Z

  4. github-actions commented on Oct 27, 2026

    @github-actions
    github-actionsBot
    on Oct 27, 2026 – with GitHub Actions
    ContributorAuthor

    Another unhealthy run

    CI: Nightly optional-deps run #230, attempt 1 concluded with failure.

    • Trigger: schedule
    • Branch: main
    • Commit: 590a8bf
    • Completed: 2026-10-27T03:14:39Z

  5. github-actions commented on Oct 28, 2026

    @github-actions
    github-actionsBot
    on Oct 28, 2026 – with GitHub Actions
    ContributorAuthor

    Another unhealthy run

    CI: Nightly optional-deps run #231, attempt 1 concluded with failure.

    • Trigger: schedule
    • Branch: main
    • Commit: 94625c1
    • Completed: 2026-10-28T03:27:15Z

  6. github-actions commented on Oct 28, 2026

    @github-actions
    github-actionsBot
    on Oct 28, 2026 – with GitHub Actions
    ContributorAuthor

    Another unhealthy run

    CI: Nightly optional-deps run #231, attempt 2 concluded with failure.

    • Trigger: schedule
    • Branch: main
    • Commit: 94625c1
    • Completed: 2026-10-28T15:05:28Z
  7. jpascucci-nv
    closed this as completedin #3263on Oct 28, 2026
  8. leofang
    reopened this on Oct 28, 2026

  9. leofang commented on Oct 28, 2026

    @leofang
    leofang
    on Oct 28, 2026
    Member

    Nightly triage, Oct 24-28

    Looked at the last five runs. Three recurring real failures and one infra-side symptom. Full matrix is otherwise green.

    1. numba-cuda-mlir on Windows MCDM, rtxpro6000 — three nights running

    Oct 24 job, Oct 25 job, Oct 26 job. Each run opens with Write-Error "Switching to driver mode MCDM failed!" from the pre-test mode switch. Oct 24 and Oct 25 end with pytest exit 127 mid-run (no test summary written); Oct 26 completes and fails tests/test_dump_nvvm.py::test_compilation_dumps_exact_nvvm_input[ptx-sm_100] with assert 0 == 1. Oct 27-28 pass, so it is intermittent — but the common ancestor is the MCDM mode-switch warning, not a code change.

    2. numba-cuda-mlir on Linux rtxpro6000, CUDA 13.4.2 wheels — Oct 27

    Job log. Two tests in tests/test_cuda_array_interface_launch_cache.py both die with cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_INVALID_CONTEXT: invalid device context:

    • test_cupy_array_signature_cache_tracks_dtype_and_ndim
    • test_cupy_launch_does_not_reconstruct_device_arrays

    Looks like test-ordering sensitivity under pytest-randomly — an earlier test tears the primary context that cupy then inherits.

    3. standard linux-aarch64 Python 3.14t, CUDA 13.4.2 local — Oct 27

    Job log. cuda_bindings_13/tests/test_cuda.py::test_coredump_bool_uses_single_byte_storage fails with assert True is False, labeled PARALLEL FAILED. The test is @pytest.mark.thread_unsafe-marked but agent-authored; the process-global LIGHTWEIGHT coredump round-trip is not surviving concurrent test execution on free-threaded 3.14t.

    (Related: the Oct 22 aarch64 run saw the same four standard jobs all fail tests/test_load_nvidia_dynamic_lib.py::test_load_nvidia_dynamic_lib[cudss] with Child process failed for libname='cudss' with exit code 0. Has not recurred since.)

    4. Oct 28 — only failure is infra, not test

    Nightly numba-cuda linux-64 CUDA 13.4.2 wheels fails at host NVIDIA driver install (580.65.06 on kernel 6.8.0-147): ERROR: Building kernel modules → ERROR: The nvidia kernel module was not created. Job log.

    #3263 was merged as the fix and auto-closed this issue, but reopening because:

    • Both Oct 28 attempts (02:35 UTC, 15:02 UTC) ran before the PR merged at 18:29 UTC, so no nightly has exercised the new installer path yet.
    • Author's note on the PR: "Live GPU validation remains pending CI."
    • The patch only diverges from the old path when /proc/version's build-time GCC major differs from the host's default gcc -dumpversion. If 6.8.0-147 is built with gcc-13 (same as the Noble default), the patch is a no-op and the DKMS failure persists.
    • It assumes the .run installer picks up the exported CC/CXX for the kernel module build; passing --cc= to the installer explicitly would be safer.
    • /var/log/nvidia-installer.log is not uploaded as an artifact, so the actual compile error — and whether it really is "GCC 13 rejects GCC 14 flags" — has not been shown.

    Letting tomorrow's scheduled run speak before re-closing.

    5. Earlier context

    Oct 16-19 wiped the entire numba-cuda matrix (six jobs) with RuntimeError: NumPy only supports stream=None. from the numpy==2.4.6 + numba-cuda==0.30.4 combination. Resolved upstream by Oct 20.

    -- Leo's bot

  10. github-actions commented on Oct 29, 2026

    @github-actions
    github-actionsBot
    on Oct 29, 2026 – with GitHub Actions
    ContributorAuthor

    Another unhealthy run

    CI: Nightly optional-deps run #232, attempt 1 concluded with failure.

    • Trigger: schedule
    • Branch: main
    • Commit: d4c71ef
    • Completed: 2026-10-29T03:44:49Z

  11. github-actions commented on Oct 30, 2026

    @github-actions
    github-actionsBot
    on Oct 30, 2026 – with GitHub Actions
    ContributorAuthor

    Another unhealthy run

    CI: Nightly optional-deps run #233, attempt 1 concluded with failure.

    • Trigger: schedule
    • Branch: main
    • Commit: 03b4153
    • Completed: 2026-10-30T02:51:44Z
  12. jpascucci-nv
    added a commit that references this issue on Oct 30, 2026

    [no-ci] Use 580.126.18 for compatibility test on new ubuntu. (#3295)

    04fe681
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    CI/CDCI/CD infrastructurebugSomething isn't workingci-workflow-healthAssigned to rolling issues managed by the ci-workflow-health.yml workflow.triageNeeds the team's attention

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions