Skip to content

[BUG]: compute-sanitzer finds CUDA_ERROR_INVALID_CONTEXT with default Device constructor #562

Closed
#566
Closed
[BUG]: compute-sanitzer finds CUDA_ERROR_INVALID_CONTEXT with default Device constructor#562
#566
Labels
P0High priority - Must do!bugSomething isn't workingcuda.coreEverything related to the cuda.core module

Description

@carterbox
carterbox
opened on Apr 21, 2025

Is this a duplicate?

Type of Bug

Silent Failure

Component

cuda.core v0.2.0 cuda-bindings v12.8.0

Describe the bug

Trying to initialize a Device with the current context when there is no current context causes an CUDA_ERROR_INVALID_CONTEXT.

How to Reproduce

Run the following script with compute-sanitizer:

import cuda.core.experimental as ccx

if __name__ == "__main__":
    d = ccx.Device(None)
    d.set_current()

Expected behavior

Device(None) should choose a default device (perhaps System.devices[0]?) to create a context if there is no existing current context.

Otherwise, if you are writing library code and you want to get the current device context but there is no guarantee that the user has already created one, then how do you get it without raising an error?

The error should be caught and invisible to compute-sanitizer.

Operating System

WSL2

nvidia-smi output

Mon Apr 21 21:09:28 2025
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 565.57.02              Driver Version: 566.03         CUDA Version: 12.7     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA RTX 1000 Ada Gene...    On  |   00000000:01:00.0 Off |                  N/A |
| N/A   36C    P4              7W /   35W |       0MiB /   6141MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

Activity

  1. carterbox
    added
    bugSomething isn't working
    on Apr 21, 2025
  2. kkraus14
    added
    P0High priority - Must do!
    cuda.coreEverything related to the cuda.core module
    on Apr 21, 2025

  3. carterbox commented on Apr 22, 2025

    @carterbox
    carterbox
    on Apr 22, 2025
    ContributorAuthor

    I think this bug may have been fixed on main already in the cuda.bindings package. I tried to explore a fix, but when I compiled cuda-bindings from main locally, my reproducer didn't reproduce. When I reverted the cuda-bindings package to the v12.8.0 tag, I was able to reproduce.

  4. leofang commented on Apr 22, 2025

    @leofang
    leofang
    on Apr 22, 2025
    Member

    I think this bug may have been fixed on main already in the cuda.bindings package.

    I actually don't understand what happens and need to look into it. cuda.core code hasn't changed much since you worked on it last time... Are you suggesting the bug is in cuda.bindings not cuda.core?

  5. leofang
    added
    triageNeeds the team's attention
    on Apr 22, 2025

  6. carterbox commented on Apr 22, 2025

    @carterbox
    carterbox
    on Apr 22, 2025
    ContributorAuthor

    Are you suggesting the bug is in cuda.bindings not cuda.core?

    Yes. I definitely should have included the log output from this error. Here:

    (test1) dching@NV-6V0FV44:~/Documents/cuda-python/cuda_core/tests$ compute-sanitizer python no-context.py
========= COMPUTE-SANITIZER
========= Program hit CUDA_ERROR_INVALID_CONTEXT (error 201) due to "invalid device context" on CUDA API call to cuCtxGetDevice.
=========     Saved host backtrace up to driver entry point at error
=========         Host Frame:  [0x504f7] in cydriver.cpython-313-x86_64-linux-gnu.so
=========         Host Frame:  [0x1b7b5] in cyruntime.cpython-313-x86_64-linux-gnu.so
=========         Host Frame:  [0x10df9] in cyruntime.cpython-313-x86_64-linux-gnu.so
=========         Host Frame:  [0x1e0619] in runtime.cpython-313-x86_64-linux-gnu.so
=========         Host Frame: PyObject_Vectorcall in call.c:327 [0x1b08fd] in python
=========         Host Frame: _PyEval_EvalFrameDefault.cold in generated_cases.c.h:813 [0x9e87c] in python
=========         Host Frame: slot_tp_new in typeobject.c:9832 [0x203de4] in python
=========         Host Frame: _PyObject_MakeTpCall in call.c:242 [0x1ae79e] in python
=========         Host Frame: _PyEval_EvalFrameDefault.cold in generated_cases.c.h:813 [0x9e87c] in python
=========         Host Frame: PyEval_EvalCode in ceval.c:604 [0x26d7c0] in python
=========         Host Frame: run_eval_code_obj in pythonrun.c:1381 [0x2aca8f] in python
=========         Host Frame: run_mod in pythonrun.c:1466 [0x2aa49b] in python
=========         Host Frame: pyrun_file in pythonrun.c:1295 [0x2a7435] in python
=========         Host Frame: _PyRun_SimpleFileObject in pythonrun.c:517 [0x2a7087] in python
=========         Host Frame: _PyRun_AnyFileObject in pythonrun.c:77 [0x2a6e7b] in python
=========         Host Frame: Py_RunMain in main.c:775 [0x2a508d] in python
=========         Host Frame: Py_BytesMain in main.c:829 [0x2596c6] in python
=========         Host Frame:  [0x29d8f] in libc.so.6
=========         Host Frame: __libc_start_main [0x29e3f] in libc.so.6
=========         Host Frame:  [0x258abd] in python
=========         Host Frame: __new__ in _device.py:960
=========         Host Frame: <module> in no-context.py:8
=========
========= ERROR SUMMARY: 1 error

  7. carterbox commented on Apr 25, 2025

    @carterbox
    carterbox
    on Apr 25, 2025
    ContributorAuthor

    We need to figure out what changes were made to the main branch that fixed this error and backport them to the 11.8.x branch.

  8. carterbox commented on Apr 25, 2025

    @carterbox
    carterbox
    on Apr 25, 2025
    ContributorAuthor

    In offline discussion with @leofang, we decided that it's too much effort to backport the fixes to CTK11 because the appropriate backport is probably #517.

  9. leofang
    removed
    triageNeeds the team's attention
    on Apr 25, 2025

  10. leofang commented on Apr 26, 2025

    @leofang
    leofang
    on Apr 26, 2025
    Member

    This is the minimal reproducer based on the traceback that Daniel provided above:

    $ compute-sanitizer python -c "import cuda.bindings.runtime as runtime; runtime.cudaGetDevice()"

    This happens because in the public releases cuda.bindings.runtime is implemented on top of the driver bindings, whereas after #517 it's no longer the case. We are seeing a bug in the cudart re-implementation (cc @vzhurba01 for vis). I suggested to document this as a bug fix in 12.9.0 and a known issue / won't fix in 11.8.7.

  11. leofang
    closed this as completedin #566on Apr 29, 2025

  12. ZzEeKkAa commented on Jun 26, 2025

    @ZzEeKkAa
    ZzEeKkAa
    on Jun 26, 2025
    Contributor

    I'm hitting exact the same error with 0.3.0 version:

    pip install 'cuda-core[cu12]'
export PATH=/usr/local/cuda-12.8/bin/:$PATH
compute-sanitizer python -c 'import cuda.core.experimental as ccx; ccx.Device()'
    ========= COMPUTE-SANITIZER
========= Program hit CUDA_ERROR_INVALID_CONTEXT (error 201) due to "invalid device context" on CUDA API call to cuCtxGetDevice.
=========     Saved host backtrace up to driver entry point at error
=========         Host Frame:  [0x5dbc8] in cydriver.cpython-311-x86_64-linux-gnu.so
=========         Host Frame:  [0x17f99] in cydriver.cpython-311-x86_64-linux-gnu.so
=========         Host Frame:  [0x292525] in driver.cpython-311-x86_64-linux-gnu.so
=========         Host Frame: PyObject_Vectorcall in call.c:299 [0x1f6a54] in python
=========         Host Frame: _PyEval_EvalFrameDefault in ceval.c:4769 [0x2719c2] in python
=========         Host Frame: _PyEval_Vector in ceval.c:6434 [0x27058d] in python
=========         Host Frame: _PyObject_Call_Prepend in call.c:482 [0x1f70e6] in python
=========         Host Frame: slot_tp_new in typeobject.c:7885 [0x2421db] in python
=========         Host Frame: type_call in typeobject.c:1091 [0x23e114] in python
=========         Host Frame: _PyObject_MakeTpCall in call.c:214 [0x1f6318] in python
=========         Host Frame: _PyEval_EvalFrameDefault in ceval.c:4769 [0x2719c2] in python
=========         Host Frame: _PyEval_Vector in ceval.c:6434 [0x2702fb] in python
=========         Host Frame: PyEval_EvalCode in ceval.c:1148 [0x2fedc6] in python
=========         Host Frame: run_eval_code_obj in pythonrun.c:1741 [0x314a48] in python
=========         Host Frame: run_mod in pythonrun.c:1762 [0x3149d3] in python
=========         Host Frame: PyRun_StringFlags in pythonrun.c:1632 [0x314946] in python
=========         Host Frame: PyRun_SimpleStringFlags in pythonrun.c:487 [0x3148a0] in python
=========         Host Frame: Py_RunMain in main.c:684 [0x31d7ce] in python
=========         Host Frame: Py_BytesMain in main.c:738 [0x31d2bc] in python
=========         Host Frame: __libc_start_main [0x240b2] in libc.so.6
=========         Host Frame: _start [0x2b4dbd] in python
=========         Host Frame: __new__ in _device.py:977
=========         Host Frame: <module> in <string>:1
=========
========= ERROR SUMMARY: 1 error

  13. leofang commented on Jun 26, 2025

    @leofang
    leofang
    on Jun 26, 2025
    Member

    As discussed offline, we need to run the sanitizer with the option --report-api-errors=no (see #687). It is not possible to fix it on our side without the compute sanitizer offering us a more fine-grained, localized control.

  14. leofang commented on Jun 26, 2025

    @leofang
    leofang
    on Jun 26, 2025
    Member

    A documentation PR to mention this would be welcomed @ZzEeKkAa 😉

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P0High priority - Must do!bugSomething isn't workingcuda.coreEverything related to the cuda.core module

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions