Repository navigation
Library bindings are not importable if the underlying library is not installed #951
Description
During the investigation of #920 I noticed that if I remove nvJitLink from my current environment, cuda.core cannot fall back to use the cuLink* driver APIs because the pathfinder error is not caught:
Traceback (most recent call last):
File "/local/home/leof/dev/debug_920.py", line 94, in <module>
program = Program(code, code_type="ptx", options=options)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/local/home/leof/dev/cuda-python/cuda_core/cuda/core/experimental/_program.py", line 411, in __init__
ObjectCode._init(code.encode(), code_type), options=self._translate_program_options(options)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/local/home/leof/dev/cuda-python/cuda_core/cuda/core/experimental/_program.py", line 420, in _translate_program_options
return LinkerOptions(
^^^^^^^^^^^^^^
File "<string>", line 24, in __init__
File "/local/home/leof/dev/cuda-python/cuda_core/cuda/core/experimental/_linker.py", line 189, in __post_init__
_lazy_init()
File "/local/home/leof/dev/cuda-python/cuda_core/cuda/core/experimental/_linker.py", line 69, in _lazy_init
_decide_nvjitlink_or_driver()
File "/local/home/leof/dev/cuda-python/cuda_core/cuda/core/experimental/_linker.py", line 47, in _decide_nvjitlink_or_driver
if inner_nvjitlink._inspect_function_pointer("__nvJitLinkVersion") == 0:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "cuda/bindings/_internal/nvjitlink.pyx", line 247, in cuda.bindings._internal.nvjitlink._inspect_function_pointer
File "cuda/bindings/_internal/nvjitlink.pyx", line 250, in cuda.bindings._internal.nvjitlink._inspect_function_pointer
File "cuda/bindings/_internal/nvjitlink.pyx", line 198, in cuda.bindings._internal.nvjitlink._inspect_function_pointers
File "cuda/bindings/_internal/nvjitlink.pyx", line 70, in cuda.bindings._internal.nvjitlink._check_or_init_nvjitlink
File "cuda/bindings/_internal/nvjitlink.pyx", line 92, in cuda.bindings._internal.nvjitlink._check_or_init_nvjitlink
File "cuda/bindings/_internal/nvjitlink.pyx", line 58, in cuda.bindings._internal.nvjitlink.load_library
File "/local/home/leof/miniforge3/envs/py312_cu129/lib/python3.12/site-packages/cuda/pathfinder/_dynamic_libs/load_nvidia_dynamic_lib.py", line 120, in load_nvidia_dynamic_lib
return _load_lib_no_cache(libname)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/local/home/leof/miniforge3/envs/py312_cu129/lib/python3.12/site-packages/cuda/pathfinder/_dynamic_libs/load_nvidia_dynamic_lib.py", line 48, in _load_lib_no_cache
found.raise_if_abs_path_is_None()
File "/local/home/leof/miniforge3/envs/py312_cu129/lib/python3.12/site-packages/cuda/pathfinder/_dynamic_libs/find_nvidia_dynamic_lib.py", line 199, in raise_if_abs_path_is_None
raise DynamicLibNotFoundError(f'Failure finding "{self.lib_searched_for}": {err}\n{att}')
cuda.pathfinder._dynamic_libs.load_dl_common.DynamicLibNotFoundError: Failure finding "libnvJitLink.so": No such file: libnvJitLink.so*, No such file: libnvJitLink.so*This echos @kkraus14's concern earlier (#728). We want to be able to catch errors at run time when the underlying library (including the driver) does not provide the needed symbol, which implies that the exception raising should be deferred to as late as when it's actually needed.
Activity
leofang commentedon Sep 8, 2025
I think
load_libraryshould be written ascc @mdboom @rwgk for vis
leofang commentedon Sep 8, 2025
@rwgk confirmed that we have a regression in cuda-bindings:
Fix NVIDIA#951: Fallback gracefully if a library can't be loaded
mdboom commentedon Sep 30, 2025
Looking at this more closely, I'm not sure it's a regression, at least at this level.
I compared the behavior of
mainto the12.8.0tag.In both cases, it was possible to
from cuda.bindings import nvjitlink, even if thenvjitlink.so is missing. It's only upon making the first API call that there is an exception. In12.8.0this was aRuntimeErrorand inmainit's acuda.pathfinder._dynamic_libs.load_dl_common.DynamicLibNotFoundError, which is a subclass ofRuntimeError, which I don't think is an API change.The fallback behavior to use the driver is determined by
_linker._decide_nvjitlink_or_driver, which should returnTruerather than raising an exception. In bothmainand12.8.0, it looks like it's throwing an exception.We could just catch this in
_decide_nvjitlink_or_driverand add some testing...?leofang commentedon Sep 30, 2025
Thanks, Mike. It appears that this code has never been working. This is with either cuda-python 12.6.2.post1 (in which we secretly shipped the nvJitLink bindings) or cuda-bindings 12.8.0
and what's shown in the OP is with cuda-bindings 12.9.x. I am scratching my head trying to understand what's missing... 🤔 Maybe the fallback to driver with this branch
cuda-python/cuda_core/cuda/core/experimental/_linker.py
Lines 47 to 49 in c4bb623
has never worked or tested... I am certain that the other branch works
cuda-python/cuda_core/cuda/core/experimental/_linker.py
Lines 43 to 45 in c4bb623
because CUDA 12.0 was tested in the CI until very recently when we dropped the inline config from the workflow:
cuda-python/.github/workflows/test-wheel-linux.yml
Lines 64 to 72 in 5a13448
in favor of a json file. @mdboom @rwgk Would you agree with this assessment and that the bug is in
cuda.corenotcuda.bindings?