Bug report

Bug description:

See: https://github.com/python/cpython/pull/112418/files

TLDR

ctypes/util.py has a find_library method that has different (although documented) contracts between OSs. It has an incorrect (in many cases) hard codded assumptions around where ldconfig lives.

Further In most cases it returns the /full/path to a native dynamic library (.so/dylib/etc) for many OSs and then falls back to to just the library_name (rather than the path) for everything else (which happens to cover all Linux distros). Whilst this is document, in an attempt to be almost OS and architecture agnostic, find_library has heaps of try this binary -> if it doesn't work/exist -> try another type algorithms. (although this ticket is not about the IO cost, inclusion of external dependencies, or security risk).

I've found that:

• calling find_library is doing unnecessary subprocess/shell invocations on to exit with $? !=0 This is simply because paths to bin tools (ldconfig) are hard coded rather than picked from the $PATH.
• Even though the full path to the library is identified during the life of find_librrary, those curious semantics of the documented Linux contract, result in just returning the file name (or a versioned symlink). This is kind of pointless as as it doesn't disambiguate for the caller where the library lives, thus once against needing to trawl the LD_LIBRARY_PATH to find the fullpath. Further we have implemented this to differ in its behaviour from invocations in other distros.

PRs

I'm providing a PR for the first issue and crossing my fingers as this has been a bug bear for a while. Happy to help with points two and three below too.

1.PR GH-112418 - This uses shtuils.which() to prefer an extent version of ldconfig over the hardcoded '/sbin/ldconfig' *removing assumption around system paths

1. If welcomed I'm working on a find_library_path counterpart to find_librarywhich reuses the same code but would provide a sibling consistent which ensures the same behaviour across all distros.
2. If welcomed, I can also throw together another version which uses pkg-config to find libraries for Linux; this would ensure it works across a wider breadth of modern Linux distros - including NixOS which one of my hosts is on. This would also save some cycles and do away with weird regular expressions scraping output.

Description

• find_library makes assumptions on binary paths and presence of specific binary and build tools (ldconfig / gcc). Further it expects some specific locations of ldconfig).
• Having it on the PATH in another location is ignored.

• Whilst not a bug, it is highly inconsistent that find_library does NOT return the path but only the name of the library (or the versioned name of the symlink without fullpath).
• All of this is very non-deterministic and is resulting in my not being able to use a vanilla version of sounddevice. You'll note that the nix package for this has had to work around this by applying a patch to a library to avoid find_library. I'd rather not have patched source code.

I've walked this code a few times and it assumes that: ldconfig lives in /sbin/ldconfig This is addressed in the PR I've provided.

EXPECTED

GIVEN the following code:

from ctypes.util import find_library as _find_library
_library = _find_library("portaudio")
print(_library)

AND an LD_LIBRARY_PATH which contains portaudio:

LD_LIBRARY_PATH=/nix/store/j61qwlv8kx91s2anpqvswg1i4lwx521c-portaudio-190700_20210406/lib

AND $PATH contains /home/me/.bin/ldconfig
AND this folder contains a symlinked .so to libportaudio.so.2
WHEN _find_library("portaudio") is invoked
THEN I expect it to terminate after returning a library found with ld_config without moving onto try gcc.

FURTHER I'd like (yes this isn't spec - but I suspect the weird Linux variation isn't either) for a new find_library_path method to exists which consistently returns the fullpath on ALL distros. Happy to implement this if there's a chance of this being used.

Eg. I expect find_library_path('portaudio') to return /nix/store/j61qwlv8kx91s2anpqvswg1i4lwx521c-portaudio-190700_20210406/lib/libportaudio.so.2 assuming that's where the library is. Ass opposed to the current 'libportaudio.so.2'

ACTUAL

1. If I have ldconfig on $PATH this is ignored and /sbin/ldconfig is the only thing use. Which means that the ldconfig invocation results in a wasted CPU cycle and fails even if I have it in another location and am following standard conventions across distros by having it on PATH. Note that nix and potentially other distros, or user space/CI installs may not necessarily have it in /sbin
2. The return value _find_library("portaudio") is just libportaudio.so.2 which I concede agrees with the docs that state '

Investigation

This one has frustrated me on a number of occasions, hence the investigation.

find_library has a lot of conditionals and cascading disjunctions/ORs which executes systems binaries such as ldconfig, gcc, and objectdump in turn to find the path to a native library.

Eg. In my case the implementation tries to find the library using specifically the fully qualified /sbin/ldconfig _executable_, using a compiler (gcc or cc if they exists on path with shutil.which()`)

            return _findSoname_ldconfig(name) or \
                   _get_soname(_findLib_gcc(name)) or _get_soname(_findLib_ld(name))

Whilst this is pretty brittle, my main struggle is with it then wrapping the returned value in _get_soname()

This _get_soname function also has some cascades (try this then try that then try..) but ultimately calls objectdump on a fully qualified path. I've debugged this and essentially it takes the correct value found by _findLib_gcc() (or other) and then does some weird parsing on the output from objectdump which actually truncates the path down to just the file name and NOT the fully qualified path to the library. This reliance on regexps of a system tool is way to brittle to see in a language library imho.

Eg. /nix/store/j61qwlv8kx91s2anpqvswg1i4lwx521c-portaudio-190700_20210406/lib/libportaudio.so results in objectdump producing some out put which gets stripped by a regexp with a capture group:

See

            res = re.search(br'\sSONAME\s+([^\s]+)', dump)
            if not res:
                return None
            return os.fsdecode(res.group(1))

This rex is applied to the following output from objectdump:

b'\n/nix/store/j61qwlv8kx91s2anpqvswg1i4lwx521c-portaudio-190700_20210406/lib/libportaudio.so:     file format elf64-x86-64\n\nProgram Header:\n    LOAD off    0x0000000000000000 vaddr 0x0000000000000000 ... flags r--\n\nDynamic Section:\n  NEEDED               librt.so.1\n  NEEDED               libasound.so.2\n  NEEDED               libjack.so.0\n  NEEDED               libm.so.6\n  NEEDED               libpthread.so.0\n  NEEDED               libc.so.6\n  SONAME               libportaudio.so.2\n...\n'

Note the ``... SONAME libportaudio.so.2\n ...`

SONAME's value is just libportaudio.so.2 with the path completely lost.
This becomes the value returned to _find_library() and ultimately to the caller.

This approach of using a rex for scraping build tools and expecting things to be installed in specific places is pretty brittle and does not scale per OS.

CPython versions tested on:

3.12, CPython main branch

Operating systems tested on:

Linux

Linked PRs

• gh-112417: Fix hardcoded ldconfig to use $PATH #112418