ctypes: We do not correctly handle NULL dlsym() return values

Bug report

Bug description:

The man(3) page for dlsym() states:

In unusual cases (see NOTES) the value of the symbol  could  actually  be
NULL. Therefore, a NULL return from dlsym() need not indicate an error.
The correct way to distinguish an error from a symbol whose value is NULL
is  to  call  dlerror(3) to clear any old error conditions, then call dl‐
sym(), and then call dlerror(3) again, saving its  return  value  into  a
variable, and check whether this saved value is not NULL.

As such, there can be cases where a call to dlsym returns NULL, while no error has been encountered and the string that dlerror() returns has not been (re)set.

Currently, (https://github.com/python/cpython/blob/main/Modules/_ctypes/_ctypes.c#L970) we do:

<...>
address = (void *)dlsym(handle, name);
if (!address) {
#ifdef __CYGWIN__
<snip Windows stuff>
#else
        PyErr_SetString(PyExc_ValueError, dlerror());
<...>

If dlsym() returns NULL, then by calling dlerror() we pass either NULL or a previous, unconsumed error string to PyErr_SetString.

In the context of ctypes, a NULL return by dlsym() might indeed always be considered an error.

To correctly express this, we should:

Call dlerror() before dlsym(), to clear any previous error
Call dlerror() after dlsym(), to see if dlsym() encountered any error
- If dlerror() returns a non-NULL value, then pass its result to PyErr_SetString
- If dlerror() returns NULL, then check if address is NULL, and if so, pass a custom error string to PyErr_SetString.

CPython versions tested on:

CPython main branch

Operating systems tested on:

Linux

Linked PRs

Is it actually a bug?
Can we make a repro?

EDIT: See Reproducer below.

I managed to create a reproducer using Indirect Functions (IFUNCS) by adapting this SO answer: https://stackoverflow.com/a/53590014

Create a file named foo.c with the following contents:

#include <stdio.h>

/* This is a 'GNU indirect function' (IFUNC) that will be called by
   dlsym() to resolve the symbol "foo" to an address. Typically, such
   a function would return the address of an actual function, but it
   can also just return NULL.  For some background on IFUNCs, see
   https://willnewton.name/uncategorized/using-gnu-indirect-functions/ */

asm (".type foo, @gnu_indirect_function");

void *
foo(void)
{
    fprintf(stderr, "foo called\n");
    return NULL;
}

Create a file named main.c with the following contents:

#include <dlfcn.h>
#include <stdio.h>
#include <stdlib.h>

int
main(int argc, char *argv[])
{
    void *handle;
    void (*funcp)(void);

    handle  = dlopen("./foo.so", RTLD_LAZY);
    if (handle == NULL) {
        fprintf(stderr, "dlopen: %s\n", dlerror());
        exit(EXIT_FAILURE);
    }

    dlerror();      /* Clear any outstanding error */

    funcp = dlsym(handle, "foo");

    printf("Results after dlsym(): funcp = %p; dlerror = %s\n",
            (void *) funcp, dlerror());

    exit(EXIT_SUCCESS);
}

Compile foo.c into a shared library named libfoo.so:

gcc -Wall -fPIC -shared -o libfoo.so foo.c

Compile the main executable:

gcc -Wall -o main main.c -L. -lfoo -ldl

Run main:

./main
foo called ### This is from the IFUNC ###
Results after dlsym(): funcp = (nil); dlerror = (null)

When using ctypes to search for symbol foo in libfoo.so we get a Segmentation Fault.
(Tested on Python 3.10 from my system, as well as one built from main.)

python3 -c 'import ctypes; C = ctypes.CDLL("./libfoo.so"); C.foo'

foo called
Segmentation fault (core dumped)

So this turned out to be a bug after all!

With the interpreter built from my PR we instead get:

./python -c 'import ctypes; C = ctypes.CDLL("./libfoo.so"); C.foo'

foo called
Traceback (most recent call last):
  File "<string>", line 1, in <module>
    import ctypes; C = ctypes.CDLL("./libfoo.so"); C.foo
                                                   ^^^^^
  File "/home/grg/a-lab/cpython/Lib/ctypes/__init__.py", line 415, in __getattr__
    func = self.__getitem__(name)
  File "/home/grg/a-lab/cpython/Lib/ctypes/__init__.py", line 420, in __getitem__
    func = self._FuncPtr((name_or_ordinal, self))
AttributeError: function 'foo' not found

@efimov-mikhail

cc: @efimov-mikhail

Fun fact:

The StackOverflow answer, which I adapted to form the reproducer is actually written by Michael Kerrisk [1].

So, he was the one who came up to clarify his statements in the man page, when noone else {c,w}ould :)

[1] https://en.wikipedia.org/wiki/Michael_Kerrisk

gh-126554: ctypes: Correctly handle NULL dlsym values (GH-126555)

pythongh-126554: ctypes: Correctly handle NULL dlsym values (pythonGH…

@grgalex

Thanks @grgalex for the fix, and @ZeroIntensity & @picnixz for the thorough reviews!

When using ctypes to search for symbol foo in libfoo.so we get a Segmentation Fault.
[...]
So this turned out to be a bug after all!

Well, when a symbol you asked for points to uninitialized memory (like a NULL), a segfault seems rather reasonable to me. (Granted, it does happen for the wrong reasons.)

Treating NULL specially could possibly be considered a feature, rather than a bug.
Given how hard this is to reproduce, I wouldn't mind having the fix to 3.14 only, and omitting the backports.

[3.13] gh-126554: ctypes: Correctly handle NULL dlsym values (GH-126555…

pythongh-126554: ctypes: Correctly handle NULL dlsym values (pythonGH…

[3.12] gh-126554: ctypes: Correctly handle NULL dlsym values (GH-126555…

pythongh-126554: ctypes: Correctly handle NULL dlsym values (pythonGH…

gh-126554: correct detection of `gcc` for `TestNullDlsym.test_null_dl…

pythongh-126554: correct detection of `gcc` for `TestNullDlsym.test_n…

[3.12] gh-126554: correct detection of `gcc` for `TestNullDlsym.test_…

[3.13] gh-126554: correct detection of `gcc` for `TestNullDlsym.test_…

Navigation Menu

ctypes: We do not correctly handle NULL dlsym() return values #126554

Bug report

Bug description:

CPython versions tested on:

Operating systems tested on:

Linked PRs

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

ctypes: We do not correctly handle NULL dlsym() return values #126554

Description

Bug report

Bug description:

CPython versions tested on:

Operating systems tested on:

Linked PRs

Activity

efimov-mikhail commented on Nov 7, 2024

grgalex commented on Nov 7, 2024

grgalex commented on Nov 7, 2024

grgalex commented on Nov 7, 2024

grgalex commented on Nov 7, 2024

grgalex commented on Nov 7, 2024

grgalex commented on Nov 8, 2024

encukou commented on Nov 15, 2024

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions