Repository navigation
pdal.Pipeline + filters.python is broken on Python >= 3.8 #75
Description
Starting in Python 3.8 (https://docs.python.org/3/whatsnew/3.8.html) C extensions to python shouldn't link against libpython:
"On Unix, C extensions are no longer linked to libpython except on Android and Cygwin"
libpdal_plugin_filter_python.dylib does link against libpython however, and this is fine when invoking pipelines containing "filters.python" from the command line, but it breaks when pipelines are constructed/run by a python script (i.e. via import pdal). In this case libpython is loaded within a python process, and various errors ensue. For me on OSX with Python3.9, it results in a double-free of the _heapq module definition object.
A workaround is to not link python_filter against libpython, and instead pass -undefined dynamic_lookup to the linker. This works for launching pipelines via python, but breaks running pdal pipeline foo.json on the command line. A hacky fix for the latter would be to call dlopen("libpython.dylib") before loading python_filter if the code is not running in a python process.
Activity
abellgithub commentedon Dec 7, 2020
We've been aware of this since the Python people broke stuff. I don't think we're going to fix it. I'm sure there's a ticket somewhere with a long history, but I can't find it at the moment.
hobu commentedon Dec 7, 2020
I have made some attempts to handle this on OSX, but it's a giant mess of permutations I wasn't able to cleanly figure out. Someone who is motivated to fix this could start there.
As @abellgithub stated, we are tired of trying to fix this. Given that that fix only needs to live in the PDAL Python codebase and PDAL proper, we can issue a new release if someone does provide a fix that does the right thing.
As for the permutations:
froody commentedon Dec 8, 2020
@hobu @abellgithub I came up with an elegant fix, can you review it? Also I think darwin tests are broken on master.
hobu commentedon Dec 8, 2020
I think you are going to need to condition this fix based on the Python version. Also, I believe Conda's Python can be built differently on OSX. Do you happen to know which Python build you are using?