Repository navigation
building wheel from sdist fails #320
Description
Activity
Do you have python3-devel installed?
Yes I do. I'm able to reproduce this at will on both CentOS 7.7 and Mac OS X 10.15.3 (Catalina) with Python 3.6, 3.7, and 3.8.
Here's the full set of commands and output that I used on CentOS 7.7 with Python 3.6. Note: I'm able to compile dozens of other Python packages using the same command both with and without Cython.
In the output below, you'll see that it compiles fine, then tries to run bdist_wheel (search for "running bdist_wheel") and then tries to compile again and that fails. Note the error at the end. That should not point at a UnixCCompiler object - it should be a string. That string gets used as a key to a dict and, in the case of running on CentOS, should be "unix". If it were, this would work.
What seems to be happening is that (somehow) your setup.py file is passing a compiler object back to a method to create a new compiler object. I've not seen any other Cython package we build using this command do that so I don't know why that is happening.
$ yum list installed | grep python3 python3.x86_64 3.6.8-10.el7 @base python3-devel.x86_64 3.6.8-10.el7 @base python3-libs.x86_64 3.6.8-10.el7 @base python3-rpm-generators.noarch 6-2.el7 @base python3-rpm-macros.noarch 3-32.el7 @base python3-setuptools.noarch 39.2.0-10.el7 @base python37.x86_64 3.7.3-1 @NFVFP-rpm-local $ ls -a . .. $ python3.6 -m venv .venv $ source .venv/bin/activate (.venv) $ pip install wheel Collecting wheel Using cached https://files.pythonhosted.org/packages/8c/23/848298cccf8e40f5bbb59009b32848a4c38f4e7f3364297ab3c3e2e2cd14/wheel-0.34.2-py2.py3-none-any.whl Installing collected packages: wheel Successfully installed wheel-0.34.2 You are using pip version 9.0.3, however version 20.0.2 is available. You should consider upgrading via the 'pip install --upgrade pip' command. (.venv) $ pip wheel --global-option build_ext --global-option --inplace uvloop==0.14.0 /home/tlevi/tmp/.venv/lib64/python3.6/site-packages/pip/commands/wheel.py:127: UserWarning: Disabling all use of wheels due to the use of --build-options / --global-options / --install-options. cmdoptions.check_install_build_global(options) Collecting uvloop==0.14.0 Using cached https://files.pythonhosted.org/packages/84/2e/462e7a25b787d2b40cf6c9864a9e702f358349fc9cfb77e83c38acb73048/uvloop-0.14.0.tar.gz Building wheels for collected packages: uvloop Running setup.py bdist_wheel for uvloop ... error Complete output from command /home/tlevi/tmp/.venv/bin/python3.6 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-1sp03mdo/uvloop/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" build_ext --inplace bdist_wheel -d /tmp/tmpre81n5yfpip-wheel-: running build_ext checking for a BSD-compatible install... /bin/install -c checking whether build environment is sane... yes checking for a thread-safe mkdir -p... /bin/mkdir -p checking for gawk... gawk checking whether make sets $(MAKE)... yes checking whether make supports nested variables... yes checking build system type... x86_64-unknown-linux-gnu checking host system type... x86_64-unknown-linux-gnu checking for gcc... gcc checking whether the C compiler works... yes checking for C compiler default output file name... a.out checking for suffix of executables... checking whether we are cross compiling... no checking for suffix of object files... o checking whether we are using the GNU C compiler... yes checking whether gcc accepts -g... yes checking for gcc option to accept ISO C89... none needed checking whether gcc understands -c and -o together... yes checking for style of include used by make... GNU checking dependency style of gcc... gcc3 checking if gcc supports -pedantic flag... yes checking for gcc way to treat warnings as errors... -Werror checking if gcc supports -fvisibility=hidden... yes checking if gcc supports -g flag... yes checking if gcc supports -std=gnu89 flag... yes checking if gcc supports -Wall flag... yes checking if gcc supports -Wextra flag... yes checking if gcc supports -Wno-long-long flag... yes checking if gcc supports -Wno-unused-parameter flag... yes checking if gcc supports -Wstrict-prototypes flag... yes checking for ar... ar checking the archiver (ar) interface... ar checking how to print strings... printf checking for a sed that does not truncate output... /bin/sed checking for grep that handles long lines and -e... /bin/grep checking for egrep... /bin/grep -E checking for fgrep... /bin/grep -F checking for ld used by gcc... /bin/ld checking if the linker (/bin/ld) is GNU ld... yes checking for BSD- or MS-compatible name lister (nm)... /bin/nm -B checking the name lister (/bin/nm -B) interface... BSD nm checking whether ln -s works... yes checking the maximum length of command line arguments... 1572864 checking whether the shell understands some XSI constructs... yes checking whether the shell understands "+="... yes checking how to convert x86_64-unknown-linux-gnu file names to x86_64-unknown-linux-gnu format... func_convert_file_noop checking how to convert x86_64-unknown-linux-gnu file names to toolchain format... func_convert_file_noop checking for /bin/ld option to reload object files... -r checking for objdump... objdump checking how to recognize dependent libraries... pass_all checking for dlltool... no checking how to associate runtime and link libraries... printf %s\n checking for archiver @FILE support... @ checking for strip... strip checking for ranlib... ranlib checking command to parse /bin/nm -B output from gcc object... ok checking for sysroot... no checking for mt... no checking if : is a manifest tool... no checking how to run the C preprocessor... gcc -E checking for ANSI C header files... yes checking for sys/types.h... yes checking for sys/stat.h... yes checking for stdlib.h... yes checking for string.h... yes checking for memory.h... yes checking for strings.h... yes checking for inttypes.h... yes checking for stdint.h... yes checking for unistd.h... yes checking for dlfcn.h... yes checking for objdir... .libs checking if gcc supports -fno-rtti -fno-exceptions... no checking for gcc option to produce PIC... -fPIC -DPIC checking if gcc PIC flag -fPIC -DPIC works... yes checking if gcc static flag -static works... no checking if gcc supports -c -o file.o... yes checking if gcc supports -c -o file.o... (cached) yes checking whether the gcc linker (/bin/ld -m elf_x86_64) supports shared libraries... yes checking whether -lc should be explicitly linked in... no checking dynamic linker characteristics... GNU/Linux ld.so checking how to hardcode library paths into programs... immediate checking whether stripping libraries is possible... yes checking if libtool supports shared libraries... yes checking whether to build shared libraries... yes checking whether to build static libraries... yes checking whether make supports nested variables... (cached) yes checking for dlopen in -ldl... yes checking for kstat_lookup in -lkstat... no checking for gethostbyname in -lnsl... yes checking for perfstat_cpu in -lperfstat... no checking for pthread_mutex_init in -lpthread... yes checking for clock_gettime in -lrt... yes checking for sendfile in -lsendfile... no checking for socket in -lsocket... no checking for special C compiler options needed for large files... no checking for _FILE_OFFSET_BITS value needed for large files... no checking sys/ahafs_evProds.h usability... no checking sys/ahafs_evProds.h presence... no checking for sys/ahafs_evProds.h... no checking that generated files are newer than configure... done configure: creating ./config.status config.status: creating Makefile config.status: creating libuv.pc config.status: executing depfiles commands config.status: executing libtool commands CC src/libuv_la-fs-poll.lo CC src/libuv_la-idna.lo CC src/libuv_la-random.lo CC src/libuv_la-inet.lo CC src/libuv_la-timer.lo CC src/libuv_la-strscpy.lo CC src/libuv_la-threadpool.lo CC src/libuv_la-version.lo CC src/libuv_la-uv-data-getter-setters.lo CC src/libuv_la-uv-common.lo CC src/unix/libuv_la-fs.lo CC src/unix/libuv_la-async.lo CC src/unix/libuv_la-core.lo CC src/unix/libuv_la-dl.lo CC src/unix/libuv_la-loop.lo CC src/unix/libuv_la-getnameinfo.lo CC src/unix/libuv_la-getaddrinfo.lo CC src/unix/libuv_la-pipe.lo CC src/unix/libuv_la-poll.lo CC src/unix/libuv_la-loop-watcher.lo CC src/unix/libuv_la-process.lo CC src/unix/libuv_la-random-devurandom.lo CC src/unix/libuv_la-signal.lo CC src/unix/libuv_la-stream.lo CC src/unix/libuv_la-tcp.lo CC src/unix/libuv_la-thread.lo CC src/unix/libuv_la-tty.lo CC src/unix/libuv_la-udp.lo CC src/unix/libuv_la-linux-core.lo CC src/unix/libuv_la-linux-inotify.lo CC src/unix/libuv_la-linux-syscalls.lo CC src/unix/libuv_la-procfs-exepath.lo CC src/unix/libuv_la-proctitle.lo CC src/unix/libuv_la-random-getrandom.lo CC src/unix/libuv_la-random-sysctl.lo CC src/unix/libuv_la-sysinfo-loadavg.lo CCLD libuv.la building 'uvloop.loop' extension creating build/temp.linux-x86_64-3.6 creating build/temp.linux-x86_64-3.6/uvloop gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I/home/tlevi/tmp/.venv/include -I/usr/include/python3.6m -I/tmp/pip-build-1sp03mdo/uvloop/vendor/libuv/include -c uvloop/loop.c -o build/temp.linux-x86_64-3.6/uvloop/loop.o -O2 uvloop/loop.c: In function ‘__pyx_f_6uvloop_4loop_4Loop__is_main_thread’: uvloop/loop.c:10041:75: warning: comparison between signed and unsigned integer expressions [-Wsign-compare] __pyx_t_1 = __Pyx_PyBool_FromLong((__pyx_v_6uvloop_4loop_MAIN_THREAD_ID == PyThread_get_thread_ident())); if (unlikely(!__pyx_t_1)) __PYX_ERR(2, 188, __pyx_L1_error) ^ creating build/lib.linux-x86_64-3.6 creating build/lib.linux-x86_64-3.6/uvloop gcc -pthread -shared -Wl,-z,relro -g build/temp.linux-x86_64-3.6/uvloop/loop.o /tmp/pip-build-1sp03mdo/uvloop/build/libuv/.libs/libuv.a -L/usr/lib64 -lpython3.6m -lrt -lpthread -o build/lib.linux-x86_64-3.6/uvloop/loop.cpython-36m-x86_64-linux-gnu.so copying build/lib.linux-x86_64-3.6/uvloop/loop.cpython-36m-x86_64-linux-gnu.so -> uvloop running bdist_wheel running build running build_py copying uvloop/__init__.py -> build/lib.linux-x86_64-3.6/uvloop copying uvloop/_patch.py -> build/lib.linux-x86_64-3.6/uvloop copying uvloop/_testbase.py -> build/lib.linux-x86_64-3.6/uvloop copying uvloop/_noop.py -> build/lib.linux-x86_64-3.6/uvloop running egg_info writing uvloop.egg-info/PKG-INFO writing dependency_links to uvloop.egg-info/dependency_links.txt writing top-level names to uvloop.egg-info/top_level.txt reading manifest file 'uvloop.egg-info/SOURCES.txt' reading manifest template 'MANIFEST.in' warning: no previously-included files matching '*' found under directory 'vendor/libuv/.git' warning: no previously-included files matching '*' found under directory 'vendor/libuv/docs' warning: no previously-included files matching '*' found under directory 'vendor/libuv/img' writing manifest file 'uvloop.egg-info/SOURCES.txt' copying uvloop/cbhandles.pxd -> build/lib.linux-x86_64-3.6/uvloop copying uvloop/cbhandles.pyx -> build/lib.linux-x86_64-3.6/uvloop copying uvloop/dns.pyx -> build/lib.linux-x86_64-3.6/uvloop copying uvloop/errors.pyx -> build/lib.linux-x86_64-3.6/uvloop copying uvloop/loop.c -> build/lib.linux-x86_64-3.6/uvloop copying uvloop/loop.pxd -> build/lib.linux-x86_64-3.6/uvloop copying uvloop/loop.pyx -> build/lib.linux-x86_64-3.6/uvloop copying uvloop/lru.pyx -> build/lib.linux-x86_64-3.6/uvloop copying uvloop/pseudosock.pyx -> build/lib.linux-x86_64-3.6/uvloop copying uvloop/request.pxd -> build/lib.linux-x86_64-3.6/uvloop copying uvloop/request.pyx -> build/lib.linux-x86_64-3.6/uvloop copying uvloop/server.pxd -> build/lib.linux-x86_64-3.6/uvloop copying uvloop/server.pyx -> build/lib.linux-x86_64-3.6/uvloop copying uvloop/sslproto.pxd -> build/lib.linux-x86_64-3.6/uvloop copying uvloop/sslproto.pyx -> build/lib.linux-x86_64-3.6/uvloop creating build/lib.linux-x86_64-3.6/uvloop/handles copying uvloop/handles/async_.pxd -> build/lib.linux-x86_64-3.6/uvloop/handles copying uvloop/handles/async_.pyx -> build/lib.linux-x86_64-3.6/uvloop/handles copying uvloop/handles/basetransport.pxd -> build/lib.linux-x86_64-3.6/uvloop/handles copying uvloop/handles/basetransport.pyx -> build/lib.linux-x86_64-3.6/uvloop/handles copying uvloop/handles/check.pxd -> build/lib.linux-x86_64-3.6/uvloop/handles copying uvloop/handles/check.pyx -> build/lib.linux-x86_64-3.6/uvloop/handles copying uvloop/handles/handle.pxd -> build/lib.linux-x86_64-3.6/uvloop/handles copying uvloop/handles/handle.pyx -> build/lib.linux-x86_64-3.6/uvloop/handles copying uvloop/handles/idle.pxd -> build/lib.linux-x86_64-3.6/uvloop/handles copying uvloop/handles/idle.pyx -> build/lib.linux-x86_64-3.6/uvloop/handles copying uvloop/handles/pipe.pxd -> build/lib.linux-x86_64-3.6/uvloop/handles copying uvloop/handles/pipe.pyx -> build/lib.linux-x86_64-3.6/uvloop/handles copying uvloop/handles/poll.pxd -> build/lib.linux-x86_64-3.6/uvloop/handles copying uvloop/handles/poll.pyx -> build/lib.linux-x86_64-3.6/uvloop/handles copying uvloop/handles/process.pxd -> build/lib.linux-x86_64-3.6/uvloop/handles copying uvloop/handles/process.pyx -> build/lib.linux-x86_64-3.6/uvloop/handles copying uvloop/handles/stream.pxd -> build/lib.linux-x86_64-3.6/uvloop/handles copying uvloop/handles/stream.pyx -> build/lib.linux-x86_64-3.6/uvloop/handles copying uvloop/handles/streamserver.pxd -> build/lib.linux-x86_64-3.6/uvloop/handles copying uvloop/handles/streamserver.pyx -> build/lib.linux-x86_64-3.6/uvloop/handles copying uvloop/handles/tcp.pxd -> build/lib.linux-x86_64-3.6/uvloop/handles copying uvloop/handles/tcp.pyx -> build/lib.linux-x86_64-3.6/uvloop/handles copying uvloop/handles/timer.pxd -> build/lib.linux-x86_64-3.6/uvloop/handles copying uvloop/handles/timer.pyx -> build/lib.linux-x86_64-3.6/uvloop/handles copying uvloop/handles/udp.pxd -> build/lib.linux-x86_64-3.6/uvloop/handles copying uvloop/handles/udp.pyx -> build/lib.linux-x86_64-3.6/uvloop/handles creating build/lib.linux-x86_64-3.6/uvloop/includes copying uvloop/includes/__init__.py -> build/lib.linux-x86_64-3.6/uvloop/includes copying uvloop/includes/compat.h -> build/lib.linux-x86_64-3.6/uvloop/includes copying uvloop/includes/consts.pxi -> build/lib.linux-x86_64-3.6/uvloop/includes copying uvloop/includes/debug.h -> build/lib.linux-x86_64-3.6/uvloop/includes copying uvloop/includes/debug.pxd -> build/lib.linux-x86_64-3.6/uvloop/includes copying uvloop/includes/flowcontrol.pxd -> build/lib.linux-x86_64-3.6/uvloop/includes copying uvloop/includes/fork_handler.h -> build/lib.linux-x86_64-3.6/uvloop/includes copying uvloop/includes/python.pxd -> build/lib.linux-x86_64-3.6/uvloop/includes copying uvloop/includes/stdlib.pxi -> build/lib.linux-x86_64-3.6/uvloop/includes copying uvloop/includes/system.pxd -> build/lib.linux-x86_64-3.6/uvloop/includes copying uvloop/includes/uv.pxd -> build/lib.linux-x86_64-3.6/uvloop/includes running build_ext error: don't know how to compile C/C++ code on platform 'posix' with '<distutils.unixccompiler.UnixCCompiler object at 0x7f9874fcce48>' compiler ---------------------------------------- Failed building wheel for uvloop Running setup.py clean for uvloop Failed to build uvloop ERROR: Failed to build one or more wheelsI think I may have found it.
In your setup.py, you're short-circuiting the call to super().initialize_options() here. That means you're forcing subsequent commands to inherit variables that they expect to set for themselves (like self.compiler, which is what is causing the build failure). The same applies to finalize_options().
If you simply want to avoid your options from being reset, you could move the 4 lines that set your options into the body of that if and always call super()initialize_options().
I'm not sure why you're avoiding those calls to super().foo() in initialize_options() and finalize_options(), but those seem to be what is messing up builds with multiple commands on the same line.
Hm. Good catch. I'll take a look and see if this hack is still necessary.
I know this is old, but it still fails. Any luck deciding on whether that hack was still required? As it stands, uvloop still can't be built from sdist 😦
- added a commit that references this issue
on Sep 9, 2022 - added a commit that references this issue
on Sep 9, 2022
PYTHONASYNCIODEBUGin env?: N/AWhat I did:
pip wheel --global-option build_ext --global-option --inplace uvloop==0.14.0What I expected:
The command would build uvloop and bundle it as a wheel for me.
What I saw:
The command fails with
error: don't know how to compile C/C++ code on platform 'posix' with '<distutils.unixccompiler.UnixCCompiler object at 0x7f054bc1fa10>' compilerafter build_ext runs (e.g. during the processing of bdist_wheel).The (heavily) snipped output from the above command is:
Other observations:
The pip wheel command essentially downloads your sdist, unwinds it, cd's to that directory, and runs
python setup.py build_ext --inplace bdist_wheelThis mimics your make file other than the fact that build_ext and bdist_wheel are on one command line.If I manually run
python setup.py build_ext --inplace && python setup.py bdist_wheel, then the wheel is successfully built. I can't tell exactly how, but it seems like something is building the extension in bdist_wheel instead of simply inheriting what was build by build_ext. That, in turn, is triggering what may be a latent distutils issue that I documented here.