Skip to content

Uploading wheels to pypi #1

Closed
Closed
Uploading wheels to pypi#1

Description

@ghost
ghost
opened on Oct 8, 2017

FYI I'm working on uploading wheels to pypi over at https://github.com/xoviat/build-python-flint. After some pain, I've managed to build python-flint on windows and will probably have builds for OSX and manylinux as well.

Activity

  2. isuruf commented on Oct 9, 2017

    @isuruf
    isuruf
    on Oct 9, 2017
    Member

    @xoviat, you have to be careful about how you build gmp/mpir when you want to distribute. Here's how I do it, https://github.com/symengine/symengine-wheels/blob/master/config.sh#L24. Note that you have to enable --enable-fat or build for a generic x86 machine instead of ivybridge in the travis machines.

  4. isuruf commented on Oct 9, 2017

    @isuruf
    isuruf
    on Oct 9, 2017
    Member

    Yes. You need to add --enable-fat for Linux and add --host=x86_64-darwin-none (not sure) on OSX (--enable-fat for mpir doesn't work on Mac). Or you can switch to gmp instead of mpir can add --enable-fat for Linux and Mac both.

  5. ghost closed this as completedon Dec 4, 2017

  6. alisianoi commented on Oct 2, 2019

    @alisianoi
    alisianoi
    on Oct 2, 2019

    Here is a wrapper project I made that builds wheels for python-flint in case anyone needs those:

    https://gitlab.com/alisianoi/flint-py

    The README.md there describes the build process a bit. The wheels are on PyPI:

    https://pypi.org/project/flint-py/

    I bumped the version from 0.3.0 to 0.3.5 but all the changes were to the build process

  7. fredrik-johansson commented on Oct 20, 2019

    @fredrik-johansson
    fredrik-johansson
    on Oct 20, 2019
    Collaborator

    @alisianoi Thanks for sharing! Would it make sense to integrate this with python-flint instead of having a separate project? It doesn't make much difference either way as long as it's maintained in some form.

  8. oscarbenjamin commented on Apr 3, 2021

    @oscarbenjamin
    oscarbenjamin
    on Apr 3, 2021
    Collaborator

    I think it would definitely make sense to integrate this with python-flint. Setting this up to compile is not completely trivial so it would be much better if there are precompiled wheels for Windows, OSX and manylinux. They could be built in github actions.

  9. fredrik-johansson commented on Apr 3, 2021

    @fredrik-johansson
    fredrik-johansson
    on Apr 3, 2021
    Collaborator

    @oscarbenjamin Indeed. Unfortunately I don't have much time as I'd like to maintain python-flint, and it'd be awesome if someone else was willing to step up. Not least considering all the new multivariate polynomial stuff (and more) in the latest Flint that hasn't been wrapped in python-flint yet.

  10. oscarbenjamin commented on Apr 3, 2021

    @oscarbenjamin
    oscarbenjamin
    on Apr 3, 2021
    Collaborator

    Okay so maybe reopen this issue and I'll see what I can do.

    I've almost managed to build a wheel on OSX (build script is below) but I'm hitting against two problems right now:

    1. Although I can install with pip install . and I can build a wheel with pip wheel . when I try to install the wheel like pip install *.whl I get this error:

       ```
 $ pip install python_flint-0.3.0-cp38-cp38-macosx_11_0_x86_64.whl
 ERROR: python_flint-0.3.0-cp38-cp38-macosx_11_0_x86_64.whl is not a supported wheel on this platform.
 ```

      I think that's telling me that the name of the wheel specifies a system different from this one somehow (even though I built it here!). I've tried a few renames of the file but haven't managed to get pip to accept it.

    1. I'll need to get delocate to be able to bundle flint/arb to make the wheel relocatable but that also doesn't seem to be working and I think it's because there is no package in python-flint (it's just a flint.* extension module). The delocate docs suggest that you need to make a package like:

       ```
 flint/
     __init__.py   # this just does from .flint import *
     flint.so
 ```

      Then the shared libraries can be bundled into the package folder:

      https://github.com/matthew-brett/delocate#external-library-is-not-bundled-into-wheel

    The script I have so far is:

    #!/usr/bin/env bash

set -o errexit

PYTHON=python3.8
FLINTVER=2.7.1
ARBVER=2.19.0

mkdir -p local
PREFIX=$(pwd)/local

curl -O https://www.flintlib.org/flint-$FLINTVER.tar.gz
curl -O -L https://github.com/fredrik-johansson/arb/archive/refs/tags/$ARBVER.tar.gz
mv $ARBVER.tar.gz arb-$ARBVER.tar.gz
git clone https://github.com/fredrik-johansson/python-flint

tar xf flint-$FLINTVER.tar.gz
cd flint-$FLINTVER
  ./configure --prefix=$PREFIX
  make -j4
  make install
cd ..

tar xf arb-$ARBVER.tar.gz
cd arb-$ARBVER
  ./configure --prefix=$PREFIX --with-flint=$PREFIX
  make -j4
  make install
cd ..

$PYTHON -m venv venv
source venv/bin/activate
pip install -U pip wheel
pip install numpy cython==0.27.3
# N.B. bugs in both older and newer Cython versions...

cd python-flint
  pip wheel . --global-option=build_ext \
                --global-option="-I$PREFIX/include" \
                --global-option="-L$PREFIX/lib"
  pip install . --global-option=build_ext \
                --global-option="-I$PREFIX/include" \
                --global-option="-L$PREFIX/lib"
cd ..

python -c 'import flint'
  11. fredrik-johansson
    reopened this on Apr 3, 2021

  12. oscarbenjamin commented on Apr 3, 2021

    @oscarbenjamin
    oscarbenjamin
    on Apr 3, 2021
    Collaborator

    Okay I've got this working to make some usable OSX wheels. Changes needed in python_flint are:

    # Move src to src/flint
mv src flint
mkdir src
mv flint src

    Apply this diff in setup.py:

    diff --git a/setup.py b/setup.py
index 0a739c9..448466a 100644
--- a/setup.py
+++ b/setup.py
@@ -21,7 +21,7 @@ default_include_dirs += [
 
 ext_modules = [
     Extension(
-        "flint", ["src/pyflint.pyx"],
+        "flint.flint", ["src/flint/pyflint.pyx"],
         libraries=libraries,
         library_dirs=default_lib_dirs,
         include_dirs=default_include_dirs)
@@ -34,6 +34,8 @@ setup(
     name='python-flint',
     cmdclass={'build_ext': build_ext},
     ext_modules=ext_modules,
+    packages=['flint'],
+    package_dir={'': 'src'},
     description='Bindings for FLINT and Arb',
     version='0.3.0',
     url='https://github.com/python-flint/python-flint',

    That gives the following package structure:

    flint/
    __init__.py
    flint.so

    Once those changes are made then this script generates a relocatable OSX wheel on OSX 11.1 with either Python 3.7 or 3.8 from python.org installers. It does not work with python 3.9. I think that is because a newer cython version is needed but the newer cython has a bug that doesn't work for python_flint (#13)

    EDIT (the changes above were made in #14 so this script works directly with master now):

    #!/usr/bin/env bash

set -o errexit

PYTHON=python3.8
GMPVER=6.2.1
MPFRVER=4.1.0
FLINTVER=2.7.1
ARBVER=2.19.0
PYTHONFLINTVER=0.3.0

mkdir -p local
PREFIX=$(pwd)/local

curl -O https://gmplib.org/download/gmp/gmp-$GMPVER.tar.xz
curl -O https://www.mpfr.org/mpfr-current/mpfr-$MPFRVER.tar.gz
curl -O https://www.flintlib.org/flint-$FLINTVER.tar.gz
curl -O -L https://github.com/fredrik-johansson/arb/archive/refs/tags/$ARBVER.tar.gz
mv $ARBVER.tar.gz arb-$ARBVER.tar.gz
rm -rf python-flint
git clone https://github.com/fredrik-johansson/python-flint

tar xf gmp-$GMPVER.tar.xz
cd gmp-$GMPVER
  ./configure --prefix=$PREFIX --enable-fat --enable-shared --host=x86_64-apple-darwin
  make -j3
  make install
cd ..

tar -xf mpfr-$MPFRVER.tar.gz
cd mpfr-$MPFRVER
  ./configure --prefix=$PREFIX --with-gmp=$PREFIX
  make -j3
  make install
cd ..

tar xf flint-$FLINTVER.tar.gz
cd flint-$FLINTVER
  ./configure --prefix=$PREFIX --with-gmp=$PREFIX --with-mpfr=$PREFIX
  make -j3
  make install
cd ..

tar xf arb-$ARBVER.tar.gz
cd arb-$ARBVER
  ./configure --prefix=$PREFIX --with-flint=$PREFIX --with-gmp=$PREFIX --with-mpfr=$PREFIX
  make -j3
  make install
cd ..

$PYTHON -m venv venv
source venv/bin/activate
pip install -U pip wheel delocate
pip install numpy cython==0.27.3
# N.B. bugs in both older and newer Cython versions...

cd python-flint
  pip wheel .   --global-option=build_ext \
                --global-option="-I$PREFIX/include" \
                --global-option="-L$PREFIX/lib"
  delocate-wheel -w .. *.whl
cd ..

tags="cp37-cp37m cp38-cp38"
osx_from="11_0"
osx_to="10_9"
for tag in $tags; do
  wheelold=python_flint-$PYTHONFLINTVER-$tag-macosx_${osx_from}_x86_64.whl
  wheelnew=python_flint-$PYTHONFLINTVER-$tag-macosx_${osx_to}_x86_64.whl
  if [ -f $wheelold ]; then
    mv $wheelold $wheelnew
  fi
  if [ -f $wheelnew ]; then
    wheelfinal=$wheelnew
  fi
done

echo ------------------------------------------
echo
echo Built wheel: $wheelfinal
echo
echo Link dependencies:
delocate-listdeps $wheelfinal
echo
pip install $wheelfinal
echo
echo Demonstration:
echo
python -c 'import flint; print("(3/2)**2 =", flint.fmpq(3, 2)**2)'
echo
echo Done!
echo
echo ------------------------------------------

  13. oscarbenjamin commented on Apr 5, 2021

    @oscarbenjamin
    oscarbenjamin
    on Apr 5, 2021
    Collaborator

    The changes for python_flint that I described above were made in #14 so the script above builds a wheel from master now.

    It doesn't build GMP or MPFR but still works on my system because I have those installed. I guess I should get it to compile them as well then I could see how to get it working in Github Actions.

    Ultimately python_flint should probably use cibuildwheels to build wheels on Windows, OSX and Linux but I'll see if I can get a single wheel build working directly first.

  14. oscarbenjamin commented on Apr 5, 2021

    @oscarbenjamin
    oscarbenjamin
    on Apr 5, 2021
    Collaborator

    I've updated the script above so that it builds GMP and MPFR as well. I'm unsure about one thing which is that delocate reports the depedencies as:

    Link dependencies:
@loader_path/.dylibs/libarb-2.10.0.dylib
@loader_path/.dylibs/libflint-15.dylib
@loader_path/libflint-15.dylib
@loader_path/libgmp.10.dylib
@loader_path/libmpfr.6.dylib

    That seems to list libflint twice. I'm not sure if something is messed up there or how to fix it...

  15. oscarbenjamin commented on Dec 3, 2022

    @oscarbenjamin
    oscarbenjamin
    on Dec 3, 2022
    Collaborator

    At the time of writing after merging #22 the situation is that wheels are built in CI for Linux and OSX using the latest releases of GMP, FMPR, Flint and Arb:
    https://github.com/fredrik-johansson/python-flint/blob/4ec3a4b9abb6afb71b68e9c181e9f51681cc30f2/bin/build_variables.sh#L16-L21
    I am trying to get a wheel built on Windows now using MSYS2 and the mingw-w64 tool chain (as is also used by numpy and gmpy2 for their PyPI wheels). As of now I've managed to build the dependencies and then build a wheel but after installing the wheel import flint fails with missing DLL error:

    $ python -c 'import flint'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\pc\src\flint\python-flint\install_venv\lib\site-packages\flint\__init__.
py", line 1, in <module>
    from ._flint import *
ImportError: DLL load failed while importing _flint: The specified module could not be fo
und.

    Although this isn't yet working I though I'd record the steps I followed for now:

    1. Install CPython 3.10 (from python.org, not MSYS2) and add to PATH.
    2. Install MSYS2
    3. Install mingw-w64 toolchain in MSYS2
    4. Run the bin/build_dependencies.sh script to build GMP, MPFR, Flint and Arb in .local.
    5. Make libpython310.a and place in the CPython libs directory.
    6. Run modified setup.py to build the wheel.
    7. Use pip to install the wheel into a venv
    8. Attempt to import flint

    Longer explanation:

    1. I think it's important that to use the official CPython binary releases from here:
      https://www.python.org/downloads/
      PyPI wheels should be compatible with those builds of CPython rather than any potentially different binary in MSYS2, Cygwin etc. Specifically I installed Python 3.10.8:
      https://www.python.org/ftp/python/3.10.8/python-3.10.8-amd64.exe

    2. I installed MSYS2 from here and specifically chose the msys2-x86_64-20221028.exe installer:
      https://repo.msys2.org/distrib/x86_64/
      https://repo.msys2.org/distrib/x86_64/msys2-x86_64-20221028.exe
      MSYS2 uses a rolling release model making it hard to pin precise versions of the toolchain for reproducibility so it is always recommended just to use the latest versions of everything. Alternatively I think that winlibs.com provides versioned binaries of mostly the same stuff that might make it easier to pin the toolchain but I haven't tried using that. After install you can run MSYS2 by choosing "MSYS2 MINGW64" from the start menu.

    3. To install the mingw-w64 toolchain use the MSYS2 package manager:

      $ pacman -S mingw-w64-x86_64-gcc m4 make

    This will just install the latest versions of a bunch of packages and I don't think it's possible (or at least it's not recommended) to try to pin them.

    1. One advantage of using MSYS2 rather than MSVC is that it provides a unix-like environment so that the build script should be more or less the same as for actual unix systems. I suspect that at least some adaptations would be needed to the bin/build_dependencies.sh script to get it working (e.g. to provide mingw-w64 related build options) but for now at least I used the script without modification.

      It takes about an hour to run on this computer which is a lot slower than in Linux (more like 5 minutes).
      Although the script appears to complete successfully there are a lot of compiler warnings.

      1. For GMP there are many warnings like dllexport attribute ignored. I'm not sure if those are an actual problem or not.

      2. For Flint and Arb there are many warnings about size mismatches like:

        In file included from C:/Users/pc/src/flint/python-flint/.local/src/arb-2.23.0/acb_poly.h:22,
from compose.c:14:
In function 'acb_set',
inlined from '_acb_poly_compose_axnc' at compose.c:43:13:
C:/Users/pc/src/flint/python-flint/.local/src/arb-2.23.0/acb.h:131:5: warning: 'arb_set'
reading 48 bytes from a region of size 32 [-Wstringop-overread]
131 |     arb_set(acb_realref(z), acb_realref(x));
|     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
C:/Users/pc/src/flint/python-flint/.local/src/arb-2.23.0/acb.h:131:5: note: referencing
argument 2 of type 'const arb_struct[1]'
In file included from C:/Users/pc/src/flint/python-flint/.local/src/arb-2.23.0/acb.h:23:
C:/Users/pc/src/flint/python-flint/.local/src/arb-2.23.0/arb.h: In function '_acb_poly_compose_axnc':
C:/Users/pc/src/flint/python-flint/.local/src/arb-2.23.0/arb.h:149:6: note: in a call to
function 'arb_set'
149 | void arb_set(arb_t x, const arb_t y);|      ^~~~~~~

        It's possible that there is some confusion over e.g. the size of fmpz perhaps related to long or long long being different in mingw-w64 or Windows or something but I haven't investigated.

        Although build_dependencies.sh completes without any "errors" I suspect that there are problems with the binaries that have been created and I'm also confused about the output files generated. The script installs into the .local prefix and results in these files:

        $ ls .local/include
acb.h            acb_modular.h    arb_mat.h    double_interval.h  mag.h
acb_calc.h       acb_poly.h       arb_poly.h   flint              mpf2mpfr.h
acb_dft.h        arb.h            arf.h        fmpr.h             mpfr.h
acb_dirichlet.h  arb_calc.h       bernoulli.h  fmpz_extras.h      partitions.h
acb_elliptic.h   arb_fmpz_poly.h  bool_mat.h   fmpzi.h
acb_hypgeom.h    arb_fpwrap.h     dirichlet.h  gmp.h
acb_mat.h        arb_hypgeom.h    dlog.h       hypgeom.h
$ ls .local/lib
libarb-2.dll  libarb.dll.2    libgmp.dll.a  libmpfr.dll.a  pkgconfig
libarb.dll    libflint.dll.a  libgmp.la     libmpfr.la
$ ls .local/bin
libflint-17.dll  libflint.dll.a  libgmp-10.dll  libmpfr-6.dll

        So e.g. the lib directory contains libarb.dll but not libflint.dll and instead we have libflint-17.dll but it has landed in the bin directory. I'm not sure exactly what combination of .dll, .dll.aand.lafiles are needed to build the wheel or to be available at runtime and what directories they are supposed to be in. I have tried copying all of the DLL files into the installedsite-packages/flintdirectory and renaming some likelibflint-17.dll -> libflint.dlland I've tried adding them toPATH` and other things but I still get the "missing DLL" error when importing flint. Frustratingly the error message doesn't say which DLL is failing to load.

    2. To make libpython310.a you need the migw-w64 gendef.exe so install that:

       $ pacman -S mingw-w64-x86_64-tools-git

      Then cd into the directory of your Python 3.10 installation and create the file e.g.:

      $ cd /c/Users/pc/AppData/Local/Programs/Python/Python310/
$ gendef python310.dll
$ dlltool --dllname python310.dll --def python310.def --output-lib libpython310.a
$ mv libpython310.a libs

    3. Since we have the dependencies in .local we need to tell distutils to add those to the includ epaths etc and we also need to tell it to use the mingw-w64 compiler:

      C_INCLUDE_PATH=.local/include/ LIBRARY_PATH=.local/lib/ .local/venv/Scripts/python setup.py build_ext -cmingw32 -f bdist_wheel

      I created a venv in .local to install the build dependencies. This needs numpy, Cython and wheel to be installed from pypi. The -cmingw32 does not mean that it is using mingw: this is still mingw-w64.

      I changed setup.py to have libraries = ["arb", "flint", "gmp", "mpfr] (i.e. remove pthreads) and also include_dirs=default_include_dirs + ['.local/include']

      This build step is a lot faster (about 1 minute) and also produces lots of warnings like

      src/flint\pyflint.c:14960:101: note: expected '__pyx_t_5flint_6_flint_fmpz_struct *' {aka 'long int *'} but argument is of type 'fmpz *' {aka 'long long int *'}
14960 | static CYTHON_INLINE int __pyx_f_5flint_6_flint_fmpz_set_pylong(__pyx_t_5flint_6_flint_fmpz_struct *__pyx_v_x, PyObject *__pyx_v_obj) {

      Again there seems to be some confusion about long vs long long. This is potentially to do with Windows having a 32 bit long but I'm not sure. I have seen mention in some places that maybe mingw-w64's long long is incompatible with the msvc one somehow.

    After all of these steps there should be a wheel in the dist subdirectory:

    $ ls dist
python_flint-0.3.0-cp310-cp310-win_amd64.whl

    As mentioned before though I found that after installing the wheel I couldn't import it because of the missing dll error.

  16. 25 remaining items

  17. andyfaff commented on Jan 8, 2023

    @andyfaff
    andyfaff
    on Jan 8, 2023

    If you're only compiling c code and not fortran, then can't you just use the normal build chain and avoid mingw64.

  18. oscarbenjamin commented on Jan 8, 2023

    @oscarbenjamin
    oscarbenjamin
    on Jan 8, 2023
    Collaborator

    To be fair I haven't actually tried to build GMP with MSVC. I was really just following the approach taken by gmpy since I know they've been working at this for longer.

    The problem with MSVC is compiling GMP. This is also why gmpy uses mingw64 for building the C dependencies (gmpy2/gmpy2#352 (comment)).

    The attraction of GMP is that it is really fast at integer arithmetic and much of what this collection of C libraries does is arranged around leveraging that speed by using fast integer multiplication to speed up everything else. GMP gets this speed in part by having a lot of hand-written assembly code for different CPUs:
    https://gmplib.org/repo/gmp-6.2/file/tip/mpn
    With MSVC we can't use --enable-fat which is what is needed to bundle the code for all these architecture variations and have the right one selected at runtime: we would need to pin a lowest common denominator at build time.

    There is a fork of GMP called MPIR which was intended to have better MSVC support among other things. I think that's what the current conda packages of python-flint for Windows use. MPIR is no longer maintained though so I think it's better to use GMP.

  19. oscarbenjamin commented on Jan 21, 2023

    @oscarbenjamin
    oscarbenjamin
    on Jan 21, 2023
    Collaborator

    I've just merged #42 which builds and tests wheels for macos on arm64. That means that CI now builds and tests the following wheels:

    • Windows x86-64
    • manylinux x86-64
    • OSX x86-64
    • OSX arm64

    I think that's good to go and it's time for a new release of python_flint so we can let people test these wheels out.

  20. BrianGladman commented on Feb 8, 2023

    @BrianGladman
    BrianGladman
    on Feb 8, 2023

    To be fair I haven't actually tried to build GMP with MSVC. I was really just following the approach taken by gmpy since I know they've been working at this for longer.

    The problem with MSVC is compiling GMP. This is also why gmpy uses mingw64 for building the C dependencies (aleaxit/gmpy#352 (comment)).
    Oscar> The attraction of GMP is that it is really fast at integer arithmetic and much of what this collection of C libraries does is arranged around leveraging that speed by using fast integer multiplication to speed up everything else. GMP gets this speed in part by having a lot of hand-written assembly code for different CPUs: https://gmplib.org/repo/gmp-6.2/file/tip/mpn With MSVC we can't use --enable-fat which is what is needed to bundle the code for all these architecture variations and have the right one selected at runtime: we would need to pin a lowest common denominator at build time.

    There is a fork of GMP called MPIR which was intended to have better MSVC support among other things. I think that's what the current conda packages of python-flint for Windows use. MPIR is no longer maintained though so I think it's better to use GMP.

    Hi Oscar, MPIR is still fully supported on WIndows x64 (by me) and I have a complete working build chain using Visual Studio for MPIR, MPFR, MPC, FLINT2 and ARB. I support GMPY2 and I would hope to support Python Flint on WIndows x64 using MPIR. If I can help in what you are doing please let me know.

  21. oscarbenjamin commented on Aug 9, 2023

    @oscarbenjamin
    oscarbenjamin
    on Aug 9, 2023
    Collaborator

    Last night I pushed a 0.4.0 release of python-flint to PyPI with binary wheels for Windows x86-64, OSX x86-64 and arm64 and manylinux_2_17 x86-64.
    https://pypi.org/project/python-flint/#files

    I was hoping I could close this issue but unfortunately I immediately discovered a problem with the Linux wheels when testing them locally:

    $ pip install 0.4.0/python_flint-0.4.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl 
Processing ./0.4.0/python_flint-0.4.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Installing collected packages: python-flint
Successfully installed python-flint-0.4.0

[notice] A new release of pip available: 22.3.1 -> 23.2.1
[notice] To update, run: pip install --upgrade pip
$ python -c 'import flint; flint.fmpz("1")'
Illegal instruction (core dumped)

    I've deleted the Linux wheels from PyPI now so pip install python-flint goes back to trying to build from the sdist on Linux. I am investigating the problem with the Linux wheels and will write a followup comment about these shortly.

    In the mean time though pip install python-flint should now install prebuilt binaries on Windows or OSX. I would appreciate if anyone could report testing these wheels which you can do like:

    pip install python-flint
git clone https://github.com/fredrik-johansson/python-flint.git
python python-flint/test/test.py  # takes about 1 second
python python-flint/test/dtest.py  # takes a couple of minutes

    I have tested locally on Windows and OSX x86-64 but I don't immediately have access to OSX arm46 (i.e. Apple M1 hardware) to test there. Theoretically all these wheels were tested in CI but as seen for the Linux wheels that is not enough to guarantee that they work outside of CI.

  22. oscarbenjamin commented on Aug 9, 2023

    @oscarbenjamin
    oscarbenjamin
    on Aug 9, 2023
    Collaborator

    So the Linux wheels seem to be double badged as manylines_2_17 and manylinux2014. I'm not sure how that happened. I can't upload an example here because the file is too large. This is the contents of the file:

    $ wheel unpack python_flint-0.4.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl 
Unpacking to: python_flint-0.4.0...OK
$ tree python_flint-0.4.0
python_flint-0.4.0
├── flint
│   ├── _flint.cpython-311-x86_64-linux-gnu.so
│   └── __init__.py
├── python_flint-0.4.0.dist-info
│   ├── LICENSE
│   ├── METADATA
│   ├── RECORD
│   ├── top_level.txt
│   └── WHEEL
└── python_flint.libs
    ├── libarb-4c2cc9a4.so.2.14.0
    ├── libflint-c1ff2639.so.17.0.0
    ├── libgmp-bfc87f94.so.10.5.0
    └── libmpfr-90ec1309.so.6.1.0

3 directories, 11 files

    It looks like auditwheel has bundled the right .so files for the immediate dependencies but has not bundled anything else. I am not sure if that should be considered correct or not.

    I guess we can check with auditwheel:

    $ auditwheel show python_flint-0.4.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl 

python_flint-0.4.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
is consistent with the following platform tag:
"manylinux_2_17_x86_64".

The wheel references external versioned symbols in these
system-provided shared libraries: libc.so.6 with versions
{'GLIBC_2.3', 'GLIBC_2.7', 'GLIBC_2.14', 'GLIBC_2.2.5'},
libpthread.so.0 with versions {'GLIBC_2.3.2', 'GLIBC_2.2.5',
'GLIBC_2.3.4'}, libm.so.6 with versions {'GLIBC_2.2.5'}

This constrains the platform tag to "manylinux_2_17_x86_64". In order
to achieve a more compatible tag, you would need to recompile a new
wheel from source on a system with earlier versions of these
libraries, such as a recent manylinux image.

    In CI when building the wheels auditwheel says:

    Repairing wheel...
  
      + sh -c 'auditwheel repair -w /tmp/cibuildwheel/repaired_wheel /tmp/cibuildwheel/built_wheel/python_flint-0.4.0-cp311-cp311-linux_x86_64.whl'
  INFO:auditwheel.main_repair:Repairing python_flint-0.4.0-cp311-cp311-linux_x86_64.whl
  INFO:auditwheel.wheeltools:Previous filename tags: linux_x86_64
  INFO:auditwheel.wheeltools:New filename tags: manylinux_2_17_x86_64, manylinux2014_x86_64
  INFO:auditwheel.wheeltools:Previous WHEEL info tags: cp311-cp311-linux_x86_64
  INFO:auditwheel.wheeltools:New WHEEL info tags: cp311-cp311-manylinux_2_17_x86_64, cp311-cp311-manylinux2014_x86_64
  INFO:auditwheel.main_repair:
  Fixed-up wheel written to /tmp/cibuildwheel/repaired_wheel/python_flint-0.4.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
      + /opt/python/cp38-cp38/bin/python -c 'import sys, json, glob; json.dump(glob.glob('"'"'/tmp/cibuildwheel/repaired_wheel/*.whl'"'"'), sys.stdout)'

    The configuration for this is here:
    https://github.com/fredrik-johansson/python-flint/blob/bde3dbdb37766c5d9d66184b886611f33d15578f/.github/workflows/buildwheel.yml#L34-L54
    So we are asking for the manylinux2014 image but somehow getting something with manylinux_2_17.

    I think that I have previously seen some issue with downloading a Linux wheel from CI and that possibly at the time I "fixed" it by using a different manylinux image: at least that is the first thing I can try.

  23. oscarbenjamin commented on Aug 9, 2023

    @oscarbenjamin
    oscarbenjamin
    on Aug 9, 2023
    Collaborator

    It seems that NumPy's wheels are also double badged and auditwheel reports a similar result:

    $ auditwheel show numpy-1.25.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl 

numpy-1.25.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
is consistent with the following platform tag:
"manylinux_2_17_x86_64".

The wheel references external versioned symbols in these
system-provided shared libraries: libm.so.6 with versions
{'GLIBC_2.2.5'}, libgcc_s.so.1 with versions {'GCC_4.2.0',
'GCC_4.3.0', 'GCC_3.0', 'GCC_3.4', 'GCC_3.3', 'GCC_4.8.0'},
libquadmath-96973f99.so.0.0.0 with versions {'QUADMATH_1.0'},
libc.so.6 with versions {'GLIBC_2.14', 'GLIBC_2.4', 'GLIBC_2.17',
'GLIBC_2.3', 'GLIBC_2.3.4', 'GLIBC_2.6', 'GLIBC_2.3.2', 'GLIBC_2.2.5',
'GLIBC_2.10', 'GLIBC_2.7'}, libpthread.so.0 with versions
{'GLIBC_2.3.2', 'GLIBC_2.3.4', 'GLIBC_2.2.5'},
libgfortran-040039e1.so.5.0.0 with versions {'GFORTRAN_8'}

This constrains the platform tag to "manylinux_2_17_x86_64". In order
to achieve a more compatible tag, you would need to recompile a new
wheel from source on a system with earlier versions of these
libraries, such as a recent manylinux image.

    Also the contents of the wheel seem similar at least for what is bundled:

    └── numpy.libs
    ├── libgfortran-040039e1.so.5.0.0
    ├── libopenblas64_p-r0-5007b62f.3.23.dev.so
    └── libquadmath-96973f99.so.0.0.0

  24. oscarbenjamin commented on Aug 9, 2023

    @oscarbenjamin
    oscarbenjamin
    on Aug 9, 2023
    Collaborator

    I think that I have previously seen some issue with downloading a Linux wheel from CI and that possibly at the time I "fixed" it by using a different manylinux image: at least that is the first thing I can try.

    I've tried manylinux2010 (#46) and manylinux_2_24 (#47) but in all cases I get the same crash.

    The problem is specifically in string conversion e.g. converting an fmpz to a decimal string so my guess is that that codepath goes through to libm or libc somehow and then some ABI compatibility breaks. I don't know whether any of GMP, MPFR, Flint or Arb is using libm or libc in a non-standard way somehow that manylinux/auditwheel just can't account for when attempting to produce a relocatable wheel.

    Converting to a hex string works for sufficiently small fmpz:

    >>> import flint
>>> hex(flint.fmpz(2))
'0x2'
>>> hex(flint.fmpz(2<<2))
'0x8'
>>> hex(flint.fmpz(2<<61))
'0x0'
>>> hex(flint.fmpz(2<<60))
'0x2000000000000000'
>>> hex(flint.fmpz(2<<62))
Illegal instruction (core dumped)

    Worth noting that the internal representation for fmpz changes at 2**62 from being a native C long (unsigned?) to being a pointer to a GMP mpz. So somehow at that point the hex conversion fails. Converting to a decimal string fails at much smaller sizes:

    >>> import flint
>>> str(flint.fmpz(0))
'0'
>>> str(flint.fmpz(1))
Illegal instruction (core dumped)

    Conversion is done using fmpz_get_str:
    https://github.com/fredrik-johansson/python-flint/blob/bde3dbdb37766c5d9d66184b886611f33d15578f/src/flint/fmpz.pyx#L166-L184
    For Flint 2.9.0 (which we use here) that call goes straight through to gmp:
    https://github.com/flintlib/flint2/blob/e143df4b0f19d2f841e36234a12b69f48c4359b9/fmpz/get_str.c#L17
    That code seems to be completely rewritten without GMP in Flint 3.0.0-alpha1:
    https://github.com/flintlib/flint2/blob/0447e52c21d559b0b581ac6ed9f3e8b2380830bb/src/fmpz/get_str.c#L237

    I have a branch that already uses flint-3.0.0-alpha1 in gh-43! Downloading those wheels on Linux also fails though but just not in exactly the same way. It works up to 2**62-1 but crashes after that:

    (Pdb)  p str(flint.fmpz(2**62-1)) == str(2**62-1)
True
(Pdb)  p str(flint.fmpz(2**62)) == str(2**62)
Illegal instruction (core dumped)

    Even if using flint-3.0.0-alpha1 did work that is really just working around the ABI issue. Somehow cibuildwheel/manylinux/auditwheel are not doing their job properly because the wheel is not relocatable.

  25. NathanDunfield commented on Aug 9, 2023

    @NathanDunfield
    NathanDunfield
    on Aug 9, 2023

    I would appreciate if anyone could report testing these wheels which you can do like:

    All tests passed on macOS 13 (Ventura) with Python 3.9, 3.10, 3.11 (downloaded from Python.org) on an iMacPro 2017. All tests passed on Windows 10 Pro with Python 3.9 and 3.11 (downloaded from Python.org) on a Surface Pro 2017 (i5-7300U). It might be worth adding the test files to the wheels inside a submodule so that one could do something like:

    python -m pip install python-flint
python -m flint.test

    to run all the tests without cloning the source repo.

  26. oscarbenjamin commented on Aug 9, 2023

    @oscarbenjamin
    oscarbenjamin
    on Aug 9, 2023
    Collaborator

    Thanks @NathanDunfield!

    on an iMacPro 2017

    Am I right in presuming that would be x86-64 (i.e. not the M1 CPU)?

    It might be worth adding the test files to the wheels inside a submodule

    Yes, that is a good idea. I've included them in the sdist but they are not in the wheels. The wheels are already quite large because of the bundled binaries so the test files would not noticeably increase the size.

  27. NathanDunfield commented on Aug 9, 2023

    @NathanDunfield
    NathanDunfield
    on Aug 9, 2023

    Am I right in presuming that would be x86-64 (i.e. not the M1 CPU)?

    Yes, it's an Intel processor (Xeon W-2150B), not Apple Silicon.

  28. oscarbenjamin commented on Aug 9, 2023

    @oscarbenjamin
    oscarbenjamin
    on Aug 9, 2023
    Collaborator

    I've just pushed a 0.4.1 release to PyPI.

    The issue with Linux wheels remains unresolved: I have no more time to work on this and it is unclear to me what to do investigate this further in any case. Unless anyone has any bright ideas I just need to defer this problem.

    Instead of fixing the Linux wheels issue I have pushed updated instructions for building from sdist on Linux:
    https://github.com/fredrik-johansson/python-flint#installation

    On the bright side the Windows and OSX wheels seem to work very well. Thanks to @NathanDunfield for testing on Windows and OSX x86-64 for a range of Python versions. I have now also tested on OSX arm64 and that seems to work great as well.

    Maybe this issue should be closed and replaced with a Linux-specific wheel issue.

  29. oscarbenjamin commented on Aug 9, 2023

    @oscarbenjamin
    oscarbenjamin
    on Aug 9, 2023
    Collaborator

    Oh, also as of the 0.4.1 release the tests can be run against the installed distribution as suggested by @NathanDunfield above:

    pip install python-flint
python -m flint.test

  30. oscarbenjamin commented on Aug 13, 2023

    @oscarbenjamin
    oscarbenjamin
    on Aug 13, 2023
    Collaborator

    I've opened #51 to track the problem with the Linux wheels. In the mean time there are wheels for Windows and OSX on PyPI and I have tested them on both platforms and both architectures for OSX.

    I'm going to close this issue, so please use #51 to followup on the problem with the Linux wheels.

    Thanks everyone above for your help!

  31. oscarbenjamin
    closed this as completedon Aug 13, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions