Repository navigation
provide binary wheels for Linux compatible with lxml #316
Description
I know this error has been added intentionally to catch other breakages, but is there a way to provide binary wheels that will work with current lxml? So that pip install lxml xmlsec will bring working combo.
Presently, it doesn't:
(.venv) $ pip install lxml xmlsec
Collecting lxml
Using cached lxml-5.2.1-cp311-cp311-manylinux_2_28_x86_64.whl.metadata (3.4 kB)
Collecting xmlsec
Downloading xmlsec-1.3.14.tar.gz (68 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 68.9/68.9 kB 549.7 kB/s eta 0:00:00
Installing build dependencies ... done
Getting requirements to build wheel ... done
Preparing metadata (pyproject.toml) ... done
Using cached lxml-5.2.1-cp311-cp311-manylinux_2_28_x86_64.whl (5.0 MB)
Building wheels for collected packages: xmlsec
Building wheel for xmlsec (pyproject.toml) ... done
Created wheel for xmlsec: filename=xmlsec-1.3.14-cp311-cp311-linux_x86_64.whl size=160772 sha256=79e2ffc317a1754761d069e1e76adf033ac7be8e4189c816411284fe4eee6d41
Stored in directory: /home/nijel/.cache/pip/wheels/98/60/8b/bc03288ec33436feed643c9a680f5124f697a735c7cb5f0d23
Successfully built xmlsec
Installing collected packages: lxml, xmlsec
Successfully installed lxml-5.2.1 xmlsec-1.3.14
(.venv) $ python -c 'import xmlsec'
Traceback (most recent call last):
File "<string>", line 1, in <module>
xmlsec.InternalError: (-1, 'lxml & xmlsec libxml2 library version mismatch')The problem is that lxml wheels are built with internal libxml while xmlsec is using system one.
Rebuilding lxml against system libxml does address the issue, but working combo out of the box would be better.
Activity
ci: do not use wheels for lxml
mxamin commentedon Apr 18, 2024
@nijel The workaround solution would be to force lxml not to use the pre-compiled wheel:
pip install lxml xmlsec --no-binary=lxmlWe already have pre-compile wheels for linux and macos (see release note: https://github.com/xmlsec/python-xmlsec/releases/tag/1.3.14) but I didn't upload them in pypi because of this version checking that was added in the latest release.
I'm gonna note this issue for the next release, meanwhile can you provide the following information:
nijel commentedon Apr 18, 2024
I know the workaround, the problem is that there is no way to tell this inside dependencies. Instead, you need to tell users to not use
pip install <package>, butpip install --no-binary=lxml <package>, what half of them won't do and complain that things do not work then :-).I'm on Debian 12 (Python 3.11, libxml2 2.9.14, xmlsec1 1.2.37), it happens on GitHub CI environment with Ubuntu 22.04 (any Python version installed by actions/setup-python, libxml 2.9.13, xmlsec1 1.2.33).
PS: I'm happy user of https://github.com/pypa/cibuildwheel and https://github.com/pypa/gh-action-pypi-publish to automate all the wheel builds and publishing.
ci: do not use wheels for lxml
mxamin commentedon Apr 18, 2024
@nijel You're right, I put it in the todo list for the next release. Hope we can solve it better :)
jonathangreen commentedon Apr 18, 2024
@mxamin i like the idea of providing Linux wheels for the next release as well. We are already making the wheels with cibuildwheel, so it's just a matter of distributing them.
I can work on a PR to automate attaching the wheels to the release in GitHub and uploading them to PyPI if you would like.
mxamin commentedon Apr 18, 2024
I could have do it manually (simply downloading the artificats and uploading them to pypi) but I decided not to do it because of libxml2 version check. Assume that lxml wheel is build with version 2.9.x and python-xmlsec with 2.11.x, we would end up with the mismatch error which is not good. So the real problem here is to fix the lxml/python-xmlsec compatibility issue for real :)
jonathangreen commentedon Apr 18, 2024
Making the integration with lxml safer, like you are suggesting, would be a win for sure.
Even if we do that though, I think there is still a strong argument for providing wheels for Linux. It makes things much easier for end users of the package, especially when installing inside a container where you don't necessarily have all the build tooling available.
mxamin commentedon Apr 18, 2024
Definitely it would help. I wonder if the created wheels would help in reported cases. I probably check that later.
nijel commentedon Apr 18, 2024
I was thinking if it is possible to build a private xmlsec1 against libxml2 from lxml and ship it inside the wheel.
jonathangreen commentedon Apr 18, 2024
@nijel would you be able to manually install one of the Linux wheels attached the the release in GitHub https://github.com/xmlsec/python-xmlsec/releases/tag/1.3.14 and see if it solves your issue?
The wheels are statically linked against libxml2 and libxmlsec, so I believe it will sort your issue. Those are the wheels we are discussing uploading to pypi.
deps: install xmlsec wheel manually
ci: install xmlsec wheel manually
3 remaining items
jonathangreen commentedon Apr 19, 2024
It looks like maybe you were installing
lxmlwith no-binary, but installing the binary wheel forxmlsec. Now that we have wheels uploaded bothlxmlandxmlsecneed to be built the same way. Either installed from wheels or built with no-binary.Because
xmlsecpasses data intolxmlboth need to be built with the same version oflibxml2. This situation is unfortunate, and we'd like to fix it, so this requirement doesn't exist. But for now it does.This means that either they both need to be installed from wheels, which includes same version of
libxml2or they both need to be compiled using no-binary so they both get the system version oflibxml2. If you mix and match these approaches you will seeversion mismatcherror.nijel commentedon Apr 19, 2024
Thanks for explanation, it now works for me. Publishing binary wheels while I was testing this in CI made results for me quite unpredictable, but now with wheels-only setup, everything works just fine. Thanks for your effort to make this work!
💚 Attempt to fix the CI build
💚 Attempt to fix the CI build
no-binary-package lxml
no-binary-package lxml
no-binary-package lxml
no-binary-package lxml
no-binary-package lxml
no-binary-package lxml
no-binary-package lxml
no-binary-package lxml
Update Dockerfile for Python 3.13
Update Dockerfile for Python 3.13