Repository navigation
Find a safer way to integrate with lxml #283
Description
lxml's maintainer here. Users have run into numerous issues in the past due to incompatible libxml2 versions being used by lxml and xmlsec. This is because xmlsec is usually built against system installed libraries, whereas lxml bundles libxml2 and libxslt to ease the otherwise difficult installation. Thus, both can end up using different library versions that make different assumptions about the C tree structure and its handling, as well as missing configuration state (see #239, for example). lxml has about 100x as many downloads as xmlsec, so most users don't use both together, and thus, both have different needs regarding their installation and user base.
I would like to improve the situation by making lxml and xmlsec more independent from each other.
Most xmlsec features seem to rely on serialisation, probably C14N, which can be left to lxml. Exchanging serialised byte buffers should trivially avoid compatibility issues.
At which places does xmlsec need actual access to libxml2 trees? How can we get them to use a safe but efficient data exchange?
Activity
akx commentedon Feb 7, 2024
Looks like
python-xmlsecuses both the libxml APIs and the lxml C API 😅Unfortunately it looks like this repo is pretty unmaintained at the moment (but here's hoping someone steps up to do something about that!).
eljeffeg commentedon Feb 26, 2024
I think @jimjag is gonna fork it. 🙏
jonathangreen commentedon Mar 18, 2024
@scoder is there a way check what version of
libxml2lxmlis using via the C api?Maybe a way to approach this is to check the
libxml2version being used by bothlxmlandxmlsecand raise and exception if it is not compatible.It's not as elegant as your suggestion but it seems like an easier change to make and it would at least alert users of the issue instead of having to try to figure out what is causing a segfault.
yhlee-tw commentedon Mar 20, 2024
@jonathangreen does it have to be C API?
from https://lxml.de/1.3/FAQ.html#i-think-i-have-found-a-bug-in-lxml-what-should-i-do
That's how I am testing a few combinations on macOS today with your
xmlsec.get_libxml_version()in 3191662jonathangreen commentedon Mar 21, 2024
@yhlee-tw it would be much easier if we could do the comparison in python but unfortunately the integration with
lxmlis done via thelxmlc api. There really isn't any python code as part of this package.I took a look at the
lxmlc api and I don't see a way to get thelibxml2version being used, but I was hoping @scoder or someone more familiar with thelxmlc api might have a suggestion.mxamin commentedon Jul 9, 2025
We need to properly handle this issue, the workaround solution to match libxml2 versions raised a lot of confusion and makes it hard to properly release wheels.
mxamin commentedon Jul 10, 2025
I created a new issue as a possible solution. Please wrote your comments there: #356