Repository navigation
PyArrow version requirement #55
Description
Hello! Is the version requirement on the pyarrow package strict?
pyarrow = "^9.0.0"
We would like to use databricks-sql-python in a project that depends on a lower version of pyarrow (8.0.0). We ran into conflicting version issues when using package management tools such as poetry. In our tests it seems databricks-sql-python can work with older versions of pyarrow such as 8.0.0. It would be great if databricks-sql-python can specify a wider range of pyarrow versions for better compatibility.
Activity
Tom-Newton commentedon Nov 10, 2022
I have a similar issue but personally I'm interested in using pyarrow
10.0.0. It would be great if this was allowed.susodapop commentedon Nov 30, 2022
@Tom-Newton pyarrow
10.0.1is now supported as ourv2.2.1release.@YS-L I'll look into reducing our pyarrow specificaiton. I think we landed on version 9 because it was the latest at the time we wrote the connector.
Tom-Newton commentedon Dec 8, 2022
Thanks @susodapop 🙂 . With 2.2.1 my pyarrow problem is solved but now numpy is an issue. We are on numpy
1.22.4with python 3.8. Is there are reason that the numpy requirement needs to be so strict?We are currently using
databricks-sql-connector==2.0.4withnumpy==1.22.4and we haven't noticed any issues.susodapop commentedon Dec 15, 2022
I want to make this work for you but I also can't reproduce your installation problem. For your info: note that our specific requirements were dictated by Poetry so we haven't deliberately excluded anyone and thus far our tests run automatically on Python 3.7-3.11 without issues.
You're welcome to open a pull request that modifies
pyproject.tomlandpoetry.lockto meet your needs. As long as it passes our build checks I'm happy to merge it. But this isn't something we're prepared to do on our own because nothing is broken for us.Tom-Newton commentedon Dec 31, 2022
I created a PR #74. It needs a review and for someone to approve running the build checks.
Tom-Newton commentedon Feb 13, 2023
@susodapop please could you approve running the github workflows
susodapop commentedon Feb 13, 2023
Done!