[Python] usability improvements for a "minimal" pyarrow

@h-vetinari

Describe the enhancement requested

Providing slimmer variants of arrow has been a topic for quite a while, but got more urgent with pandas plan to depend on pyarrow, which would bring quite a substantial installation size increase, due to the way pyarrow gets packaged (this is true even more so in conda-forge, where we package a "maximal" version of arrow -- since it's so hard to build from source -- that generally contains more in terms of transitive dependencies than the wheels).

Through work on the feedstock, the conda-forge side of arrow is now ready to split up libarrow 14.0 into several pieces (currently libarrow-{acero,dataset,flight,flight-sql,gandiva,substrait} + libparquet), but we're still having pyarrow depend on the entirety of libarrow, not least because the python bindings link to everything but libarrow-flight-sql directly:

INFO (pyarrow,lib/python3.12/site-packages/pyarrow/_acero.cpython-312-x86_64-linux-gnu.so):           Needed DSO lib/libarrow_acero.so.1400     found in libarrow-acero-14.0.0-h59595ed_0_cpu
INFO (pyarrow,lib/python3.12/site-packages/pyarrow/_dataset.cpython-312-x86_64-linux-gnu.so):         Needed DSO lib/libarrow_dataset.so.1400   found in libarrow-dataset-14.0.0-h59595ed_0_cpu
INFO (pyarrow,lib/python3.12/site-packages/pyarrow/_dataset_orc.cpython-312-x86_64-linux-gnu.so):     Needed DSO lib/libarrow_dataset.so.1400   found in libarrow-dataset-14.0.0-h59595ed_0_cpu
INFO (pyarrow,lib/python3.12/site-packages/pyarrow/_dataset_parquet.cpython-312-x86_64-linux-gnu.so): Needed DSO lib/libarrow_dataset.so.1400   found in libarrow-dataset-14.0.0-h59595ed_0_cpu
INFO (pyarrow,lib/python3.12/site-packages/pyarrow/_flight.cpython-312-x86_64-linux-gnu.so):          Needed DSO lib/libarrow_flight.so.1400    found in libarrow-flight-14.0.0-h35bba4a_0_cpu
INFO (pyarrow,lib/python3.12/site-packages/pyarrow/_substrait.cpython-312-x86_64-linux-gnu.so):       Needed DSO lib/libarrow_substrait.so.1400 found in libarrow-substrait-14.0.0-hab2db56_0_cpu
INFO (pyarrow,lib/python3.12/site-packages/pyarrow/gandiva.cpython-312-x86_64-linux-gnu.so):          Needed DSO lib/libgandiva.so.1400         found in libarrow-gandiva-14.0.0-hacb8726_0_cpu
INFO (pyarrow,lib/python3.12/site-packages/pyarrow/libarrow_python_flight.so):                        Needed DSO lib/libarrow_flight.so.1400    found in libarrow-flight-14.0.0-h35bba4a_0_cpu
INFO (pyarrow,lib/python3.12/site-packages/pyarrow/libarrow_python.so):                               Needed DSO lib/libparquet.so.1400         found in libparquet-14.0.0-h352af49_0_cpu

While it would be theoretically possible to also build various pyarrow-* variants, that's quite unappealing IMO from a packaging perspective, and it would be nicer if pyarrow just depended on the (core) libarrow, but provided helpful error messages where any missing libarrow-* libraries actually get used. In such a scenario (c.f. discussion in conda-forge/arrow-cpp-feedstock#1035),

@h-vetinari: [...] I think in terms of user-friendliness, we need to provide a better message than:
ImportError: libarrow_dataset.so.1400: cannot open shared object file: No such file or directory
I think it would make sense for arrow to define which libraries can be removed while still expecting core functionality to work, which dependencies each remaining artefact has, and provide some error message upon not finding the respective library (which we can then patch on the feedstock to add messages like "install this additional package to get it"; alternatively, arrow could of course integrate that into the messages directly in this repo, à la "if you're using arrow from conda-forge, just install libarrow-dataset").

Such an approach would presumably also make it easier for the wheel side of things (i.e. not having N pyarrow-* variants), though of course, providing the equivalent of the libarrow-* outputs from conda-forge through wheels would be quite a headache. It's possible that the best solution for wheels looks different (or ends up being sliced differently, like e.g. having two wheels pyarrow and pyarrow-minimal, or pyarrow and pyarrow[full]).

Note also that (from conda-forge/arrow-cpp-feedstock#1175):

@h-vetinari: [...] the new libarrow core library still depends on some of the most heavy-weight libraries at runtime (e.g. libgoogle-cloud, which is around ~30MB). I think it would make sense to separate out the pieces that depend on cloud-provider bindings into a separate output. Not sure how much work that is...

This is now being tracked in ~~#38309~~ #49399

Component(s)

Packaging, Python

it would be nicer if pyarrow just depended on the (core) libarrow, but provided helpful error messages where any missing libarrow-* libraries actually get used

That should already have been tackled (although there might be some libs that were missed): #36553 / #36554.
Of course, that's still a generic message, while in theory for conda-forge we could give a more useful error message pointing to the additional package they have to install. But I am not sure that's something we want to bake into pyarrow.

Ah, nice! I guess we can patch that, though it would be nice to have a template that we only need to patch once¹, rather than have to chase every import that's potentially affected by a missing DSO.

assuming pyarrow would not be willing to detect e.g. that the interpreter is from conda, and adapting the message ↩

@h-vetinari

Hi @h-vetinari!

Do you know if the package split introduced for conda-forge is coming to PyPi?

BTW, should this issue still be opened?

Do you know if the package split introduced for conda-forge is coming to PyPi?

There's some discussions in order to split the wheels and do something "similar" but apart from some exploration there hasn't been anything concrete done yet. I plan to work on it but I have to schedule it along with other priorities

Thanks for all your hard work, guys! 👍

Hi everyone!

Still no news regarding PyPi package split? 🙏

Still no news regarding PyPi package split?

Unfortunately not. This is a big chunk of work and we need someone to be able to work on it.

I haven't noticed this before but the pyarrow wheel size is getting out of hand, I guess it's time for me to see how we can permanently move away from pyarrow for delta-rs.

In lieu of adding more pyarrow- variants to pypi, another option would be to leverage the work done for adding meson-python support. If that goes live, we could document how users can set applicable options when building from an sdist.

For example, if a user wanted to build minified libraries without gandvia or substrait, they could do something like:

pip install --no-binary pyarrow \
    -Csetup-args=-Dbuildtype=minsize \
    -Csetup-args=-Dgandiva=disabled \
    -Csetup-args=-Dsubstrait=disabled

There's of course a good deal more work to go with the implementation in #45854, but it may be a reasonable option to solve this

[...] adding more pyarrow- variants to pypi [or] document how users can set applicable options when building from an sdist.

Those two things are almost entirely orthogonal. The overwhelming majority of user will not compile from an sdist, even if some aspects could be improved (e.g. -march=native). I'm willing to bet that the ratio of wheel-users to those willing to compile from source is 100:1 or more.

Fair point - I think its in any case a question of what percentage of users care about a smaller wheel in the first place. In the general case I would agree with the ratio, but in the subset of users that care about the wheel size that same ratio might not hold.

Providing that sdist use case is just another lever to pull for developers, in theory without too much incremental effort, especially when compared against distributing and versioning multiple variants of the library

As mentioned in the original issue, when Pandas requires Arrow (seems to be more imminent now from the now-postponed vote), Arrow is going to be installed just from installing Pandas from pip, and the large default wheel size for Arrow will result in that size downloaded and installed for every pip installation of Pandas. My personal experience is that compiling Arrow correctly is still incredibly hard. In addition, space-constrained environments wouldn't have the space (or resources) to compile in the first place. Therefore, it seems like the option suggested above wouldn't really solve the issue for the "average" developer with space constraints or for reducing download volume from PyPI.

I am working on moving s3 into its own librarry as seen on the PR above and I wasn't sure where to land this thoughts but I am going to post here some experiments I am currently doing:
I've manually removed libarrow_s3.so from a local pyarrow build:

$ python
Python 3.13.12 (main, Feb  4 2026, 15:06:39) [GCC 15.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pyarrow._s3fs
Traceback (most recent call last):
  File "<python-input-0>", line 1, in <module>
    import pyarrow._s3fs
ImportError: libarrow_s3.so.2500: cannot open shared object file: No such file or directory
>>>

And have created a minimal wheel with only libarrow_s3.so.2500 and a minimal __init__.py that loads the DLL:

$ python
Python 3.13.12 (main, Feb  4 2026, 15:06:39) [GCC 15.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pyarrow_s3
>>> import pyarrow._s3fs
>>> import pyarrow.fs as fs
>>> fs.FileSystem.from_uri("s3://bucket/key")
(<pyarrow._s3fs.S3FileSystem object at 0x7fa3b05a3330>, 'bucket/key')

This UX should change, users shouldn't have to explicitly import pyarrow_s3. When import pyarrow.fs tries importing s3 it should automatically register by importing pyarrow_s3.

The current __init__.py is hardcoded but can be updated to make it portable and cross-OS:

import os
import ctypes
# ensure libarrow.so.2500 is loaded first (libarrow_s3 depends on it)
import pyarrow

ctypes.CDLL(
    os.path.join(os.path.dirname(__file__), "libarrow_s3.so.2500"),
    mode=ctypes.RTLD_GLOBAL,
)

About the versioning, we could make the metadata pin handle the required pyarrow version and tie them together. All provided wheels must match the versions. Otherwise it gets messy for users when we expand this functionality to have other wheels like pyarrow_s3, pyarrow_flight, pyarrow_gcs. We could expose pyarrow extras with optional-dependencies, to have a better UX and have a tighter version matching, like:

[project.optional-dependencies]
s3     = ["pyarrow-s3==25.0.0"]
gcs    = ["pyarrow-gcs==25.0.0"]
flight = ["pyarrow-flight==25.0.0"]

The optional wheels should also match and pin the version pyarrow==25.0.0.

Basically we are making those extra wheels just a wrapper for a single .so.

This is against a local build on this branch, not yet two auditwheel/delvewheel-repaired wheels in a clean venv, that's the next thing I plan to validate.

To be fair this is trying to extend the current proven conda model into separated wheels.

Caveats:

Upgrade path for users. The exact pins == make it an all-or-nothing for users. Not too different from what we have today as we ship a single wheel but worth noting.
What happens when other external dependencies pin versions of arrow? The exact pins might complicate things for users and library maintainers.
Added complexity of managing all the extra wheels.

Navigation Menu

Repository navigation

[Python] usability improvements for a "minimal" pyarrow #38536

Describe the enhancement requested

Component(s)

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[Python] usability improvements for a "minimal" pyarrow #38536

Description

Describe the enhancement requested

Component(s)

Activity

jorisvandenbossche commented on Nov 1, 2023

h-vetinari commented on Nov 2, 2023

Footnotes

ZupoLlask commented on Sep 10, 2024

raulcd commented on Sep 10, 2024

ZupoLlask commented on Sep 10, 2024

ZupoLlask commented on Jan 18, 2025

raulcd commented on Jan 20, 2025

ion-elgreco commented on May 22, 2025

WillAyd commented on May 22, 2025

h-vetinari commented on May 22, 2025

WillAyd commented on May 22, 2025

firai commented on Jun 14, 2025

raulcd commented on Jun 23, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions