This repository was archived by the owner on Mar 6, 2026. It is now read-only.
Repository navigation
This repository was archived by the owner on Mar 6, 2026. It is now read-only.
pull google-cloud-bigquery-storage and pyarrow into required dependencies #757
Description
https://github.com/googleapis/python-bigquery-storage/pull/224/files#r667241381
We'll still want to keep bqstorage around in "extras" so that pip install google-cloud-bigquery[bqstorage] still succeeds, but the [bqstorage part will be a no-op.
Activity
tswast commentedon Jul 13, 2021
This will likely result in a lot of code cleanup work to remove dead code paths that assume the BQ Storage client library is not available.
plamut commentedon Jul 14, 2021
I'll be happy to clean that stuff, I never really liked it. :)
plamut commentedon Jul 14, 2021
@tswast Do you know how many users use BigQuery without extras AND would not be able to install them if we make them required? In other words, would making some of the extras required require a major version bump?
tswast commentedon Jul 14, 2021
I'm not too concerned about
google-cloud-bigquery-storage, as we already directly depend ongrpciopython-bigquery/setup.py
Line 32 in 5deef6f
proto-pluspython-bigquery/setup.py
Line 34 in 5deef6f
pyarrowis the more concerning dependency. There were problems in the past where Spark had pinned an old version of PyArrow due to wire format changes (https://spark.apache.org/docs/latest/api/python/user_guide/arrow_pandas.html#compatibility-setting-for-pyarrow-0-15-0-and-spark-2-3-x-2-4-x) It appears they've been able to resolve this with an environment variable, but that'll break BigQuery as it uses the latest wire format version.tswast commentedon Jul 14, 2021
Looks like Spark >= 3.0 is using the latest wire format: dotnet/spark#370 (comment)
plamut commentedon Jul 15, 2021
So the only major concern is users that use Spark < 3.0 alongside BigQuery without extras? And even they can either upgrade Spark, if possible, or cap the BigQuery version as a workaround?
tswast commentedon Jul 15, 2021
That's the only concern that comes to mind. And yes, they should have workarounds as they can pin older versions that don't need arrow / google-cloud-bigquery-storage.
I suppose the other concern is if someone manually builds their own C++ Arrow library and tries to use it in the same environment as
pyarrowthat wasn't built with exactly the same compiler settings, but I think that's more a problem withpyarrow+pipthan it is something we need to worry about.tswast commentedon Oct 6, 2021
Closed by #776