This repository was archived by the owner on Mar 6, 2026. It is now read-only.
Repository navigation
This repository was archived by the owner on Mar 6, 2026. It is now read-only.
RowIterator to_dataframe requires pyarrow >= 1.0.0 to work #249
Description
Currently the google-cloud-bigquery library requires pyarrow > 0.16.0, however the method RowIterator.to_dataframe adds the kwarg "timestamp_as_object", which is only supported in pyarrow >= 1.0.0. If install pyarrow >= 1.0.0, everything works as expected, however we are using other libraries which require pyarrow < 1.0.0.
So the requirements should either be updated to require pyarrow >= 1.0.0, or backported to support versions less than 1.
Environment details
- OS type and version: Any
- Python version: 3.6.9
- pip version: 20.2.2
google-cloud-bigqueryversion: 1.27.2
Steps to reproduce
- Use pyarrow < 1.0.0
- Run RowIterator to_dataframe
Stack trace
# result = future.result()
File "<path>/python3.6/concurrent/futures/_base.py", line 425, in result
return self.__get_result()
File "<path>/python3.6/concurrent/futures/_base.py", line 384, in __get_result
raise self._exception
File "<path>/python3.6/concurrent/futures/thread.py", line 56, in run
result = self.fn(*self.args, **self.kwargs)
File "<path>", line 133, in run_query
bqstorage_client=client_storage
File "<path>/python3.6/site-packages/google/cloud/bigquery/table.py", line 1757, in to_dataframe
df = record_batch.to_pandas(date_as_object=date_as_object, **extra_kwargs)
File "pyarrow/array.pxi", line 503, in pyarrow.lib._PandasConvertible.to_pandas
TypeError: to_pandas() got an unexpected keyword argument 'timestamp_as_object'
Activity
HemangChothani commentedon Aug 28, 2020
@BradLewis Minimum required version is specified in extras_require in setup.py file
python-bigquery/setup.py
Lines 50 to 53 in a587de4
BradLewis commentedon Aug 28, 2020
@HemangChothani That is only the case when using the pyarrow extra.
For more context, we are using the google-cloud-bigquery library with the google-cloud-bigquery-storage library with pandas and fastavro, similar to what is documented here: https://cloud.google.com/bigquery/docs/bigquery-storage-python-pandas.
In the extra section for bqstorage the required pyarrow is >= 0.16.0.
For example, with the below code and running pyarrow < 1.0.0,
this will give the same error
This is the PR that introduced this #209.
As it mentions it requires pyarrow >= 1.0.0, however if you don't install the pyarrow extra with the intention of not using it, and have another dependency that requires pyarrow < 1.0.0, it'll run through this path and give you the error.
HemangChothani commentedon Aug 28, 2020
@BradLewis Thanks for the explanation got the point, i have raised PR which bump the minimum requirement of
pyarrowversion forbigquery-storageas well.