Repository navigation
[CI][Python] Nightly test for PySpark 3.2.0 fail with AttributeError on numpy.bool #33697
Description
Describe the bug, including details regarding any error messages, version, and platform.
Nightly integration tests with PySpark 3.2.0 are failing with the following error:
test-conda-python-3.8-spark-v3.2.0
ERROR: test_with_key_complex (pyspark.sql.tests.test_pandas_cogrouped_map.CogroupedMapInPandasTests)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/spark/python/pyspark/sql/tests/test_pandas_cogrouped_map.py", line 160, in test_with_key_complex
result = self.data1 \
File "/spark/python/pyspark/sql/pandas/conversion.py", line 168, in toPandas
pandas_type = PandasConversionMixin._to_corrected_pandas_type(field.dataType)
File "/spark/python/pyspark/sql/pandas/conversion.py", line 238, in _to_corrected_pandas_type
return np.bool
File "/opt/conda/envs/arrow/lib/python3.8/site-packages/numpy/__init__.py", line 284, in __getattr__
raise AttributeError("module {!r} has no attribute "
AttributeError: module 'numpy' has no attribute 'bool'
Component(s)
Continuous Integration, Python
Activity
raulcd commentedon Jan 16, 2023
cc @AlenkaF @jorisvandenbossche
AlenkaF commentedon Jan 16, 2023
Numpy deprecated the use of
np.boolin version 1.24:the CI build for Nightly integration tests with PySpark 3.2.0 is using this (1.24) version of NumPy, but the fix in Apache Spark has already been merged: apache/spark#37817.
Am continuing to look into it ...
AlenkaF commentedon Jan 16, 2023
... ah yes, the version of PySpark is 3.2.0 in the CI build (could just look at the title, not the CI setup :D).
I guess we should change numpy version till the new release of PySpark is out with the fix?
jorisvandenbossche commentedon Jan 16, 2023
That sounds correct
AlenkaF commentedon Jan 16, 2023
Great, will make a PR.
GH-33697: [CI][Python] Nightly test for PySpark 3.2.0 fail with Attri…