Skip to content

pyarrow latest parquet map column type isn't supported #2262

Closed

Description

@den-rifiniti

Hello
When I trying read parquet file with column of type map, pyarrow.lib.ArrowNotImplementedError: lists with structs are not supported. exception is throws.

Seems like this was fixed here #1530, and only release is required?

Activity

  1. xhochy commented on Jul 14, 2018

    @xhochy
    Member

    No, this would also require apache/parquet-cpp#462

  2. wesm commented on Jul 19, 2018

    @wesm
    Member

    Assistance with this would be much appreciated. Unfortunately we haven't been able to get this done in time for 0.10, so it will have to be later this year

  3. damache commented on Dec 7, 2018

    @damache

    was this fixed? I installed the following

    !conda install -c conda-forge pyarrow
    
    Fetching package metadata .............
    Solving package specifications: .
    
    Package plan for installation in environment /opt/conda/envs/DSX-Python35:
    
    The following NEW packages will be INSTALLED:
    
        arrow-cpp:   0.10.0-py35h70250a7_0 conda-forge
        boost-cpp:   1.67.0-h3a22d5f_0     conda-forge
        parquet-cpp: 1.5.0.pre-h83d4a3d_0  conda-forge
        pyarrow:     0.10.0-py35hfc679d8_0 conda-forge
    
    boost-cpp-1.67 100% |################################| Time: 0:00:00  90.87 MB/s
    arrow-cpp-0.10 100% |################################| Time: 0:00:00  47.85 MB/s
    parquet-cpp-1. 100% |################################| Time: 0:00:00  68.96 MB/s
    pyarrow-0.10.0 100% |################################| Time: 0:00:00  62.08 MB/s
    

    then tried this code

    import io
    import pandas as pd
    import pyarrow.parquet as pq
    
    # Read the parquet file
    buffer = io.BytesIO()
    object = cos.Object('*********','*****************')
    object.download_fileobj(buffer)
    table = pq.read_table(buffer)
    df = table.to_pandas()
    print(df.head())
    

    but I get this error

    ArrowNotImplementedError                  Traceback (most recent call last)
    <ipython-input-11-a1e8748910ba> in <module>()
          7 object = cos.Object('********','*************')
          8 object.download_fileobj(buffer)
    ----> 9 table = pq.read_table(buffer)
         10 df = table.to_pandas()
         11 print(df.head())
    
    /opt/conda/envs/DSX-Python35/lib/python3.5/site-packages/pyarrow/parquet.py in read_table(source, columns, nthreads, metadata, use_pandas_metadata)
       1048     pf = ParquetFile(source, metadata=metadata)
       1049     return pf.read(columns=columns, nthreads=nthreads,
    -> 1050                    use_pandas_metadata=use_pandas_metadata)
       1051 
       1052 
    
    /opt/conda/envs/DSX-Python35/lib/python3.5/site-packages/pyarrow/parquet.py in read(self, columns, nthreads, use_pandas_metadata)
        150             columns, use_pandas_metadata=use_pandas_metadata)
        151         return self.reader.read_all(column_indices=column_indices,
    --> 152                                     nthreads=nthreads)
        153 
        154     def scan_contents(self, columns=None, batch_size=65536):
    
    /opt/conda/envs/DSX-Python35/lib/python3.5/site-packages/pyarrow/_parquet.pyx in pyarrow._parquet.ParquetReader.read_all()
    
    /opt/conda/envs/DSX-Python35/lib/python3.5/site-packages/pyarrow/error.pxi in pyarrow.lib.check_status()
    
    ArrowNotImplementedError: lists with structs are not supported.
    
  4. wesm commented on Dec 8, 2018

    @wesm
    Member

    No It has not yet been implemented

  5. wesm commented on Dec 9, 2018

    @wesm
    Member

    @damache would anyone from IBM like to get involved with Parquet development? We could really use the help.

  6. sujayramaiah commented on Jan 24, 2020

    @sujayramaiah

    Most of the data files in our data lake has map columns. Not being able to read parquet files with map columns using pyarrow creates dependency on Spark. Is there a plan to support map columns?

  7. wesm commented on Jan 28, 2020

    @wesm
    Member

    Yes, but someone has to do the implementation work. See ARROW-1644 and related issues

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions