Skip to content

Error occurs when only using partition columns in query #1999

Closed

Description

@pjmore

Describe the bug
When using a partitioned datasource selecting only the partitioned columns causes errors.

To Reproduce
Steps to reproduce the behavior:
Create sample parquet/csv file content doesn't matter. Copy file to two separate paths using path partitioning scheme. E.g.
"year=2021/month=09/day=09/file.parquet",
"year=2021/month=10/day=09/file.parquet",

Register listing table and execute query
select distinct year,month, day from t

Expected behavior
Should return record batch with:

+------+-------+-----+
| year | month | day||
+------+-------+-----+
| 2021 | 09    | 09  |
| 2021 | 10    | 09  |
| 2021 | 10    | 28  |
+------+-------+-----+ 

Additional context
This was verified to be a problem with parquet and csv formats. Unsure about avro or json.

Activity

  1. Jefffrey commented on Nov 14, 2022

    @Jefffrey
    Contributor

    Issue seems resolves as of 96512ac as unable to reproduce:

    Created file structure as shown:

    jeffrey:~/tmpdata$ pwd
    /home/jeffrey/tmpdata
    jeffrey:~/tmpdata$ tree *test
    csvtest
    └── year=2021
        ├── month=09
        │   └── day=09
        │       └── time.csv
        └── month=10
            └── day=09
                └── time.csv
    parttest
    └── year=2021
        ├── month=09
        │   └── day=09
        │       └── part-0.parquet
        └── month=10
            └── day=09
                └── part-0.parquet
    

    Then selecting their partitioned columns:

    DataFusion CLI v14.0.0
    ❯ create external table parttest stored as parquet partitioned by (year, month, day) location '/home/jeffrey/tmpdata/parttest';
    0 rows in set. Query took 0.003 seconds.
    ❯ select distinct year, month, day from parttest;
    +------+-------+-----+
    | year | month | day |
    +------+-------+-----+
    | 2021 | 10    | 09  |
    | 2021 | 09    | 09  |
    +------+-------+-----+
    2 rows in set. Query took 0.105 seconds.
    ❯ create external table csvtest stored as csv partitioned by (year, month, day) location '/home/jeffrey/tmpdata/csvtest';
    0 rows in set. Query took 0.018 seconds.
    ❯ select distinct year, month, day from csvtest;
    +------+-------+-----+
    | year | month | day |
    +------+-------+-----+
    | 2021 | 09    | 09  |
    | 2021 | 10    | 09  |
    +------+-------+-----+
    2 rows in set. Query took 0.008 seconds.

    Can see selects fine with no errors.

  2. pjmore commented on Nov 15, 2022

    @pjmore
    ContributorAuthor

    @Jefffrey Yeah you're right this was fixed by #2000 and I should've closed it then. Thanks for the heads up on this one.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions