Skip to content

Upgrade to sqlparser 0.55.0 #15071

Closed

Description

@alamb

Is your feature request related to a problem or challenge?

The new version of sqlparser has been released:

https://crates.io/crates/sqlparser/0.55.0

Datafusion is still on sqlparser 0.54.0:

sqlparser = { version = "0.54.0", features = ["visitor"] }

Describe the solution you'd like

I would like to upgrade datafusion to 0.55.0 of sqlparser

This one may be tricky due to the change in ObjectName

Describe alternatives you've considered

Here is an example showing how it was done last time

Additional context

No response

Activity

  1. jonahgao commented on Mar 8, 2025

    @jonahgao
    Member

    take

  2. PokIsemaine commented on Mar 8, 2025

    @PokIsemaine
    Contributor

    @alamb @jonahgao
    I have already made some progress on this part. If you don't mind, I would like to finish it. I want to use this update to become more familiar with DataFusion's code, and I would greatly appreciate any guidance you can offer.

    Here are some sqlparser Pull Requests that involve modifications to DataFusion, which you can use as references:

    Changelog: https://github.com/apache/datafusion-sqlparser-rs/blob/main/changelog/0.55.0.md

    OrderBy: apache/datafusion-sqlparser-rs#1724
    SQLExpr::Case: apache/datafusion-sqlparser-rs#1733
    ColumnDef Collation: apache/datafusion-sqlparser-rs#1731
    DataType: apache/datafusion-sqlparser-rs#1739
    EXECUTE: apache/datafusion-sqlparser-rs#1717
    CreateTable: apache/datafusion-sqlparser-rs#1664
    QualifiedWildcard apache/datafusion-sqlparser-rs#1680
    TypedString: apache/datafusion-sqlparser-rs#1739
    SelectFlavor: apache/datafusion-sqlparser-rs#1713
    StartTransaction: apache/datafusion-sqlparser-rs#1718

  3. jonahgao commented on Mar 8, 2025

    @jonahgao
    Member

    Thank you @PokIsemaine for your work. Assigned this issue to you ❤️

  4. PokIsemaine commented on Mar 9, 2025

    @PokIsemaine
    Contributor

    I want to discuss JoinType:

    JoinOperator

    Do we need to distinguish between Inner/Join, Left/LeftOuter, and Right/RightOuter in JoinType, like this?

    pub enum JoinType {
        Inner,
        Join,
        Left,
        LeftOuter,
        Right,
        RightOuter,
        // other...
    }

    Since sqlparser now differentiates between INNER JOIN and JOIN, the parsed result no longer matches the original test cases. For example:

    TestStatementWithDialect {
        sql: "
            SELECT
              j1_string,
              j2_string
            FROM
              (
                SELECT
                  distinct j1_id,
                  j1_string,
                  j2_string
                from
                  j1
                  INNER join j2 ON j1.j1_id = j2.j2_id
                order by
                  j1.j1_id desc
                limit
                  10
              ) abc
            ORDER BY
              abc.j2_string",
        expected: r#"SELECT abc.j1_string, abc.j2_string FROM (SELECT DISTINCT j1.j1_id, j1.j1_string, j2.j2_string FROM j1 JOIN j2 ON (j1.j1_id = j2.j2_id) ORDER BY j1.j1_id DESC NULLS FIRST LIMIT 10) AS abc ORDER BY abc.j2_string ASC NULLS LAST"#,
        parser_dialect: Box::new(GenericDialect {}),
        unparser_dialect: Box::new(UnparserDefaultDialect {}),
    },

    Should the JOIN in expected be changed to INNER JOIN?

    If we make this distinction, it will affect printing and the unparser, which will indirectly impact many test cases. Do you have any good suggestions?

  5. jonahgao commented on Mar 9, 2025

    @jonahgao
    Member

    Do we need to distinguish between Inner/Join, Left/LeftOuter, and Right/RightOuter in JoinType, like this?

    I think we don't need to distinguish them as they are equivalent. Introducing new join type variants would result in more significant changes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

enhancementNew feature or requesthelp wantedExtra attention is needed

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions