Skip to content

Parse identifiers properly for TableReferences - #5183

Merged
alamb merged 3 commits into
apache:masterfrom
Jefffrey:table_reference_parsing
Feb 6, 2023
Merged

alamb merged 3 commits into
apache:masterfrom
Jefffrey:table_reference_parsing

Conversation

@Jefffrey

@Jefffrey Jefffrey commented Feb 5, 2023

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Closes #4532

Rationale for this change

Be able to parse identifiers properly for TableReference:

  • normalize unquoted identifiers
  • able to parse quoted identifiers with special characters (e.g. period)

What changes are included in this PR?

Change TableReference to hold Cows of its str's since need to allocate new String due to parsing (not to mention for normalization as well)

  • need to fix usages since can't derive Copy anymore

Use SQL parser to parse the possible multipart identifier, which handles quoted identifiers correctly

Are these changes tested?

New unit test added

Are there any user-facing changes?

TableReference public API is changed (Cow<str> instead of raw &str)

@github-actions github-actions Bot added core Core DataFusion crate sql SQL Planner substrait Changes to the substrait crate labels Feb 5, 2023
Comment on lines +212 to +225
1 => Self::Bare {
table: parts.remove(0).into(),
},
2 => Self::Partial {
schema: parts[0],
table: parts[1],
schema: parts.remove(0).into(),
table: parts.remove(0).into(),
},
3 => Self::Full {
catalog: parts[0],
schema: parts[1],
table: parts[2],
catalog: parts.remove(0).into(),
schema: parts.remove(0).into(),
table: parts.remove(0).into(),
},
_ => Self::Bare { table: s },
// TODO: should normalize?
_ => Self::Bare { table: s.into() },

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

parse_identifiers will return String anyway (wrapped in Ident) so only this last case could actually be Cow::Borrowed(...)

also unsure about how to handle this last case, i followed in footsteps of what was done before (just taking entire string), since alternative would be to panic which seems undesirable. also unsure if desirable to normalize it or not

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think documenting the behavior is about the best we can do in this case.

Comment on lines +204 to +208
.map(|id| match id.quote_style {
Some(_) => id.value,
// TODO: someway to be able to toggle this functionality?
None => id.value.to_ascii_lowercase(),
})

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

comparing with datafusion-sql package where some functionality is duplicated:

https://github.com/apache/arrow-datafusion/blob/ac876dbc9729b16e272e00496c51e53d9f649173/datafusion/sql/src/utils.rs#L540-L546

unsure if also should have a way to toggle normalization, as for datafusion-sql package it allows it:

https://github.com/apache/arrow-datafusion/blob/c323721192ba2733a56c4201b2255a36b1eaa859/datafusion/sql/src/planner.rs#L424-L430

but since this parse_str is meant to be used to implement From<&str>, there wouldn't really be a way to pass information on whether to toggle or not

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree -- if users want to normalize their identifiers I think they'll have to do it manually.

}

/// Parse a string into a TableReference, by splittig on `.`
fn parse_identifiers(s: &str) -> Result<Vec<Ident>> {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could maybe upstream this whole function into sqlparser-rs instead?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That certainly seems like a good idea (at least to file a ticket to track)

@Jefffrey

Jefffrey commented Feb 5, 2023

Copy link
Copy Markdown
Contributor Author

@alamb you mentioned possibility of removing OwnedTableReference #4532 (comment)

however even though using Cow now for TableReference so could own the string, it still carries lifetime requirements which would make it tedious to replace usages of OwnedTableReference with TableReference and having to ensure all the lifetime requirements, so will keep it around for now (unless theres some other solution)

@alamb

alamb commented Feb 5, 2023

Copy link
Copy Markdown
Contributor

I think CI is failing due to an issue fixed in #5177

@alamb

alamb commented Feb 5, 2023

Copy link
Copy Markdown
Contributor

however even though using Cow now for TableReference so could own the string, it still carries lifetime requirements which would make it tedious to replace usages of OwnedTableReference with TableReference and having to ensure all the lifetime requirements, so will keep it around for now (unless theres some other solution)

I think that makes sense for this PR. Thanks @Jefffrey

@alamb alamb left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks reasonable to me, thank you @Jefffrey

Comment on lines +204 to +208
.map(|id| match id.quote_style {
Some(_) => id.value,
// TODO: someway to be able to toggle this functionality?
None => id.value.to_ascii_lowercase(),
})

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree -- if users want to normalize their identifiers I think they'll have to do it manually.

/// See more detail in <https://github.com/apache/arrow-datafusion/issues/4532>
/// Will normalize (convert to lowercase) any unquoted identifiers.
/// e.g. `Foo` will be parsed as `foo`, and `"Foo"".bar"` will be parsed as
/// `Foo".bar` (note the preserved case and requiring two double quotes to represent

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
/// `Foo".bar` (note the preserved case and requiring two double quotes to represent
/// `Foo.bar` (note the preserved case and requiring two double quotes to represent

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It needs to contain the single double quote since it was escaped by prepending it with another double quote in the original, unless you mean to change the example?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, I think I was confused. Sorry about that

Comment on lines +212 to +225
1 => Self::Bare {
table: parts.remove(0).into(),
},
2 => Self::Partial {
schema: parts[0],
table: parts[1],
schema: parts.remove(0).into(),
table: parts.remove(0).into(),
},
3 => Self::Full {
catalog: parts[0],
schema: parts[1],
table: parts[2],
catalog: parts.remove(0).into(),
schema: parts.remove(0).into(),
table: parts.remove(0).into(),
},
_ => Self::Bare { table: s },
// TODO: should normalize?
_ => Self::Bare { table: s.into() },

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think documenting the behavior is about the best we can do in this case.

}

/// Parse a string into a TableReference, by splittig on `.`
fn parse_identifiers(s: &str) -> Result<Vec<Ident>> {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That certainly seems like a good idea (at least to file a ticket to track)

@alamb
alamb merged commit 3f9b996 into apache:master Feb 6, 2023
@ursabot

ursabot commented Feb 6, 2023

Copy link
Copy Markdown

Benchmark runs are scheduled for baseline = 0dfc66d and contender = 3f9b996. 3f9b996 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Skipped ⚠️ Benchmarking of arrow-datafusion-commits is not supported on ec2-t3-xlarge-us-east-2] ec2-t3-xlarge-us-east-2
[Skipped ⚠️ Benchmarking of arrow-datafusion-commits is not supported on test-mac-arm] test-mac-arm
[Skipped ⚠️ Benchmarking of arrow-datafusion-commits is not supported on ursa-i9-9960x] ursa-i9-9960x
[Skipped ⚠️ Benchmarking of arrow-datafusion-commits is not supported on ursa-thinkcentre-m75q] ursa-thinkcentre-m75q
Buildkite builds:
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java

@Jefffrey
Jefffrey deleted the table_reference_parsing branch February 6, 2023 21:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

core Core DataFusion crate sql SQL Planner substrait Changes to the substrait crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

String --> TableReference parsing does not properly handle " and .

3 participants