Repository navigation
fix(e6): map empty identifiers to empty_alias on the native executor - #323
Open
tkaunlaky-e6 wants to merge 2 commits into
Open
tkaunlaky-e6 wants to merge 2 commits into
tkaunlaky-e6 wants to merge 2 commits into
Conversation
e6 rejects an empty identifier, so a Databricks `count(*) AS `` ` came out as `COUNT(*) AS ""` and failed to parse (Encountered "\""). With E6_EXECUTOR_TYPE=native, an empty alias is now dropped and the engine names the column (e.g. EXPR$1); the values are unchanged. The java path is unchanged.
Dropping an empty alias broke queries that refer to that column by its empty
name, e.g. SELECT `` FROM (SELECT count(*) AS `` ...): Databricks resolves ``
as a normal column name (in SELECT, qualified s.``, CTEs, ORDER BY, GROUP BY),
but e6 cannot parse "". On the native executor every empty identifier is now
emitted as empty_alias, so the alias and its references resolve to the same
column. Values match Databricks; only the column name differs ('' there,
empty_alias here). Two empty-named columns referenced by name are ambiguous
in both engines.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Databricks treats
`` (an empty name) as a normal column name. Checked on Spark 4.2, ANSI mode on and off:count(*) AS ``produces a column named'';`` everywhere: outerSELECT, qualified `s.```, through a CTE, `ORDER BY`, `GROUP BY`.e6 cannot parse an empty identifier, so the transpiled
AS ""failed withEncountered "\"". This is the "double-quoted string" variant in the TomTom native coverage sheet: 2 statements, one of which ran OK on Databricks.Fix
With
E6_EXECUTOR_TYPE=native,identifier_sqlwrites every empty identifier asempty_alias. Because the alias and every reference to it get the same name, they stay linked, e.g.SELECT `` FROM (SELECT count(*) AS `` FROM t) s ORDER BY ``→SELECT empty_alias FROM (SELECT COUNT(*) AS empty_alias FROM t) AS s ORDER BY empty_aliasThe java path is unchanged.
Behaviour vs Databricks
''empty_aliasAMBIGUOUS_REFERENCEerrorColumn 'empty_alias' is ambiguouserrorShapes checked on both, with the same results: single and multiple empty aliases, empty alias on a plain column, subquery and CTE with
SELECT *, outerSELECT ```, qualifieds., CTE column referenced by empty name, `ORDER BY,GROUP BY ```,GROUP BYwithORDER BYordinals,UNION ALL`, and window functions.Testing
make checkpasses: ruff, ruff-format, mypy, and the full suite with both tokenizers (1013 tests each).test_empty_alias_native_executorcovers the alias, a reference through a subquery plusORDER BY, a normal alias (unchanged), and java (unchanged). 2 of its checks fail without the fix.