Skip to content

fix(spark): handle overflowing ports in parse_url - #390

Open
Subodh-17 wants to merge 1 commit into
apache:mainfrom
Subodh-17:fix/parse-url-overflow-port-365
Open

Subodh-17 wants to merge 1 commit into
apache:mainfrom
Subodh-17:fix/parse-url-overflow-port-365

Conversation

@Subodh-17

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

In Apache Spark, parse_url delegates component extraction to java.net.URI, which tokenizes the authority component without enforcing WHATWG 16-bit integer bounds (0 <= port <= 65535). Consequently, Spark successfully extracts requested components (such as HOST, AUTHORITY, PATH, QUERY, REF, and PROTOCOL) for URLs containing ports above 65535 (e.g. parse_url('http://spark.apache.org:99999/path', 'HOST') -> 'spark.apache.org').

Previously, datafusion-spark relied exclusively on Rust's url::Url::parse, which fails with ParseError::InvalidPort for any port above 65535, causing parse_url to return NULL in non-ANSI mode and raise an InvalidPort execution error in ANSI mode.

What changes are included in this PR?

  • Adds a targeted port-normalization fallback in ParseUrl::parse activated strictly upon Err(ParseError::InvalidPort):
    • Isolates the authority component while safely respecting userinfo (user:pass@), bracketed IPv6 addresses ([::1]:99999), and delimiters (/, ?, #).
    • Rejects authorities containing whitespace, maintaining parity with JVM URISyntaxException.
    • Replicates Spark/JVM signed 32-bit port semantics: ports <= 2147483647 extract host; ports > 2147483647 or non-numeric ports (:abc, :-80) return NULL for host while preserving raw authority, path, query, and ref.
    • Substitutes a valid placeholder port (:80) in a sanitized URL string for WHATWG component extraction, but injects the verbatim unmutated authority string when part == "AUTHORITY".
    • Preserves empty-path semantics ("" vs "/") by checking Self::path against the original URL.
  • Valid URLs (0 <= port <= 65535, default ports, schemeless) continue to evaluate through the standard fast path with zero overhead.

Are these changes tested?

  • Added unit tests in crates/spark/src/function/url/parse_url.rs:
    • test_parse_url_port_overflow_and_spark_parity: tests standard ports (8080, 80), boundary ports (65535, 65536), overflow ports (99999, 1000000), 32-bit overflow ports (9999999999), empty ports, no ports, and non-numeric ports (:abc).
    • test_parse_url_structures_with_overflow_port: tests IPv4, bracketed IPv6, userinfo, encoded path characters, empty paths, and query-only paths with port 99999.
    • test_parse_url_overflow_port_eval_modes: tests legacy, try, and ANSI evaluation modes.
  • Updated crates/sqllogictest/test_files/url/parse_url.slt:
    • Removed # xfail spark: ... (#365) comments on lines 339 and 364 per runner guidelines.
    • Updated expected result to 'spark.apache.org' for both legacy and ANSI modes.
    • Added SLT test queries for AUTHORITY, PATH, QUERY, REF, PROTOCOL, and bracketed IPv6 with port 99999.
  • Verified expected outputs against live JVM java.net.URI execution and PySpark 4.2.0.

Are there any user-facing changes?

Yes. parse_url no longer rejects URLs containing ports above 65535, matching Apache Spark behavior.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] parse_url rejects a port above 65535, Spark returns the requested part

1 participant