Skip to content

feat: add datafusion-spark-cli, a SQL shell with the Spark functions - #326

Open
andygrove wants to merge 5 commits into
apache:mainfrom
andygrove:feat-spark-cli
Open

andygrove wants to merge 5 commits into
apache:mainfrom
andygrove:feat-spark-cli

Conversation

@andygrove

Copy link
Copy Markdown
Member

Which issue does this PR close?

Part of #56. It covers the datafusion-cli --spark references in the "Update the imported docs" item, since those READMEs now point to the new CLI.

Rationale for this change

apache/datafusion#27337 removed datafusion-cli --spark, which enabled the Spark SQL dialect and registered the Spark-compatible functions. Since then there is no way to try the functions interactively, and crates/spark/README.md and crates/sqllogictest/test_files/README.md still tell people to run datafusion-cli --spark.

What changes are included in this PR?

  • crates/cli, a new datafusion-spark-cli binary. It uses the datafusion-cli library for the REPL, for running files and commands, for output formats, and for querying files by path or URL (SELECT * FROM 'data.parquet'). Its main.rs is about 200 lines, following datafusion-cli's.
    • Always Spark: the Spark SQL dialect and the Spark functions are always enabled. The Spark functions override DataFusion's functions of the same name.
    • ANSI mode on by default, as in Spark 4 (spark.sql.ansi.enabled = true). --ansi false gives Spark's legacy behavior. The startup banner says which mode is on.
    • Options kept from datafusion-cli: -c, -f, -r, -p, -b, --format, -q, --maxrows and --color. The memory and disk tuning options are left out for now.
  • Not published: publish = false.
  • Not a default workspace member: datafusion-cli brings in the object store and AWS dependencies, so a plain cargo build or cargo test at the root skips this crate. The clippy job uses --workspace, so it still lints the crate.
  • New workflow, .github/workflows/cli.yml. It builds the CLI and runs a smoke test:
    • sha2 returns Spark's result.
    • abs(cast(-2147483648 AS int)) fails by default, because ANSI mode is on.
    • The same query returns -2147483648 with --ansi false.
  • Docs: crates/spark/README.md, test_files/README.md, the root README.md and AGENTS.md describe the new CLI.

datafusion-cli on DataFusion's branch-56 still depends on that branch's own datafusion-spark, so the build compiles that copy too. Only this repository's functions are registered. The extra copy goes away with DataFusion 57, where datafusion-cli no longer depends on datafusion-spark.

Are these changes tested?

  • The smoke test's three queries pass locally.
  • I also tried the REPL reading from stdin, a backquoted identifier (Spark dialect), --help and the banner.
  • cargo clippy --all-targets --all-features --workspace -- -D warnings is clean.
  • cargo test -p datafusion-spark and cargo test -p datafusion-spark-sqllogictest still pass.
  • actionlint reports no problems with the new workflow.

Are there any user-facing changes?

There is a new, unpublished datafusion-spark-cli binary. You run it with cargo run -p datafusion-spark-cli.

This PR was written by an LLM (Claude) and reviewed before opening.

🤖 Generated with Claude Code

apache/datafusion#27337 removed `datafusion-cli --spark`, so there is no
longer a way to try the Spark-compatible functions interactively. Add a
`datafusion-spark-cli` binary that uses the `datafusion-cli` library for
the shell, with the Spark SQL dialect and functions always enabled and
ANSI mode on by default, as in Spark 4.

The crate is not published, and is not a default workspace member, so a
plain `cargo build` or `cargo test` does not build its dependencies. A
new workflow builds it and runs a smoke test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@kosiew kosiew left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@andygrove, thanks for this. Reusing DataFusion's CLI while configuring Spark-specific behavior is the right approach.

Requesting changes because relative script paths can resolve incorrectly or panic when combined with --data-path. The remaining suggestions improve CI coverage and documentation.

Comment thread crates/cli/src/main.rs
);
}

if let Some(path) = &args.data_path {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With -p, relative -f and -r paths can execute the wrong file or panic. Please absolutize script paths before changing directories, preserving /dev/stdin, and use exec_from_lines to handle file-open errors. Test both flags with conflicting filenames.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, thanks. Fixed in f0cf6de. With -p, the -f and -r paths are now made absolute before the directory change, and the stdin names are left as they are. The files are opened here and run with exec_from_lines, so a file that can't be opened is an error rather than a panic. I also found that -f /dev/stdin was rejected at argument parsing because a pipe isn't a regular file, so parse_valid_file now accepts the stdin names. CI runs both flags with a q.sql in the working directory and a different q.sql in the data directory. The step fails without the fix.

Comment thread .github/workflows/cli.yml
run: |
set -euo pipefail
cli=target/debug/datafusion-spark-cli
$cli -q --format csv -c "SELECT sha2('Spark', 256) AS h" \

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we add a stdin REPL smoke test using CAST(1 AS LONG) and assert the result row? Current queries also pass with the generic dialect, so they do not verify Spark parsing.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added in f0cf6de. CI pipes SELECT CAST(1 AS LONG) AS l; into the REPL and checks for the row 1.

Comment thread .github/workflows/cli.yml Outdated
cli=target/debug/datafusion-spark-cli
$cli -q --format csv -c "SELECT sha2('Spark', 256) AS h" \
| grep -x '529bc3b07127ecb7e53a4dcf1991d9152c24537d919178022b2c42657f79a26b'
if $cli -q -c "SELECT abs(cast(-2147483648 AS int))"; then

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please also assert [ARITHMETIC_OVERFLOW] in the captured output. A startup failure or panic could otherwise satisfy the nonzero-exit check.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in f0cf6de. The step captures the output and checks it contains [ARITHMETIC_OVERFLOW].

Comment thread crates/spark/README.md Outdated
The option is disabled by default because Spark-compatible functions can
override DataFusion functions with the same name.
The Spark functions override DataFusion functions with the same name. ANSI mode
is on by default, as in Spark 4. Pass `--ansi false` for Spark's legacy behavior.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please document cargo run -p datafusion-spark-cli -- --ansi false here and in the SQLLogicTest README. The -- separator is required to pass --ansi to the CLI.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in f0cf6de. Both READMEs now show cargo run -p datafusion-spark-cli -- --ansi false and explain the --.

With -p the CLI changed directory before opening the -f and -r files, so a
relative path ran the file of that name in the data directory, or panicked if
there was none. Make those paths absolute first, open the files ourselves so
a failure is an error rather than a panic, and accept -f /dev/stdin.

CI now also runs the REPL from stdin with a Spark-only type, checks the
ARITHMETIC_OVERFLOW message, and runs -f and -r with a data path. The READMEs
show that --ansi false goes after -- with cargo run.

@kosiew kosiew left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@andygrove, thanks for addressing the previous feedback. The original issues are resolved, and the added tests and documentation look good.

One regression remains: a later file-open failure can occur after earlier scripts have executed. Please restore open-before-execute ordering before merging.

Comment thread crates/cli/src/main.rs Outdated
files: &[String],
print_options: &PrintOptions,
) -> Result<()> {
for path in files {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Opening files individually allows earlier scripts to produce side effects before a later file fails to open. Could we open all -f/-r files into a Vec before set_current_dir, then execute them? Please test that an unreadable second script leaves no COPY output.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point, thanks. Fixed in 224e924. All the -f or -r files are now opened into a Vec<File> before anything runs. CI has a new step where the first file does a COPY ... TO 'out.csv' and the second is unreadable. It checks for the open error and that out.csv was never written. That step fails on the previous commit.

Comment thread crates/cli/src/main.rs Outdated
if is_stdin_location(path) {
return Ok(path.clone());
}
Ok(absolute(path)?.to_string_lossy().into_owned())

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-UTF-8 launch directories on Linux can make to_string_lossy() corrupt otherwise valid paths. Opening files before changing directories removes this issue; otherwise, preserve PathBuf through File::open.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 224e924 the way you suggested. The files are opened before changing to the data directory, so absolute_paths and its to_string_lossy() are gone and nothing converts the path.

Comment thread .github/workflows/cli.yml
echo "SELECT 'data' AS s;" > data/q.sql
$cli -q --format csv -p data -f q.sql | grep -x 'cwd'
echo "" | $cli -q --format csv -p data -r q.sql | grep -x 'cwd'
echo "SELECT 'stdin' AS s;" | $cli -q --format csv -p data -f /dev/stdin | grep -x 'stdin'

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we also test that -p resolves the script from the launch directory while relative CSV paths resolve from the data directory? Use distinct CSV contents to verify both behaviors.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added in 224e924. The launch directory and data/ each have a t.csv with different contents, and data/ also has a decoy csv.sql. The script is run with -p data. CI checks that the script runs from the launch directory and reads data/t.csv.

A file that cannot be opened now fails before an earlier -f or -r script has
run. Opening the files before changing to the -p data directory also replaces
the conversion to absolute paths, which went through to_string_lossy.
@andygrove

Copy link
Copy Markdown
Member Author

@kosiew thanks for the second pass. 224e924 opens every script before any of them run and before the directory change, which also removes the to_string_lossy() path. It also adds CI coverage for both the unreadable second file and -p path resolution. Could you take another look?

@andygrove
andygrove requested a review from kosiew November 20, 2026 15:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants