Skip to content

[ZEPPELIN-6457] Extract shared Python ZeppelinContext helpers for PySpark and PyFlink - #5405

Merged
ParkGyeongTae merged 1 commit into
apache:masterfrom
move-hoon:ZEPPELIN-6457-python-context
Aug 9, 2026
Merged

ParkGyeongTae merged 1 commit into
apache:masterfrom
move-hoon:ZEPPELIN-6457-python-context

Conversation

@move-hoon

Copy link
Copy Markdown
Contributor

What is this PR for?

PySpark and PyFlink currently define their ZeppelinContext subclasses separately in the classic Python and IPython bootstrap resources, even though the implementations are effectively the same for each backend.

This PR moves the PySpark and PyFlink implementations into the shared zeppelin_context.py resource and reuses them from both bootstrap paths.

Backend-specific behavior remains unchanged:

  • PySpark handles Spark DataFrame objects through _jdf.
  • PyFlink handles Flink Table objects through _j_table.
  • PyFlink retains support for the stream_type argument.
  • Other object types fall back to PyZeppelinContext.show().
  • IPySparkZeppelinContext and IPyFlinkZeppelinContext remain available as compatibility aliases.

No new dependencies are introduced.

What type of PR is it?

Refactoring

Todos

  • Move the PySpark ZeppelinContext implementation to the shared resource
  • Move the PyFlink ZeppelinContext implementation to the shared resource
  • Reuse the shared implementations from Python and IPython bootstraps
  • Keep the existing IPython class names as compatibility aliases
  • Verify Spark and Flink interpreter packaging
  • Verify PySpark z.show(DataFrame) behavior

What is the Jira issue?

https://issues.apache.org/jira/browse/ZEPPELIN-6457

How should this be tested?

The following checks passed successfully:

./mvnw test \
  -pl python \
  -Dtest=PythonInterpreterTest#testBackendZeppelinContextsAvailable \
  -DfailIfNoTests=false \
  -Dmaven.gitcommitid.skip=true

Result: 1 test passed.

PYSPARK_PYTHON="$(command -v python3.12)" \
PYSPARK_DRIVER_PYTHON="$(command -v python3.12)" \
./mvnw test \
  -pl spark/interpreter \
  --am \
  -Pspark-3.5 \
  -Pspark-scala-2.12 \
  -Dtest=PySparkInterpreterTest#testPySpark \
  -Dsurefire.failIfNoSpecifiedTests=false \
  -Dmaven.gitcommitid.skip=true

Results:

  • PySparkInterpreterTest#testPySpark passed.
  • Existing ScalaTest suite: 18 tests passed.
  • z.show(DataFrame) was exercised successfully.
./mvnw package \
  -pl spark/interpreter \
  --am \
  -Pspark-3.5 \
  -Pspark-scala-2.12 \
  -DskipTests \
  -Dmaven.gitcommitid.skip=true

Result: 12-module reactor build succeeded.

./mvnw package \
  -pl flink/flink-scala-2.12 \
  --am \
  -Pflink-1.20 \
  -DskipTests \
  -Dmaven.gitcommitid.skip=true

Result: 13-module reactor build succeeded.

The generated Spark and Flink interpreter JARs were also verified to contain the shared context resource and the corresponding classic and IPython bootstrap resources.

git diff --check also passes.

Screenshots (if appropriate)

Not applicable. This PR does not change the user interface.

Questions

  • Do the license files need to be updated? No.
  • Are there breaking changes for older versions? No. The existing IPython context names remain available as compatibility aliases.
  • Does this need documentation? No. There are no user-facing behavior or configuration changes.

@move-hoon
move-hoon marked this pull request as ready for review August 9, 2026 09:05

@ParkGyeongTae ParkGyeongTae left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM — clean fix for the long-standing PySpark/PyFlink ZeppelinContext duplication TODO. Backend-specific show() logic and compat aliases are preserved, and there's test coverage for the alias contract. One non-blocking nit inline.

super(IPySparkZeppelinContext, self).show(obj, **kwargs)

z = __zeppelin__ = IPySparkZeppelinContext(intp.getZeppelinContext(), gateway)
z = __zeppelin__ = PySparkZeppelinContext(intp.getZeppelinContext(), gateway)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: PySparkZeppelinContext isn't imported here on purpose — IPythonInterpreter already execs zeppelin_context.py into this same kernel session first. Worth a one-line comment so it's not mistaken for a missing import.

super(IPyFlinkZeppelinContext, self).show(obj, **kwargs)

z = __zeppelin__ = IPyFlinkZeppelinContext(intp.getZeppelinContext(), gateway)
z = __zeppelin__ = PyFlinkZeppelinContext(intp.getZeppelinContext(), gateway)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: same as the PySpark side — PyFlinkZeppelinContext is only in scope because zeppelin_context.py was already exec'd into this kernel session.

@ParkGyeongTae
ParkGyeongTae merged commit e1d96d6 into apache:master Aug 9, 2026
34 of 36 checks passed
@ParkGyeongTae

Copy link
Copy Markdown
Member

Merged into master (e1d96d6).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants