Skip to content

fix: Airflow demo improvements - #457

Merged
adwk67 merged 3 commits into
mainfrom
fix/airflow-demo-improvements
Oct 2, 2026
Merged

adwk67 merged 3 commits into
mainfrom
fix/airflow-demo-improvements

Conversation

@adwk67

@adwk67 adwk67 commented Sep 30, 2026 •

Copy link
Copy Markdown
Member

This PR fixes:

  • replaces cluster-wide permissions with namespace-scoped ones
  • replaces minio and mc images from quay.io/minio with ones from docker.io/pgsty/mc
  • the airflow versions so it uses 3.3.1 (and not 3.1.6 as determined previously by the podOverrides)

And changes:

  • the spark sensor such that logs are streamed from the driver and executor pods to the Airflow UI
  • N.B. this does not necessitate granting Airflow any extra permissions that it has already anyway to access Spark logs (at whatever level)
image

@adwk67 adwk67 self-assigned this Sep 30, 2026
@sweb
sweb self-requested a review September 30, 2026 09:19
@adwk67 adwk67 moved this to Development: In Review in Stackable Engineering Sep 30, 2026
sweb
sweb previously approved these changes Oct 2, 2026

@sweb sweb left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minor nit, lgtm

Comment thread demos/airflow-scheduled-job/01-airflow-demo-clusterrole.yaml
Comment thread stacks/airflow/airflow.yaml Outdated

@sweb sweb left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@adwk67
adwk67 merged commit 43005a8 into main Oct 2, 2026
2 checks passed
@adwk67
adwk67 deleted the fix/airflow-demo-improvements branch October 2, 2026 14:35
@adwk67 adwk67 moved this from Development: In Review to Development: Done in Stackable Engineering Oct 2, 2026
@lfrancke lfrancke moved this from Development: Done to Done in Stackable Engineering Oct 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Development

Successfully merging this pull request may close these issues.

3 participants