Skip to content

chore: refactor the spark connect image and bump deps for Spark 4.2.0 - #1713

Open
razvan wants to merge 4 commits into
mainfrom
refactor/spark-connect-client
Open

razvan wants to merge 4 commits into
mainfrom
refactor/spark-connect-client

Conversation

@razvan

@razvan razvan commented Oct 22, 2026 •

Copy link
Copy Markdown
Member

Description

Images rebuilt and tested locally.

Fixes this CI problem https://github.com/stackabletech/docker-images/actions/runs/39698894427/attempts/1

Definition of Done Checklist

Note

Not all of these items are applicable to all PRs, the author should update this template to only leave the boxes in that are relevant.

Please make sure all these things are done and tick the boxes

  • Changes are OpenShift compatible
  • All added packages (via microdnf or otherwise) have a comment on why they are added
  • Things not downloaded from Red Hat repositories should be mirrored in the Stackable repository and downloaded from there
  • All packages should have (if available) signatures/hashes verified
  • Add an entry to the CHANGELOG.md file
  • Integration tests ran successfully
TIP: Running integration tests with a new product image

The image can be built and uploaded to the kind cluster with the following commands:

boil build <IMAGE> --image-version <RELEASE_VERSION> --strip-architecture --load
kind load docker-image <MANIFEST_URI> --name=<name-of-your-test-cluster>

See the output of boil to retrieve the image manifest URI for <MANIFEST_URI>.

@razvan razvan self-assigned this Oct 22, 2026
@razvan
razvan marked this pull request as ready for review October 22, 2026 08:06
@razvan razvan moved this to Development: Waiting for Review in Stackable Engineering Oct 22, 2026
Comment on lines +10 to +12
jupyterlab-version = "4.4.1"
scikit-learn-version = "1.3.1"
matplotlib-version = "3.10.1"

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any guidance on how to find the right versions to use?

For future SDP releases, should existing Spark Connect version's dependencies be bumped (patch levels at least). And for new Spark Connect versions added, can we bump to the latest available or do we need to look somewhere for valid versions?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NickLarsenNZ
NickLarsenNZ previously approved these changes Oct 22, 2026

@NickLarsenNZ NickLarsenNZ left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM but maybe add some guidance for future maintenance if you can.

@NickLarsenNZ NickLarsenNZ moved this from Development: Waiting for Review to Development: In Review in Stackable Engineering Oct 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Development: In Review

Development

Successfully merging this pull request may close these issues.

2 participants