Skip to content

More cert patches - #1567

Closed
SohierDane wants to merge 1 commit into
certifi-patchfrom
more_certificate_patches
Closed

SohierDane wants to merge 1 commit into
certifi-patchfrom
more_certificate_patches

Conversation

@SohierDane

Copy link
Copy Markdown
Contributor

Summary

Follow-up to the certifi/pip/uv patch (stacked on certifi-patch). The uv advisory says to check every package that ships its own copy of the CA bundle. A scan of the built CPU image found the compromised emSign ECC Root CA - G3 still
trusted outside of certifi/pip/uv. This PR removes it, or bypasses the copy where no fix exists yet.

Changes

Fixed upstream:

  • OS certificate store (/etc/ssl/certs; used by OpenSSL, curl, wget, git, apt, Python ssl): upgrade ca-certificates from 20260601 to 20260816ubuntu0.24.04.1 (noble-security), which blocks G3.
    • Remove once the base image ships that version.
  • Java cacerts: the package upgrade doesn't rebuild it, and neither does update-ca-certificates --fresh on its own. Delete it and regenerate it with update-ca-certificates --fresh.
    • Remove once the base image ships a cacerts without G3.
  • wandb: 0.28.1 has G3 built into librust_parquet_ffi.so. Pin wandb>=0.31.0, whose binaries I checked and found clean.
    • Remove once the base image ships wandb >= 0.31.0.

No upstream fix yet, so these now use the patched system bundle:

  • grpcio 1.84.0 (latest) ships its own roots.pem. Set GRPC_DEFAULT_SSL_ROOTS_FILE_PATH.
    • Remove once a grpcio release drops G3.
  • Google Cloud SDK 585 has five bundled copies across gcloud, gsutil and bq. The latest release, 590, predates the fix. Set CLOUDSDK_CORE_CUSTOM_CA_CERTS_FILE; gcloud passes it on to gsutil and bq.
    • Remove once an SDK release drops G3.
  • Node v20.19.0 (end-of-life) has G3 in its built-in root store. Set NODE_OPTIONS=--use-openssl-ca.
    • Remove once a Node release drops G3.

Each workaround has a "Remove once the base image ships …" comment, worded like the existing pillow pin. This PR also adds that comment to the earlier uv, certifi and pip changes.

The gcloud setting is an environment variable because gcloud config set --installation fails in this base image. gcloud can't find its install directory, so it ignores /tools/google-cloud-sdk/properties. That also means the base image's
disable_usage_reporting = True in that file has no effect, which is worth raising with the Colab team.

Testing

Built and tested on CPU (./build && ./test): 201 tests passed, 13 skipped.

Six new tests in tests/test_certifi.py; all six fail on the image before this change and pass after:

  • test_system_store_patched: decodes the bundle and the individual *.pem files. It also checks that the unaffected emSign G1 root is still trusted, so an empty store can't pass.
  • test_java_keystore_patched: same check, using keytool.
  • test_gcloud_uses_system_store, test_grpc_uses_system_store, test_node_uses_system_store: check the redirects. The Node test also starts Node, because Node refuses to start if NODE_OPTIONS contains a flag it doesn't accept.
  • test_wandb_binaries_patched: scans the bytes of each file in wandb/bin.

The GPU image was not built or tested locally (no GPU on the build machine); CI's GPU job needs to cover it.

Known remaining copies (not addressed)

A rescan of the new image still finds G3 in:

  • Bypassed by this PR, still on disk: gcloud's bundled certifi copies and both gRPC roots.pem files.
  • Left alone as low exposure:
    • tensorboard_data_server: unmaintained since 2023; only goes online to read gs:// logs.
    • R openssl cacert.pem: R's curl uses the system store.
    • Julia cert.pem: believed to be only a fallback behind the system store; not tested.
    • libnssckbi.so: no Ubuntu update yet, and nothing in the image appears to use it.

@SohierDane
SohierDane requested a review from rosbo October 20, 2026 21:38
@SohierDane
SohierDane added this pull request to stack #1568 October 20, 2026 21:38
@SohierDane

Copy link
Copy Markdown
Contributor Author

@rosbo should I close this PR since we're planning on inheriting a patch from Colab?

@rosbo rosbo closed this Oct 21, 2026
An error occurred while trying to automatically change base from certifi-patch to main October 21, 2026 20:01
@rosbo

rosbo commented Oct 21, 2026

Copy link
Copy Markdown
Contributor

gcloud and grpc is being fixed upstream. I will rely on this to address these issues.

@SohierDane
SohierDane removed this pull request from stack #1568 October 21, 2026 20:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants