Skip to content

Skip the Db2 rows when the runner cannot fetch the Db2 image - #16929

Open
matrei wants to merge 2 commits into
8.0.xfrom
test/skip-db2-rows-when-image-unreachable
Open

matrei wants to merge 2 commits into
8.0.xfrom
test/skip-db2-rows-when-image-unreachable

Conversation

@matrei

@matrei matrei commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Description

Since the Db2 rows of the Hibernate 7 specs landed in #16750, six Hibernate7 Functional Tests jobs and two Coverage jobs have failed on them with

org.testcontainers.containers.ContainerFetchException: Can't get Docker image: RemoteDockerImage(imageName=icr.io/db2_community/db2:11.5.9.0, ...)
Caused by: org.testcontainers.shaded.org.awaitility.core.ConditionTimeoutException: Condition with Lambda expression in org.testcontainers.images.RemoteDockerImage was not fulfilled within 2 minutes.

All eight ran on runners in the Azure region westus3, which the header of the job log names. About 8 of the 19 jobs that ran there failed, and none of the 36 in other regions. The test output in the build scan shows that the runner's Docker daemon gets no answer from IBM's registry at all. Every attempt fails within about 15 seconds with

Status 500: {"message":"Get \"https://icr.io/v2/\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"}

or with context deadline exceeded. A longer pull timeout, or pulling the image before the tests, would not get it either.

Such a job loses about 20 minutes before it fails:

  • Testcontainers retries the pull for two minutes.
  • GenericContainer.doStart() retries it in full once more while it builds its error message.
  • Each of the four specs with a Db2 row starts over.

The specs now start the container of a row through DatabaseContainers.start(). For Db2, it fetches the image once per test JVM before it starts the container. When the fetch fails, it throws TestAbortedException, so the Db2 rows are reported as skipped with the reason, as they are on arm64. The other Hibernate 7 jobs of the run still run them. A container that fails to start once its image is there still fails its row.

On GitHub Actions, a test task that skipped Db2 rows for that reason ends with a ::warning workflow command, so the job gets a "Db2 rows skipped" annotation with the number of rows and the reason. It shows on the summary page of the run and on the checks of the pull request, not only in the test report.

DatabaseContainers.start() replaces the start block that the seven specs using the shared containers each repeated. For the three specs without a Db2 row, nothing changes.

Verification

  • The four specs with a Db2 row pass, all 32 rows, the Db2 rows included.
  • With the Db2 image pointed at a registry that cannot be resolved, db2.invalid, the first Db2 row gave up after one round of attempts, 206 s. The other three were skipped within 2 s, each with "The Db2 image could not be fetched, so the Db2 rows are skipped: …", and the build passed. With GITHUB_ACTIONS=true, the build printed one ::warning title=Db2 rows skipped::4 Db2 rows of :grails-data-hibernate7-core:test skipped. The Db2 image could not be fetched, … line, also when it reused the configuration cache entry.
  • All tests of grails-data-hibernate7-core pass: 4,607 tests, with the 15 skipped as before.

Merging

It merges up cleanly to 8.1.x, 9.0.x, 9.1.x and 10.0.x.

The Docker daemon of some GitHub-hosted runners, so far all in the Azure
region westus3, gets no answer from IBM's registry, icr.io, where the
Db2 image comes from: every pull fails within about 15 seconds with
Docker's 500 for "Get https://icr.io/v2/", "request canceled while
waiting for connection" or "context deadline exceeded". Since the Db2
rows landed in #16750, that has failed six Hibernate7 Functional Tests
jobs and two Coverage jobs, each after about 20 minutes: Testcontainers
retries the pull for two minutes, GenericContainer.doStart() retries it
in full once more while it builds its error message, and each of the
four specs with a Db2 row starts over.

The specs now start the container of a row through
DatabaseContainers.start(). For Db2 it fetches the image once per test
JVM before it starts the container, and when the fetch fails it throws
TestAbortedException, so the Db2 rows are reported as skipped with the
reason, as they are on arm64. The other Hibernate 7 jobs of the run
still run them. A container that fails to start once its image is there
still fails its row.
@codecov

codecov Bot commented Nov 11, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 62.5751%. Comparing base (87231f2) to head (506eaa4).
⚠️ Report is 132 commits behind head on 8.0.x.

Additional details and impacted files

Impacted file tree graph

@@                Coverage Diff                 @@
##                8.0.x     #16929        +/-   ##
==================================================
+ Coverage     62.3608%   62.5751%   +0.2142%     
- Complexity      27305      27545       +240     
==================================================
  Files            2248       2249         +1     
  Lines          111469     111725       +256     
  Branches        20422      20486        +64     
==================================================
+ Hits            69513      69912       +399     
+ Misses          32622      32454       -168     
- Partials         9334       9359        +25     

see 42 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@jamesfredley jamesfredley left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not approving. One failed image fetch skips every later row in that process, even after the image can be fetched.

}

private static synchronized void fetchDb2Image() {
if (db2FetchFailure == null) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The first row cannot fetch the image, then the registry recovers before the next row. The next row is skipped without trying again. A row that can fetch the image should run. Try the fetch again after a failure.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Each failed job so far already did what this asks: every Db2 row retried for about 2.5 minutes on its own, so each job retried for about 20 minutes, and none of the 28 rows in the 7 failed jobs I checked got the image (all 16 attempts in the build scan failed too). A short registry outage is already covered by the retries inside the first fetch. A later retry would add about 7.5 minutes to each such job and has not recovered once so far. A skipped row shows in the report with its reason, and the other Hibernate 7 jobs of the run still run the Db2 rows. If we ever see a runner recover, retrying is a small change.

@matrei
matrei requested a review from jamesfredley November 14, 2026 21:53
@jdaugherty

Copy link
Copy Markdown
Contributor

Doesn't this have the potential to mask failures? Since they'll always be skipped? There's no indication on a PR when a test is skipped this way.

A Db2 row skipped because the image could not be fetched showed only in
the test report, so neither the job nor its pull request showed that the
Db2 rows had not run there.

On GitHub Actions, a test task of grails-data-hibernate7-core that
skipped Db2 rows for that reason now ends with a warning workflow
command, which adds an annotation with the number of rows and the reason
to the job.
@matrei

matrei commented Nov 21, 2026

Copy link
Copy Markdown
Contributor Author

They are skipped only on a runner whose Docker daemon cannot fetch the image from icr.io. Everything after the fetch fails as before: a Db2 container that does not start, or a Db2 row whose assertions fail. On this PR's own CI, the Db2 rows ran and passed in all four jobs that run them.

Those four jobs are the three Hibernate7 Functional Tests jobs that run grails-data-hibernate7-core (one for each Java and indy combination) and Coverage - grails-core. A Db2 failure would go unnoticed only if all four landed on runners that cannot reach icr.io. Since this PR was opened, eight more jobs have failed on the fetch, on PRs into 8.0.x and 9.0.x and on pushes to 8.1.x and 9.1.x. All eight ran on westus3, and nothing else failed in any of them. In each of the seven CI runs among them, the other Hibernate 7 jobs passed, so the Db2 rows ran and passed there. Each of the eight jobs ran for 50 to 75 minutes and then had to be rerun.

You're right that the PR showed nothing when the rows were skipped. The new commit adds a warning: on GitHub Actions, a test task of grails-data-hibernate7-core that skipped Db2 rows because the image could not be fetched ends with a ::warning workflow command. The job then gets an annotation, which shows on the run's summary page and on the PR's Checks tab, for example:

Db2 rows skipped
4 Db2 rows of :grails-data-hibernate7-core:test skipped. The Db2 image could not be fetched, so the Db2 rows are skipped: Can't get Docker image: RemoteDockerImage(imageName=icr.io/db2_community/db2:11.5.9.0, …)

With the image pointed at an unreachable registry, the build passed with the 4 Db2 rows of this branch skipped and printed that warning once, also when the configuration cache entry was reused. With the real image, every row ran and there was no warning.

@testlens-app

testlens-app Bot commented Nov 21, 2026

Copy link
Copy Markdown

✅ All tests passed ✅

🏷️ Commit: 506eaa4
▶️ Tests: 94272 executed
⚪️ Checks: 96/96 completed


Learn more about TestLens at testlens.app/docs.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

3 participants