Repository navigation
Conversation
The Docker daemon of some GitHub-hosted runners, so far all in the Azure region westus3, gets no answer from IBM's registry, icr.io, where the Db2 image comes from: every pull fails within about 15 seconds with Docker's 500 for "Get https://icr.io/v2/", "request canceled while waiting for connection" or "context deadline exceeded". Since the Db2 rows landed in #16750, that has failed six Hibernate7 Functional Tests jobs and two Coverage jobs, each after about 20 minutes: Testcontainers retries the pull for two minutes, GenericContainer.doStart() retries it in full once more while it builds its error message, and each of the four specs with a Db2 row starts over. The specs now start the container of a row through DatabaseContainers.start(). For Db2 it fetches the image once per test JVM before it starts the container, and when the fetch fails it throws TestAbortedException, so the Db2 rows are reported as skipped with the reason, as they are on arm64. The other Hibernate 7 jobs of the run still run them. A container that fails to start once its image is there still fails its row.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## 8.0.x #16929 +/- ##
==================================================
+ Coverage 62.3608% 62.5751% +0.2142%
- Complexity 27305 27545 +240
==================================================
Files 2248 2249 +1
Lines 111469 111725 +256
Branches 20422 20486 +64
==================================================
+ Hits 69513 69912 +399
+ Misses 32622 32454 -168
- Partials 9334 9359 +25 🚀 New features to boost your workflow:
|
jamesfredley
left a comment
There was a problem hiding this comment.
Not approving. One failed image fetch skips every later row in that process, even after the image can be fetched.
| } | ||
|
|
||
| private static synchronized void fetchDb2Image() { | ||
| if (db2FetchFailure == null) { |
There was a problem hiding this comment.
The first row cannot fetch the image, then the registry recovers before the next row. The next row is skipped without trying again. A row that can fetch the image should run. Try the fetch again after a failure.
There was a problem hiding this comment.
Each failed job so far already did what this asks: every Db2 row retried for about 2.5 minutes on its own, so each job retried for about 20 minutes, and none of the 28 rows in the 7 failed jobs I checked got the image (all 16 attempts in the build scan failed too). A short registry outage is already covered by the retries inside the first fetch. A later retry would add about 7.5 minutes to each such job and has not recovered once so far. A skipped row shows in the report with its reason, and the other Hibernate 7 jobs of the run still run the Db2 rows. If we ever see a runner recover, retrying is a small change.
|
Doesn't this have the potential to mask failures? Since they'll always be skipped? There's no indication on a PR when a test is skipped this way. |
A Db2 row skipped because the image could not be fetched showed only in the test report, so neither the job nor its pull request showed that the Db2 rows had not run there. On GitHub Actions, a test task of grails-data-hibernate7-core that skipped Db2 rows for that reason now ends with a warning workflow command, which adds an annotation with the number of rows and the reason to the job.
|
They are skipped only on a runner whose Docker daemon cannot fetch the image from icr.io. Everything after the fetch fails as before: a Db2 container that does not start, or a Db2 row whose assertions fail. On this PR's own CI, the Db2 rows ran and passed in all four jobs that run them. Those four jobs are the three Hibernate7 Functional Tests jobs that run You're right that the PR showed nothing when the rows were skipped. The new commit adds a warning: on GitHub Actions, a test task of
With the image pointed at an unreachable registry, the build passed with the 4 Db2 rows of this branch skipped and printed that warning once, also when the configuration cache entry was reused. With the real image, every row ran and there was no warning. |
✅ All tests passed ✅🏷️ Commit: 506eaa4 Learn more about TestLens at testlens.app/docs. |
Description
Since the Db2 rows of the Hibernate 7 specs landed in #16750, six Hibernate7 Functional Tests jobs and two Coverage jobs have failed on them with
All eight ran on runners in the Azure region westus3, which the header of the job log names. About 8 of the 19 jobs that ran there failed, and none of the 36 in other regions. The test output in the build scan shows that the runner's Docker daemon gets no answer from IBM's registry at all. Every attempt fails within about 15 seconds with
or with
context deadline exceeded. A longer pull timeout, or pulling the image before the tests, would not get it either.Such a job loses about 20 minutes before it fails:
GenericContainer.doStart()retries it in full once more while it builds its error message.The specs now start the container of a row through
DatabaseContainers.start(). For Db2, it fetches the image once per test JVM before it starts the container. When the fetch fails, it throwsTestAbortedException, so the Db2 rows are reported as skipped with the reason, as they are on arm64. The other Hibernate 7 jobs of the run still run them. A container that fails to start once its image is there still fails its row.On GitHub Actions, a test task that skipped Db2 rows for that reason ends with a
::warningworkflow command, so the job gets a "Db2 rows skipped" annotation with the number of rows and the reason. It shows on the summary page of the run and on the checks of the pull request, not only in the test report.DatabaseContainers.start()replaces the start block that the seven specs using the shared containers each repeated. For the three specs without a Db2 row, nothing changes.Verification
db2.invalid, the first Db2 row gave up after one round of attempts, 206 s. The other three were skipped within 2 s, each with "The Db2 image could not be fetched, so the Db2 rows are skipped: …", and the build passed. WithGITHUB_ACTIONS=true, the build printed one::warning title=Db2 rows skipped::4 Db2 rows of :grails-data-hibernate7-core:test skipped. The Db2 image could not be fetched, …line, also when it reused the configuration cache entry.grails-data-hibernate7-corepass: 4,607 tests, with the 15 skipped as before.Merging
It merges up cleanly to 8.1.x, 9.0.x, 9.1.x and 10.0.x.