Repository navigation
Time out the functional tests at twice their slowest run - #16941
Merged
Merged
Conversation
On 9.x the functional job stops a cell at 70 minutes and the job at 80. Those limits came with the workaround for the Groovy 6 worker teardown hang, from 8.0.x runs in which shard 0 took 48 to 55 minutes. Shard 0 builds every module and runs the checks as well as its half of the tests, and on 9.x and 10.x it now takes 41 to 70 minutes. A slow runner pushes it past the limit: four of the last eleven runs of the Java 21, indy=false shard 0 on 9.x and 10.x were stopped while still building, with no test failing. The job now gets 150 minutes, twice its slowest run rounded up to the half hour. That is the rule the other test jobs follow, and what the same job already has on 8.0.x and 8.1.x. The step stops every cell but the known hanging one at 140 minutes, so that a cell that times out still uploads its Geb reports. The hanging cell keeps its 45 minutes.
🚨 All tests passed but jobs failed 🚨Failed Jobs without Test Failures❌ Groovy Snapshot Canary Build / Build Grails (shard 0) 🏷️ Commit: 4810d22 Rerun ControlsClick the checkbox to trigger a rerun:
Learn more about TestLens at testlens.app/docs. |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## 9.0.x #16941 +/- ##
==================================================
+ Coverage 65.3123% 65.3140% +0.0017%
- Complexity 32320 32322 +2
==================================================
Files 2430 2430
Lines 120429 120429
Branches 22290 22290
==================================================
+ Hits 78655 78657 +2
- Misses 31781 31783 +2
+ Partials 9993 9989 -4 🚀 New features to boost your workflow:
|
jamesfredley
approved these changes
Nov 12, 2026
jamesfredley
left a comment
Contributor
There was a problem hiding this comment.
The functional test timeout is longer. The known hanging cell keeps its shorter limit.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
On 9.x,
Functional Tests (Java 21, indy=false), which is shard 0 of that cell, keeps hitting its time limit. The step stops a cell at 70 minutes and the job at 80. Four of the last eleven runs of that cell on 9.x and 10.x were stopped with exit code 124 while still building, with no task or test failing:Two more runs passed at 69 and 70 minutes.
The limits came with the workaround for the Groovy 6 worker teardown hang (403f43d). They were based on 8.0.x runs in which shard 0 took 48 to 55 minutes. Shard 0 builds every module and runs the checks as well as its half of the tests, while shard 1 runs only its half. The step runs with
--rerun-tasks, which a78c022 added so that CI always runs the tests, so every run builds everything from scratch, without the build cache. This PR leaves that as it is. On 9.x and 10.x, shard 0 now takes 41 to 70 minutes and shard 1 takes 23 to 41. Shard 0 has about 55 minutes of work, and runner speed varies by 30 to 40% from one job to the next, so a slow runner pushes it past 70 minutes.The job now gets 150 minutes. That is twice its slowest run, rounded up to the half hour, which is the rule the comment above the test jobs states and the other test jobs follow. 8.0.x and 8.1.x, which don't have the hang workaround, already give this job 150 minutes. The step stops every cell except the known hanging one at 140 minutes, 10 minutes before the job's limit, so that a cell that times out still uploads its Geb reports. The hanging cell, Java 21, indy=true, shard 1, keeps its 45 minutes and its recovery.
A test JVM that never exits now holds a runner for up to 140 minutes instead of 70. That is the same limit the other test jobs already give such a hang.
Verification
timeout-minutes: 150, with step limits of 45 and 140 minutes.