Skip to content

Time out the functional tests at twice their slowest run - #16941

Merged
matrei merged 1 commit into
9.0.xfrom
ci/functional-shard0-timeout
Nov 12, 2026
Merged

matrei merged 1 commit into
9.0.xfrom
ci/functional-shard0-timeout

Conversation

@matrei

@matrei matrei commented Nov 12, 2026

Copy link
Copy Markdown
Contributor

Description

On 9.x, Functional Tests (Java 21, indy=false), which is shard 0 of that cell, keeps hitting its time limit. The step stops a cell at 70 minutes and the job at 80. Four of the last eleven runs of that cell on 9.x and 10.x were stopped with exit code 124 while still building, with no task or test failing:

Two more runs passed at 69 and 70 minutes.

The limits came with the workaround for the Groovy 6 worker teardown hang (403f43d). They were based on 8.0.x runs in which shard 0 took 48 to 55 minutes. Shard 0 builds every module and runs the checks as well as its half of the tests, while shard 1 runs only its half. The step runs with --rerun-tasks, which a78c022 added so that CI always runs the tests, so every run builds everything from scratch, without the build cache. This PR leaves that as it is. On 9.x and 10.x, shard 0 now takes 41 to 70 minutes and shard 1 takes 23 to 41. Shard 0 has about 55 minutes of work, and runner speed varies by 30 to 40% from one job to the next, so a slow runner pushes it past 70 minutes.

The job now gets 150 minutes. That is twice its slowest run, rounded up to the half hour, which is the rule the comment above the test jobs states and the other test jobs follow. 8.0.x and 8.1.x, which don't have the hang workaround, already give this job 150 minutes. The step stops every cell except the known hanging one at 140 minutes, 10 minutes before the job's limit, so that a cell that times out still uploads its Geb reports. The hanging cell, Java 21, indy=true, shard 1, keeps its 45 minutes and its recovery.

A test JVM that never exits now holds a runner for up to 140 minutes instead of 70. That is the same limit the other test jobs already give such a hang.

Verification

  • The workflow parses, and the functional job reads timeout-minutes: 150, with step limits of 45 and 140 minutes.
  • The change merges cleanly into 9.1.x and 10.0.x.

On 9.x the functional job stops a cell at 70 minutes and the job at 80.
Those limits came with the workaround for the Groovy 6 worker teardown
hang, from 8.0.x runs in which shard 0 took 48 to 55 minutes. Shard 0
builds every module and runs the checks as well as its half of the
tests, and on 9.x and 10.x it now takes 41 to 70 minutes. A slow runner
pushes it past the limit: four of the last eleven runs of the Java 21,
indy=false shard 0 on 9.x and 10.x were stopped while still building,
with no test failing.

The job now gets 150 minutes, twice its slowest run rounded up to the
half hour. That is the rule the other test jobs follow, and what the
same job already has on 8.0.x and 8.1.x. The step stops every cell but
the known hanging one at 140 minutes, so that a cell that times out
still uploads its Geb reports. The hanging cell keeps its 45 minutes.
@testlens-app

testlens-app Bot commented Nov 12, 2026 •

Copy link
Copy Markdown

🚨 All tests passed but jobs failed 🚨

Failed Jobs without Test Failures

❌ Groovy Snapshot Canary Build / Build Grails (shard 0)
❌ Groovy Snapshot Canary Build / Build Grails (shard 1)
❌ Groovy Snapshot Canary Build / Build Grails (shard 2)

🏷️ Commit: 4810d22
▶️ Tests: 110389 executed
⚪️ Checks: 96/96 completed

Rerun Controls

Click the checkbox to trigger a rerun:

  • Rerun jobs

Learn more about TestLens at testlens.app/docs.

@codecov

codecov Bot commented Nov 12, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 65.3140%. Comparing base (97d4c9c) to head (4810d22).

Additional details and impacted files

Impacted file tree graph

@@                Coverage Diff                 @@
##                9.0.x     #16941        +/-   ##
==================================================
+ Coverage     65.3123%   65.3140%   +0.0017%     
- Complexity      32320      32322         +2     
==================================================
  Files            2430       2430                
  Lines          120429     120429                
  Branches        22290      22290                
==================================================
+ Hits            78655      78657         +2     
- Misses          31781      31783         +2     
+ Partials         9993       9989         -4     

see 3 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@jamesfredley jamesfredley left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The functional test timeout is longer. The known hanging cell keeps its shorter limit.

@matrei
matrei merged commit a74c662 into 9.0.x Nov 12, 2026
96 of 99 checks passed
@matrei
matrei deleted the ci/functional-shard0-timeout branch November 12, 2026 21:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

2 participants