Repository navigation
9.0 (fix): Track the asynchronous failure Tomcat reports three times, and note what Spring Framework 7.1.1 makes stale - #17101
Conversation
…ror controller fails An asynchronous dispatch that fails, answered by an error controller that fails with a server error Spring MVC resolves, is reported once on Jetty and Undertow, and three times on Tomcat. Tomcat tells the listeners of an asynchronous dispatch that ends in an error it sent that the processing completed, then that it failed, then that it completed again (Tomcat bug 70362), and Grails cannot add a listener during that dispatch to correct it. The case is a row of the Jetty and Undertow specs, and a pending feature on Tomcat, so the Tomcat upgrade that fixes it says so.
From 7.1.1 (spring-projects/spring-framework#37731), the observation filter reads only what was recorded as handled once asynchronous processing completes, so removing the container's error attribute there is only needed on 7.1.0, and the Javadoc that says the filter reads the failure off the request no longer holds for an asynchronous request. Also say that clearing the handled failure with null steps outside Spring's contract, and wrap the class Javadoc at 120 characters.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## 9.0.x #17101 +/- ##
=============================================
Coverage 65.6431% 65.6431%
Complexity 33022 33022
=============================================
Files 2446 2446
Lines 121408 121408
Branches 22524 22524
=============================================
Hits 79696 79696
+ Misses 31647 31642 -5
- Partials 10065 10070 +5
🚀 New features to boost your workflow:
|
jamesfredley
left a comment
There was a problem hiding this comment.
Requested changes. One asynchronous failure is still reported three times.
matrei
left a comment
There was a problem hiding this comment.
Reviewed at 5c80974 (base 9.0.x). It merges cleanly with today's 9.0.x (733397b). Each change is what the description says, and 9.0.x is the right branch: ServerErrorObservation and these rows exist only there.
What I ran:
-
Unit tests of
grails-web-commonat the head: 434 tests, all pass. -
FailedActionObservationSpec,JettyFailedActionObservationSpecandUndertowFailedActionObservationSpecat the head:- Results: 31 tests on Tomcat (one skipped as pending), 27 on Jetty and 27 on Undertow.
- The new row passes on Jetty and Undertow.
- On Tomcat, the pending feature fails on the
NOTIFIEDassertion (FailedActionObservationSpec.groovy:146), as intended. The 503 and theSTOPPEDassertions before it pass.
-
Two stand-ins for a Spring Framework change to
recordHandledException(request, null)(ServerErrorObservation.groovy:147), to check the new comment's claim:- When the call throws, as Spring rejecting null would make it, three rows of
ServerErrorObservationSpecfail: those where the container reports the failure. - When the call is dropped, as Spring ignoring null would amount to, the row "…when the container reports it and a failure Spring MVC records as handled follows" fails.
So the claim holds either way.
- When the call throws, as Spring rejecting null would make it, three rows of
On James's point
James asks to report the failure once rather than leave the test pending. That isn't something Grails can do here, and the pending feature is what shows the gap:
- The three notifications and the second stop come from Tomcat calling the listeners of the processing three times:
onComplete,onError,onComplete. That's bz 70362. - On apache/tomcat#1264, a Tomcat committer reproduced it on Tomcat's main branch with the plain servlet from the bug,
[onComplete, onError(null), onComplete], and got[onComplete]with that PR. - Spring's listener stops the observation on every
onCompleteit hears. During that asynchronous dispatch, Grails can't add a listener, so it can't take any of the extra calls back. - The feature runs on every build and fails as pending. Once a Tomcat release with the fix arrives, it fails the build instead, and the annotation comes off.
I'd keep it as it is.
Notes
-
If #17100 takes the Javadoc fix for
AsyncErrorCompletionHandler(Tomcat doesn't complete the processing, it leaves it to the asynchronous timeout; only Jetty completes it), I'd make the same edit here. The 9.0.x copy says the same, and identical additions on both branches leave the merge-up of #17100 nothing to resolve in that file. -
Nit: in the Javadoc of the new feature (
FailedActionObservationSpec.groovy:127-129), "an error it sent" reads as if Tomcat sent the error, and "with or without Grails" ends up describing the observation:* Tomcat tells the listeners of an asynchronous dispatch that ends in {@code sendError} that the processing * completed, then that it failed, then that it completed again, with or without Grails. So the observation reports * the failure three times and stops twice, and Grails can't add a listener during that dispatch to correct it.
Verified as correct
-
The two #37731 comments say what goes with Spring Framework 7.1.1 and what stays:
- The removal of
jakarta.servlet.error.exception(ServerErrorObservation.groovy:141-143) is only needed while completion reads that attribute, which it does on 7.1.0 only. - The
recordHandledException(request, null)call stays after the bump: 7.1.1 still reads the handled attribute once the processing completes.
- The removal of
-
The
recordServerErrorJavadoc (:71-72) now says that, from 7.1.1, the filter stops reading the failure off the request once asynchronous processing completes. -
The rewrapped
FirstFailureListenerJavadoc fits in 120 columns. The new feature's labels are single-quoted, and its name matches the unrolled name of the Jetty and Undertow row. -
The #17088 description now names the Tomcat case under "Limitation".
-
CI fails only where nothing in this PR reaches:
- The Groovy Snapshot Canary fails as it did on #17088.
Build Grails Forge (Java 25, indy=false)failed resolving dependencies (Could not find groovy-bom-4.0.33.pomfrom repo.grails.org).Hibernate5 Functional Tests (Java 21, indy=false)was cancelled.
A rerun of the last two should do.
… who reports it The Javadoc read as if Tomcat sent the error, and put "with or without Grails" on the observation rather than on what Tomcat tells the listeners.
|
@matrei Thanks.
|
🚨 All tests passed but jobs failed 🚨Failed Jobs without Test Failures❌ Groovy Snapshot Canary Build / Build Grails (shard 0) 🏷️ Commit: 0d700e2 Rerun ControlsClick the checkbox to trigger a rerun:
Learn more about TestLens at testlens.app/docs. |
jamesfredley
left a comment
There was a problem hiding this comment.
The items this review found have been addressed. The repeated reports come from the server and Spring, not from this change, and the pending test records them.
Follow-up to #17088.
The Tomcat case that is still reported three times
On Tomcat, an asynchronous dispatch can fail and then be answered by an error controller that fails as well, with a server error that Spring MVC resolves (a 503
ResponseStatusException). That failure is still reported three times, and its observation stops twice. Tomcat tells the listeners of any asynchronous dispatch that ends insendErrorthat the processing completed, then that it failed, then that it completed again, with or without Grails. That is Tomcat bug 70362, with a fix in apache/tomcat#1264. Grails can't add a listener during that dispatch to correct it.The case is now a row of
FailedActionObservationSpecand its two copies:@PendingFeature(reason = 'https://bz.apache.org/bugzilla/show_bug.cgi?id=70362'). A Tomcat upgrade that fixes it turns the feature into a failure that says so.Notes in
ServerErrorObservationfor the Spring Framework 7.1.1 bumpspring-projects/spring-framework@088fb71f (7.1.1, for spring-projects/spring-framework#37731) changes
ObservationAsyncListener.onCompleteto read only whatrecordHandledExceptionrecorded, notjakarta.servlet.error.exception. Each case is still reported once with it, but two things go stale. Both now point at #37731:request.removeAttribute(RequestDispatcher.ERROR_EXCEPTION)inFirstFailureListener.onCompleteis only needed on 7.1.0.recordServerError(request, failure, errorHandler)says the filter reads the failure off the request once the exchange completes. From 7.1.1 that no longer holds for a request that started asynchronous processing.The comment on
recordHandledException(request, null)now says the call steps outside Spring's contract.org.springframework.web.filteris@NullMarked, anderrorisn't@Nullable. Spring has no public way to clear the attribute, so the call stays.ServerErrorObservationSpecfails if Spring starts rejecting null.The class Javadoc line that ran to 168 characters is rewrapped at 120.
The Undertow completion fix from #17088 goes to 8.0.x separately, in #17100.