Repository navigation
9.0 (fix): Report a failure an error handler answered once on Spring Framework 7.1 - #17088
Conversation
Undertow restores the dispatcher type once an include or a forward to a named servlet returns, but not once a forward to a path does, so the request went on reading as a forward. A filter that ran before the forward read it as one too once its chain had returned: Spring Framework 7.1's ServerHttpObservationFilter took an asynchronous dispatch that had forwarded, as one to an error controller does, for a plain dispatch, stopped its observation there as well as when the asynchronous processing completed, and reported the failure twice. An inner handler chain wrapper now puts back the dispatcher type of the dispatch a forward returns to, as Tomcat and Jetty do.
Spring Framework 7.1.0 reads the failure of an asynchronous request off the request once the processing completes, as it does at the end of any other, and offers ServerHttpObservationFilter.recordHandledException for a failure a framework handled. Grails still told the observation itself whenever the filter had not read the failure at the end of an asynchronous dispatch, so the handlers of the observation, one of which marks the span, heard of it twice. The resolvers now hand the failure to the filter, which reports it once the exchange completes. The first failure of a request is the one it reports: what fails after it, such as asynchronous processing the action started timing out, or an error handler finding the response already used, does not replace it. When Tomcat reports the failure an error controller left on a request that started asynchronous processing, it is taken off the request before the filter would read it again. This replaces the reading of the filter's private observation attribute and the decision, made once a dispatch returned, of whether the filter would read the failure itself.
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## 9.0.x #17088 +/- ##
==================================================
+ Coverage 65.5441% 65.5491% +0.0049%
- Complexity 32764 32774 +10
==================================================
Files 2439 2442 +3
Lines 121094 121123 +29
Branches 22434 22435 +1
==================================================
+ Hits 79370 79395 +25
- Misses 31698 31700 +2
- Partials 10026 10028 +2
🚀 New features to boost your workflow:
|
jamesfredley
left a comment
There was a problem hiding this comment.
Requested changes. A later failure is reported, and the first failure is not.
| * @param failure what failed, as the observation reports it | ||
| */ | ||
| static void recordServerError(HttpServletRequest request, Throwable failure) { | ||
| if (request.getAttribute(RECORDED_ATTRIBUTE) != null) { |
There was a problem hiding this comment.
An action fails. The error handler then fails too. The second failure is recorded first, so the first failure is not the one that is reported. Record the first failure before the error handler runs.
There was a problem hiding this comment.
@jamesfredley Thanks, you're right. Fixed in 1d2e3cc.
The error controller's own failure was resolved inside the forward to it, so it got recorded first, and the action's failure was skipped once the forward returned. GrailsExceptionResolver now passes the error handler to ServerErrorObservation.recordServerError(request, failure, errorHandler). That claims the action's failure before the handler runs and records it once the handler has answered. A failure resolved inside the handler is left alone, and the action's failure is recorded over one that Spring MVC records as a handled server error while the handler runs, such as a 503 ResponseStatusException. If the handler throws without answering, the claim is dropped and its own failure reaches the observation, as before.
Tests:
GrailsExceptionResolverSpec: an error controller that fails through the real dispatcher servlet, with the failure resolved by Grails and by Spring MVC'sResponseStatusExceptionResolver. Both cases failed before this commit.ServerErrorObservationSpec: the handler overload, including a handler that fails without answering.FailedActionObservationSpecand its Jetty and Undertow copies: a new case, "an error controller that fails". On the previous head it reported the error controller'sIllegalArgumentExceptioninstead of the action'sIndexOutOfBoundsException, on all three containers. It now passes: 28/28 on Tomcat, 24/24 on Jetty, 24/24 on Undertow.
One combination is still not covered: an action that starts async processing and fails, on Tomcat, with an error controller that then fails with a 5xx that Spring MVC resolves. Tomcat reports the action's failure to the async listeners itself, so this code takes it off the request rather than recording it again. That leaves Spring's later record to be read when the request completes. Covering it means removing Spring's private handled-exception attribute, or recording the action's failure a second time. Spring Framework 7.1.0 reports an async failure more than once even without Grails, and I've opened spring-projects/spring-framework#37731 for that, with a fix ready. Once it ships, recording the failure again in that case is reported once, and the gap can close.
An error controller that failed as well had its own failure resolved inside the forward to it, and that failure was recorded on the observation first, so the one the action threw, recorded once the forward returned, did not replace it. The failure is now claimed before the error handler runs and recorded once it has answered, over a failure Spring MVC recorded as a handled server error meanwhile.
jamesfredley
left a comment
There was a problem hiding this comment.
Requested changes. A later failure is still the one that is reported.
…corded as handled When the container had already reported the failure an error handler answered, as Tomcat does for one left on a request that started asynchronous processing, a server error Spring MVC recorded as handled after it, such as the 503 of the processing timing out or of an error controller that fails, was read when the processing completed and became the error of the observation. That later record is now taken back, so the observation keeps the first failure and its handlers hear of it once.
…t dispatch sent is answered A request dispatch that started asynchronous processing and then sent an error, as Spring MVC does for a server error it resolves, left that processing open forever on Undertow. Undertow sends the error, through the error page that maps to it, once the dispatch has returned, and ends the exchange with it. The processing then never completed and never timed out, so its listeners were never told: the observation of the request never stopped, and a DeferredResult was never resolved. A handler now completes the processing once the exchange has ended, as Tomcat and Jetty do.
🚨 All tests passed but jobs failed 🚨Failed Jobs without Test Failures❌ Groovy Snapshot Canary Build / Build Grails (shard 0) 🏷️ Commit: 1acb304 Rerun ControlsClick the checkbox to trigger a rerun:
Learn more about TestLens at testlens.app/docs. |
matrei
left a comment
There was a problem hiding this comment.
Reviewed at 1acb304 (base 9.0.x). The merge base is fdde94a, and the head also merges cleanly with today's 9.0.x (8d2ef9e). I checked the code against the sources of Spring Framework 7.1.0, Tomcat 11.0.27 and undertow-servlet 2.0.3.
The fix is right. The resolvers now hand the failure to the filter through recordHandledException, and the claim around the error handler keeps the first failure. FirstFailureListener covers the request whose dispatch started asynchronous processing on both kinds of container, Tomcat, which reports the failure left on the request, and Jetty and Undertow, which don't. Both Undertow handlers fix what they say they fix.
What I ran:
- Unit tests of
grails-web-common,grails-web-mvc,grails-web-url-mappingsandgrails-undertow-spring-bootat the head: 1122 tests, all pass. FailedActionObservationSpec,JettyFailedActionObservationSpecandUndertowFailedActionObservationSpecat the head: 30, 26 and 26 tests, all pass.- The head's tests on fdde94a, without the change:
- 11 of the 70 features of
GrailsExceptionResolverSpecfail. Those are the asynchronous cases, and the error controller and later failure cases. - 3 of the 16 tests of
UndertowServletWebServerFactoryTestsfail: the dispatcher type test and the two completion tests.
- 11 of the 70 features of
- The Undertow completion tests on 8.0.x, with and without
AsyncErrorCompletionHandler. See 1. - Two more rows on all three containers, for an asynchronous dispatch that fails and is answered by an error controller that fails as well. See 2.
Before or after merge
1. The Undertow completion fix also belongs on 8.0.x.
grails-undertow is on 8.0.x with the same Undertow 2.4.4 / undertow-servlet 2.0.3. Its servlet package and UndertowServletWebServerFactoryTests are identical to the ones on 9.0.x.
On origin/8.0.x (96a4ea4), asynchronousProcessingCompletesOnceAnErrorItsRequestDispatchSentIsAnswered and asynchronousProcessingCompletesOnceTheErrorPageHasAnsweredAnErrorItsRequestDispatchSent fail. With only AsyncErrorCompletionHandler and its line in the factory added, all three completion tests pass.
On 8.0.x this is a plain bug: the observation of such a request never stops, and a DeferredResult never resolves. It doesn't depend on Spring Framework 7.1.
I'd keep it here, since the 9.0.x specs need it, and open a PR to 8.0.x with the handler and its three tests. The new file merges up as an identical addition, and the factory line next to ForwardDispatcherTypeHandler is a one-line conflict.
The other two parts stay on 9.0.x only:
- The Spring Framework 7.1 parts: 8.x is on Boot 4.1.3, which keeps the 8.0.x workarounds this PR removes.
- The dispatcher type restore: it changes what a filter reads, and it has its upgrade note.
If the completion fix stays on 9.0.x only, the Undertow bullet of the upgrade note needs a sentence for it too: AsyncListener.onComplete now fires for such a request, and with it the completion callbacks of a DeferredResult. As a fix on 8.0.x, it needs no note.
2. One combination is still reported three times on Tomcat. The case: an asynchronous dispatch fails, and its error controller fails with a 503 that Spring MVC resolves. I added these rows to the three apps:
def answeredByControllerFailingWithServerErrorAsynchronously() {
failAsynchronously(new IllegalMonitorStateException('answered by an error controller that fails with a server error'))
}'asynchronously' | '/failing/answeredByControllerFailingWithServerErrorAsynchronously'Jetty and Undertow pass. Tomcat gives NOTIFIED == [IllegalMonitorStateException, IllegalMonitorStateException, IllegalMonitorStateException], and the observation stops twice. The handler saw the calls in this order:
onCompletefromAsyncContextImpl.doInternalDispatch→recordError, then stop.onErrorfromCoyoteAdapter.asyncDispatch→setErrorState, becauseresponse.isError()after thesendError(503).onCompleteagain fromAsyncStateMachine.asyncPostProcess→recordError, then stop.
That's the [onComplete, onError, onComplete] from your bz 70362, which any asynchronous dispatch that ends in sendError gets on Tomcat, with or without Grails. This PR doesn't cause it, and Grails can't add a listener during that dispatch to correct it.
I'd still add the row, so the gap is tracked:
- Add it to the Jetty and Undertow specs.
- On Tomcat, add it as a separate feature with
@PendingFeature(reason = 'https://bz.apache.org/bugzilla/show_bug.cgi?id=70362').
Then the Tomcat upgrade that fixes it turns the feature into a failure that says so. The description should also name this case, since it says the failure is reported once.
The variant where Grails resolves the error controller's failure (IndexOutOfBoundsException, answered by failingErrorHandler after a DeferredResult error) is reported once on all three containers.
Notes
-
Spring Framework 7.1.1 changes what completion reads. Commit spring-projects/spring-framework@088fb71f, milestone 7.1.1, closed your #37731. With it,
ObservationAsyncListener.onCompletereads only whatrecordHandledExceptionrecorded, notjakarta.servlet.error.exception.I traced each case against that diff, and each is still reported once. Where Tomcat doesn't report the failure,
FirstFailureListenerrecords it itself. Two things go stale with the bump, though:request.removeAttribute(RequestDispatcher.ERROR_EXCEPTION)inFirstFailureListener.onComplete(ServerErrorObservation.groovy:140) is only needed on 7.1.0.- The class Javadoc at
:70-71says the filter reads the failure off the request once the exchange completes. For an asynchronous request that no longer holds.
A comment that points at #37731 would tell whoever bumps Boot what can go.
-
ServerHttpObservationFilter.recordHandledException(request, null)(ServerErrorObservation.groovy:142) passes null where Spring doesn't allow it.org.springframework.web.filteris@NullMarked, anderrorisn't@Nullable. It works becausesetAttribute(name, null)removes the attribute.There's no public way to clear it, so I'd keep it. The rows "…when the container reports it and a failure Spring MVC records as handled follows" pin it, so if Spring starts rejecting null, a test fails. The comment could say that the call steps outside Spring's contract, not only that it removes the attribute.
-
Nit:
ServerErrorObservation.groovy:107is 168 characters long. The rest of the file wraps at 120.
Verified as correct
- James's two points are fixed.
the exception of an action stays the error of the observation when its error controller fails as wellanda failure that follows the one an error handler answered does not replace itfail on fdde94a and pass here, with both Grails and Spring MVC resolving. - Listener order:
FirstFailureListeneris added during the dispatch, and the filter adds its own listener once the chain has returned. SoFirstFailureListenerhearsonCompletefirst, on all three containers, and also afteronStartAsyncre-adds both. - Tomcat reports a failure left on a request whose dispatch started asynchronous processing (
CoyoteAdapter.java:366-372). That's the casereportedByContainercatches. After an asynchronous dispatch, Tomcat reports one only whenresponse.isError()(:240-245). A forward to an error controller there sets a status and sends no error, so Tomcat doesn't report it, andrecordHandledExceptionalone reports it once. - Undertow's forward to a path goes through
dispatchToPath(RequestDispatcherImpl.java:186), which never restores the dispatcher type.dispatchToServletand includes (:347) do restore it.- The inner chain wrapper runs inside every dispatch, so nested forwards restore the type of the dispatch they return to.
- The error dispatch runs after the chain has returned (
ServletInitialHandler.java:285), outside the wrapper.
AsyncErrorCompletionHandler:handleFirstRequestskipsctx.complete()(:339) becausestartAsyncdispatched the exchange.- On the request thread,
complete()only flags the processing as completed (AsyncContextImpl.java:264), hence thehandleCompletedBeforeInitialRequestDone()call. - A dispatch that completes the processing itself is left alone. The third test, which passes on fdde94a too, guards that.
- Nothing refers to the removed workarounds any more: the private
.observationattribute,holdFailureLeftOnRequestand thedoServiceoverride.holdFailureLeftOnRequestis not in v8.0.0. - CI is green apart from the Groovy Snapshot Canary. Its
Build Grailsshards fail compilinggrails-coreandgrails-async-groovy-concurrentagainst the Groovy snapshot, modules this PR doesn't touch. The canary offeat/forge-plugin-commands-9.0.xfails in the same fourcompileGroovytasks. Two Ubuntu core builds were still running when I wrote this.
|
Merging since this fixes functional tests. Will create follow up PR for non blocking |
|
@matrei Thanks. #17088 was already merged, so the follow-ups went into new PRs:
|
Since #17084 moved 9.0.x to Spring Boot 4.2.0, the functional tests fail on every lane:
FailedActionObservationSpecand its Jetty and Undertow copies find the failure of an asynchronous request reported twice, for exampleNOTIFIED[path] == [UnsupportedOperationException, UnsupportedOperationException]. The same spec passes on 4.2.0-RC2 and fails on 4.2.0 with nothing else changed.Why
Spring Framework 7.1.0 (Boot 4.2.0; 7.1.0-RC2 did not) changed how
ServerHttpObservationFilterends an asynchronous request: its listener now reads the failure off the request once the processing completes, as the filter does at the end of any other request, before it stops the observation. It also addedServerHttpObservationFilter.recordHandledException(request, error), which Spring MVC's own exception resolvers now use for a server error they handled.Grails still told the observation itself whenever the filter had not read the failure at the end of an asynchronous dispatch, so the filter and Grails both reported it. Two container quirks then reported some of these requests a second time on their own:
onError, and again from the request ononComplete.What changed
ServerErrorObservation(grails-web-common) is the one place bothGrailsExceptionResolverandActionExceptionHandlerResolverrecord a server error. It hands the failure to the filter throughrecordHandledException, or leaves it injakarta.servlet.error.exceptionwhen an error controller left it there, and the filter reports it once the exchange completes.DefaultHandlerExceptionResolvernow records as a 503), or an error handler finding the response already used, does not replace it. That includes an error controller that fails as well: the action's failure is claimed before the error handler runs and recorded once it has answered, over whatever the handler's own failure recorded. For a request that started asynchronous processing, a listener added ahead of the filter's restores the first failure when the processing completes. If Tomcat already reported the failure, the listener instead takes it off the request so the filter does not read it again, together with any server error Spring MVC recorded as handled after it.grails-undertowrestores the dispatcher type of a dispatch once a forward it made returns, through an inner handler chain wrapper, as Tomcat and Jetty do. A filter that readsgetDispatcherType()once its chain has returned now seesREQUESTorASYNCrather thanFORWARD; this has an entry in the 9.0 upgrade guide and the upgrade skill.grails-undertowalso completes the asynchronous processing a request dispatch started once an error that dispatch sent has been answered, as Tomcat and Jetty do. Undertow sends the error once the dispatch returns and ends the exchange with it, but never completed the processing or timed it out, so its listeners never heard of it: the observation of such a request never stopped, and aDeferredResultwas never resolved. Spring MVC sends such an error for a server error it resolves, for example when an error controller fails with aResponseStatusExceptionafter the action started asynchronous processing. The same fix goes to 8.0.x in 8.0 (fix): Complete asynchronous processing on Undertow once an error its request dispatch sent is answered #17100, where it doesn't depend on Spring Framework 7.1.GrailsExceptionResolver.holdFailureLeftOnRequestand thedoServiceoverride inGrailsDispatcherServlet).Limitation
On Tomcat, one case is still reported three times, and its observation stops twice. In that case, an asynchronous dispatch fails and is answered by an error controller, which then fails with a server error that Spring MVC resolves, such as a 503
ResponseStatusException. Tomcat tells the listeners of any asynchronous dispatch that ends insendErrorthat the processing completed, then that it failed, then that it completed again, with or without Grails. That is Tomcat bug 70362, with a fix in apache/tomcat#1264. Grails can't add a listener during that dispatch to correct it. Jetty and Undertow report the case once. #17101 tracks it as a pending feature ofFailedActionObservationSpec.Tests
FailedActionObservationSpec,JettyFailedActionObservationSpecandUndertowFailedActionObservationSpecpass locally: 30, 26 and 26 tests.GrailsExceptionResolverSpecnow runs its asynchronous cases through the real lifecycle: a request dispatch that starts asynchronous processing, the asynchronous dispatch, and the completion the filter's listener hears of. It adds cases for an action that started asynchronous processing before it failed (with and without the container reporting the failure), and for a later timeout resolved by Spring MVC or by Grails. All seven fail against 9.0.x without this change. It also covers an error controller that fails as well, its failure resolved by Grails or by Spring MVC, and a later failure resolved once the error view or an error controller has answered. The functional specs on all three containers gain an error controller that fails, and one that fails with a 503 Spring MVC resolves; that also runs after the action started asynchronous processing, which never completed on Undertow before this change. The asynchronous variants that simulated Undertow's leftover forward are gone, sincegrails-undertownow corrects it.ServerErrorObservationSpecis new.ActionExceptionProblemDetailsSpecnow observes through the filter instead of a bare context, which nothing reports any more.UndertowServletWebServerFactoryTestschecks the dispatcher type a filter reads after a forward from a request dispatch and from an asynchronous dispatch. Without the wrapper it readsFORWARDboth times. It also checks that asynchronous processing a request dispatch started completes once an error it sent is answered, with and without an error page, and once only when the dispatch completes it itself. Without the new handler the listeners never hearonComplete.grails-web-common,grails-web-mvc,grails-web-url-mappingsandgrails-undertow-spring-boottests, Checkstyle and CodeNarc pass.