Skip to content

9.0 (fix): Report a failure an error handler answered once on Spring Framework 7.1 - #17088

Merged
codeconsole merged 5 commits into
apache:9.0.xfrom
codeconsole:fix/async-observation-reported-twice-9.0.x
Nov 20, 2026
Merged

codeconsole merged 5 commits into
apache:9.0.xfrom
codeconsole:fix/async-observation-reported-twice-9.0.x

Conversation

@codeconsole

@codeconsole codeconsole commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Since #17084 moved 9.0.x to Spring Boot 4.2.0, the functional tests fail on every lane: FailedActionObservationSpec and its Jetty and Undertow copies find the failure of an asynchronous request reported twice, for example NOTIFIED[path] == [UnsupportedOperationException, UnsupportedOperationException]. The same spec passes on 4.2.0-RC2 and fails on 4.2.0 with nothing else changed.

Why

Spring Framework 7.1.0 (Boot 4.2.0; 7.1.0-RC2 did not) changed how ServerHttpObservationFilter ends an asynchronous request: its listener now reads the failure off the request once the processing completes, as the filter does at the end of any other request, before it stops the observation. It also added ServerHttpObservationFilter.recordHandledException(request, error), which Spring MVC's own exception resolvers now use for a server error they handled.

Grails still told the observation itself whenever the filter had not read the failure at the end of an asynchronous dispatch, so the filter and Grails both reported it. Two container quirks then reported some of these requests a second time on their own:

  • Undertow restores the dispatcher type once an include returns, but not once a forward to a path does. The filter therefore took an asynchronous dispatch that had forwarded to an error controller for a plain one, and stopped the observation and reported the failure there as well as on completion.
  • Tomcat reports the failure an error controller left on a request that started asynchronous processing to the listeners of that processing. The filter's listener reports it from onError, and again from the request on onComplete.

What changed

  • ServerErrorObservation (grails-web-common) is the one place both GrailsExceptionResolver and ActionExceptionHandlerResolver record a server error. It hands the failure to the filter through recordHandledException, or leaves it in jakarta.servlet.error.exception when an error controller left it there, and the filter reports it once the exchange completes.
  • The first failure of a request is the one reported. What fails after it, such as asynchronous processing the action started timing out (which Spring MVC's DefaultHandlerExceptionResolver now records as a 503), or an error handler finding the response already used, does not replace it. That includes an error controller that fails as well: the action's failure is claimed before the error handler runs and recorded once it has answered, over whatever the handler's own failure recorded. For a request that started asynchronous processing, a listener added ahead of the filter's restores the first failure when the processing completes. If Tomcat already reported the failure, the listener instead takes it off the request so the filter does not read it again, together with any server error Spring MVC recorded as handled after it.
  • grails-undertow restores the dispatcher type of a dispatch once a forward it made returns, through an inner handler chain wrapper, as Tomcat and Jetty do. A filter that reads getDispatcherType() once its chain has returned now sees REQUEST or ASYNC rather than FORWARD; this has an entry in the 9.0 upgrade guide and the upgrade skill.
  • grails-undertow also completes the asynchronous processing a request dispatch started once an error that dispatch sent has been answered, as Tomcat and Jetty do. Undertow sends the error once the dispatch returns and ends the exchange with it, but never completed the processing or timed it out, so its listeners never heard of it: the observation of such a request never stopped, and a DeferredResult was never resolved. Spring MVC sends such an error for a server error it resolves, for example when an error controller fails with a ResponseStatusException after the action started asynchronous processing. The same fix goes to 8.0.x in 8.0 (fix): Complete asynchronous processing on Undertow once an error its request dispatch sent is answered #17100, where it doesn't depend on Spring Framework 7.1.
  • This removes the workarounds the 8.0.x change needed on Spring Framework 7.0: reading the filter's private observation attribute (spring-framework#37658), and deciding, once a dispatch returned, whether the filter would read the failure itself (GrailsExceptionResolver.holdFailureLeftOnRequest and the doService override in GrailsDispatcherServlet).

Limitation

On Tomcat, one case is still reported three times, and its observation stops twice. In that case, an asynchronous dispatch fails and is answered by an error controller, which then fails with a server error that Spring MVC resolves, such as a 503 ResponseStatusException. Tomcat tells the listeners of any asynchronous dispatch that ends in sendError that the processing completed, then that it failed, then that it completed again, with or without Grails. That is Tomcat bug 70362, with a fix in apache/tomcat#1264. Grails can't add a listener during that dispatch to correct it. Jetty and Undertow report the case once. #17101 tracks it as a pending feature of FailedActionObservationSpec.

Tests

  • FailedActionObservationSpec, JettyFailedActionObservationSpec and UndertowFailedActionObservationSpec pass locally: 30, 26 and 26 tests.
  • GrailsExceptionResolverSpec now runs its asynchronous cases through the real lifecycle: a request dispatch that starts asynchronous processing, the asynchronous dispatch, and the completion the filter's listener hears of. It adds cases for an action that started asynchronous processing before it failed (with and without the container reporting the failure), and for a later timeout resolved by Spring MVC or by Grails. All seven fail against 9.0.x without this change. It also covers an error controller that fails as well, its failure resolved by Grails or by Spring MVC, and a later failure resolved once the error view or an error controller has answered. The functional specs on all three containers gain an error controller that fails, and one that fails with a 503 Spring MVC resolves; that also runs after the action started asynchronous processing, which never completed on Undertow before this change. The asynchronous variants that simulated Undertow's leftover forward are gone, since grails-undertow now corrects it.
  • ServerErrorObservationSpec is new. ActionExceptionProblemDetailsSpec now observes through the filter instead of a bare context, which nothing reports any more.
  • UndertowServletWebServerFactoryTests checks the dispatcher type a filter reads after a forward from a request dispatch and from an asynchronous dispatch. Without the wrapper it reads FORWARD both times. It also checks that asynchronous processing a request dispatch started completes once an error it sent is answered, with and without an error page, and once only when the dispatch completes it itself. Without the new handler the listeners never hear onComplete.
  • grails-web-common, grails-web-mvc, grails-web-url-mappings and grails-undertow-spring-boot tests, Checkstyle and CodeNarc pass.

Undertow restores the dispatcher type once an include or a forward to a
named servlet returns, but not once a forward to a path does, so the
request went on reading as a forward. A filter that ran before the
forward read it as one too once its chain had returned: Spring Framework
7.1's ServerHttpObservationFilter took an asynchronous dispatch that had
forwarded, as one to an error controller does, for a plain dispatch,
stopped its observation there as well as when the asynchronous processing
completed, and reported the failure twice.

An inner handler chain wrapper now puts back the dispatcher type of the
dispatch a forward returns to, as Tomcat and Jetty do.
Spring Framework 7.1.0 reads the failure of an asynchronous request off
the request once the processing completes, as it does at the end of any
other, and offers ServerHttpObservationFilter.recordHandledException for
a failure a framework handled. Grails still told the observation itself
whenever the filter had not read the failure at the end of an
asynchronous dispatch, so the handlers of the observation, one of which
marks the span, heard of it twice.

The resolvers now hand the failure to the filter, which reports it once
the exchange completes. The first failure of a request is the one it
reports: what fails after it, such as asynchronous processing the action
started timing out, or an error handler finding the response already
used, does not replace it. When Tomcat reports the failure an error
controller left on a request that started asynchronous processing, it
is taken off the request before the filter would read it again.

This replaces the reading of the filter's private observation attribute
and the decision, made once a dispatch returned, of whether the filter
would read the failure itself.
@codecov

codecov Bot commented Nov 20, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 83.78378% with 12 lines in your changes missing coverage. Please review.
✅ Project coverage is 65.5491%. Comparing base (fdde94a) to head (1acb304).

Files with missing lines Patch % Lines
...rg/grails/web/errors/ServerErrorObservation.groovy 81.2500% 5 Missing and 1 partial ⚠️
...rtow/core/servlet/AsyncErrorCompletionHandler.java 82.3529% 0 Missing and 3 partials ⚠️
...tow/core/servlet/ForwardDispatcherTypeHandler.java 80.0000% 2 Missing and 1 partial ⚠️
Additional details and impacted files

Impacted file tree graph

@@                Coverage Diff                 @@
##                9.0.x     #17088        +/-   ##
==================================================
+ Coverage     65.5441%   65.5491%   +0.0049%     
- Complexity      32764      32774        +10     
==================================================
  Files            2439       2442         +3     
  Lines          121094     121123        +29     
  Branches        22434      22435         +1     
==================================================
+ Hits            79370      79395        +25     
- Misses          31698      31700         +2     
- Partials        10026      10028         +2     
Files with missing lines Coverage Δ
.../core/servlet/UndertowServletWebServerFactory.java 56.0870% <100.0000%> (+2.1396%) ⬆️
...org/grails/web/errors/GrailsExceptionResolver.java 83.3333% <100.0000%> (-1.7413%) ⬇️
...ils/web/servlet/mvc/GrailsDispatcherServlet.groovy 67.2414% <ø> (-0.5006%) ⬇️
.../mapping/mvc/ActionExceptionHandlerResolver.groovy 67.4419% <100.0000%> (-1.7889%) ⬇️
...rtow/core/servlet/AsyncErrorCompletionHandler.java 82.3529% <82.3529%> (ø)
...tow/core/servlet/ForwardDispatcherTypeHandler.java 80.0000% <80.0000%> (ø)
...rg/grails/web/errors/ServerErrorObservation.groovy 81.2500% <81.2500%> (ø)

... and 8 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@jamesfredley jamesfredley left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requested changes. A later failure is reported, and the first failure is not.

* @param failure what failed, as the observation reports it
*/
static void recordServerError(HttpServletRequest request, Throwable failure) {
if (request.getAttribute(RECORDED_ATTRIBUTE) != null) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

An action fails. The error handler then fails too. The second failure is recorded first, so the first failure is not the one that is reported. Record the first failure before the error handler runs.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jamesfredley Thanks, you're right. Fixed in 1d2e3cc.

The error controller's own failure was resolved inside the forward to it, so it got recorded first, and the action's failure was skipped once the forward returned. GrailsExceptionResolver now passes the error handler to ServerErrorObservation.recordServerError(request, failure, errorHandler). That claims the action's failure before the handler runs and records it once the handler has answered. A failure resolved inside the handler is left alone, and the action's failure is recorded over one that Spring MVC records as a handled server error while the handler runs, such as a 503 ResponseStatusException. If the handler throws without answering, the claim is dropped and its own failure reaches the observation, as before.

Tests:

  • GrailsExceptionResolverSpec: an error controller that fails through the real dispatcher servlet, with the failure resolved by Grails and by Spring MVC's ResponseStatusExceptionResolver. Both cases failed before this commit.
  • ServerErrorObservationSpec: the handler overload, including a handler that fails without answering.
  • FailedActionObservationSpec and its Jetty and Undertow copies: a new case, "an error controller that fails". On the previous head it reported the error controller's IllegalArgumentException instead of the action's IndexOutOfBoundsException, on all three containers. It now passes: 28/28 on Tomcat, 24/24 on Jetty, 24/24 on Undertow.

One combination is still not covered: an action that starts async processing and fails, on Tomcat, with an error controller that then fails with a 5xx that Spring MVC resolves. Tomcat reports the action's failure to the async listeners itself, so this code takes it off the request rather than recording it again. That leaves Spring's later record to be read when the request completes. Covering it means removing Spring's private handled-exception attribute, or recording the action's failure a second time. Spring Framework 7.1.0 reports an async failure more than once even without Grails, and I've opened spring-projects/spring-framework#37731 for that, with a fix ready. Once it ships, recording the failure again in that case is reported once, and the gap can close.

An error controller that failed as well had its own failure resolved
inside the forward to it, and that failure was recorded on the
observation first, so the one the action threw, recorded once the
forward returned, did not replace it. The failure is now claimed before
the error handler runs and recorded once it has answered, over a
failure Spring MVC recorded as a handled server error meanwhile.
@codeconsole

Copy link
Copy Markdown
Contributor Author

@jamesfredley jamesfredley left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requested changes. A later failure is still the one that is reported.

…corded as handled

When the container had already reported the failure an error handler answered, as Tomcat does
for one left on a request that started asynchronous processing, a server error Spring MVC
recorded as handled after it, such as the 503 of the processing timing out or of an error
controller that fails, was read when the processing completed and became the error of the
observation. That later record is now taken back, so the observation keeps the first failure
and its handlers hear of it once.
…t dispatch sent is answered

A request dispatch that started asynchronous processing and then sent an error, as Spring MVC
does for a server error it resolves, left that processing open forever on Undertow. Undertow
sends the error, through the error page that maps to it, once the dispatch has returned, and
ends the exchange with it. The processing then never completed and never timed out, so its
listeners were never told: the observation of the request never stopped, and a DeferredResult
was never resolved. A handler now completes the processing once the exchange has ended, as
Tomcat and Jetty do.
@testlens-app

testlens-app Bot commented Nov 20, 2026 •

Copy link
Copy Markdown

🚨 All tests passed but jobs failed 🚨

Failed Jobs without Test Failures

❌ Groovy Snapshot Canary Build / Build Grails (shard 0)
❌ Groovy Snapshot Canary Build / Build Grails (shard 1)
❌ Groovy Snapshot Canary Build / Build Grails (shard 2)

🏷️ Commit: 1acb304
▶️ Tests: 137734 executed
⚪️ Checks: 100/100 completed

Rerun Controls

Click the checkbox to trigger a rerun:

  • Rerun jobs

Learn more about TestLens at testlens.app/docs.

@matrei matrei left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at 1acb304 (base 9.0.x). The merge base is fdde94a, and the head also merges cleanly with today's 9.0.x (8d2ef9e). I checked the code against the sources of Spring Framework 7.1.0, Tomcat 11.0.27 and undertow-servlet 2.0.3.

The fix is right. The resolvers now hand the failure to the filter through recordHandledException, and the claim around the error handler keeps the first failure. FirstFailureListener covers the request whose dispatch started asynchronous processing on both kinds of container, Tomcat, which reports the failure left on the request, and Jetty and Undertow, which don't. Both Undertow handlers fix what they say they fix.

What I ran:

  • Unit tests of grails-web-common, grails-web-mvc, grails-web-url-mappings and grails-undertow-spring-boot at the head: 1122 tests, all pass.
  • FailedActionObservationSpec, JettyFailedActionObservationSpec and UndertowFailedActionObservationSpec at the head: 30, 26 and 26 tests, all pass.
  • The head's tests on fdde94a, without the change:
    • 11 of the 70 features of GrailsExceptionResolverSpec fail. Those are the asynchronous cases, and the error controller and later failure cases.
    • 3 of the 16 tests of UndertowServletWebServerFactoryTests fail: the dispatcher type test and the two completion tests.
  • The Undertow completion tests on 8.0.x, with and without AsyncErrorCompletionHandler. See 1.
  • Two more rows on all three containers, for an asynchronous dispatch that fails and is answered by an error controller that fails as well. See 2.

Before or after merge

1. The Undertow completion fix also belongs on 8.0.x.

grails-undertow is on 8.0.x with the same Undertow 2.4.4 / undertow-servlet 2.0.3. Its servlet package and UndertowServletWebServerFactoryTests are identical to the ones on 9.0.x.

On origin/8.0.x (96a4ea4), asynchronousProcessingCompletesOnceAnErrorItsRequestDispatchSentIsAnswered and asynchronousProcessingCompletesOnceTheErrorPageHasAnsweredAnErrorItsRequestDispatchSent fail. With only AsyncErrorCompletionHandler and its line in the factory added, all three completion tests pass.

On 8.0.x this is a plain bug: the observation of such a request never stops, and a DeferredResult never resolves. It doesn't depend on Spring Framework 7.1.

I'd keep it here, since the 9.0.x specs need it, and open a PR to 8.0.x with the handler and its three tests. The new file merges up as an identical addition, and the factory line next to ForwardDispatcherTypeHandler is a one-line conflict.

The other two parts stay on 9.0.x only:

  • The Spring Framework 7.1 parts: 8.x is on Boot 4.1.3, which keeps the 8.0.x workarounds this PR removes.
  • The dispatcher type restore: it changes what a filter reads, and it has its upgrade note.

If the completion fix stays on 9.0.x only, the Undertow bullet of the upgrade note needs a sentence for it too: AsyncListener.onComplete now fires for such a request, and with it the completion callbacks of a DeferredResult. As a fix on 8.0.x, it needs no note.

2. One combination is still reported three times on Tomcat. The case: an asynchronous dispatch fails, and its error controller fails with a 503 that Spring MVC resolves. I added these rows to the three apps:

def answeredByControllerFailingWithServerErrorAsynchronously() {
    failAsynchronously(new IllegalMonitorStateException('answered by an error controller that fails with a server error'))
}
'asynchronously'                 | '/failing/answeredByControllerFailingWithServerErrorAsynchronously'

Jetty and Undertow pass. Tomcat gives NOTIFIED == [IllegalMonitorStateException, IllegalMonitorStateException, IllegalMonitorStateException], and the observation stops twice. The handler saw the calls in this order:

  1. onComplete from AsyncContextImpl.doInternalDispatch → recordError, then stop.
  2. onError from CoyoteAdapter.asyncDispatch → setErrorState, because response.isError() after the sendError(503).
  3. onComplete again from AsyncStateMachine.asyncPostProcess → recordError, then stop.

That's the [onComplete, onError, onComplete] from your bz 70362, which any asynchronous dispatch that ends in sendError gets on Tomcat, with or without Grails. This PR doesn't cause it, and Grails can't add a listener during that dispatch to correct it.

I'd still add the row, so the gap is tracked:

  • Add it to the Jetty and Undertow specs.
  • On Tomcat, add it as a separate feature with @PendingFeature(reason = 'https://bz.apache.org/bugzilla/show_bug.cgi?id=70362').

Then the Tomcat upgrade that fixes it turns the feature into a failure that says so. The description should also name this case, since it says the failure is reported once.

The variant where Grails resolves the error controller's failure (IndexOutOfBoundsException, answered by failingErrorHandler after a DeferredResult error) is reported once on all three containers.

Notes

  1. Spring Framework 7.1.1 changes what completion reads. Commit spring-projects/spring-framework@088fb71f, milestone 7.1.1, closed your #37731. With it, ObservationAsyncListener.onComplete reads only what recordHandledException recorded, not jakarta.servlet.error.exception.

    I traced each case against that diff, and each is still reported once. Where Tomcat doesn't report the failure, FirstFailureListener records it itself. Two things go stale with the bump, though:

    • request.removeAttribute(RequestDispatcher.ERROR_EXCEPTION) in FirstFailureListener.onComplete (ServerErrorObservation.groovy:140) is only needed on 7.1.0.
    • The class Javadoc at :70-71 says the filter reads the failure off the request once the exchange completes. For an asynchronous request that no longer holds.

    A comment that points at #37731 would tell whoever bumps Boot what can go.

  2. ServerHttpObservationFilter.recordHandledException(request, null) (ServerErrorObservation.groovy:142) passes null where Spring doesn't allow it. org.springframework.web.filter is @NullMarked, and error isn't @Nullable. It works because setAttribute(name, null) removes the attribute.

    There's no public way to clear it, so I'd keep it. The rows "…when the container reports it and a failure Spring MVC records as handled follows" pin it, so if Spring starts rejecting null, a test fails. The comment could say that the call steps outside Spring's contract, not only that it removes the attribute.

  3. Nit: ServerErrorObservation.groovy:107 is 168 characters long. The rest of the file wraps at 120.

Verified as correct

  • James's two points are fixed. the exception of an action stays the error of the observation when its error controller fails as well and a failure that follows the one an error handler answered does not replace it fail on fdde94a and pass here, with both Grails and Spring MVC resolving.
  • Listener order: FirstFailureListener is added during the dispatch, and the filter adds its own listener once the chain has returned. So FirstFailureListener hears onComplete first, on all three containers, and also after onStartAsync re-adds both.
  • Tomcat reports a failure left on a request whose dispatch started asynchronous processing (CoyoteAdapter.java:366-372). That's the case reportedByContainer catches. After an asynchronous dispatch, Tomcat reports one only when response.isError() (:240-245). A forward to an error controller there sets a status and sends no error, so Tomcat doesn't report it, and recordHandledException alone reports it once.
  • Undertow's forward to a path goes through dispatchToPath (RequestDispatcherImpl.java:186), which never restores the dispatcher type. dispatchToServlet and includes (:347) do restore it.
    • The inner chain wrapper runs inside every dispatch, so nested forwards restore the type of the dispatch they return to.
    • The error dispatch runs after the chain has returned (ServletInitialHandler.java:285), outside the wrapper.
  • AsyncErrorCompletionHandler:
    • handleFirstRequest skips ctx.complete() (:339) because startAsync dispatched the exchange.
    • On the request thread, complete() only flags the processing as completed (AsyncContextImpl.java:264), hence the handleCompletedBeforeInitialRequestDone() call.
    • A dispatch that completes the processing itself is left alone. The third test, which passes on fdde94a too, guards that.
  • Nothing refers to the removed workarounds any more: the private .observation attribute, holdFailureLeftOnRequest and the doService override. holdFailureLeftOnRequest is not in v8.0.0.
  • CI is green apart from the Groovy Snapshot Canary. Its Build Grails shards fail compiling grails-core and grails-async-groovy-concurrent against the Groovy snapshot, modules this PR doesn't touch. The canary of feat/forge-plugin-commands-9.0.x fails in the same four compileGroovy tasks. Two Ubuntu core builds were still running when I wrote this.

@codeconsole

Copy link
Copy Markdown
Contributor Author

Merging since this fixes functional tests. Will create follow up PR for non blocking

@codeconsole

Copy link
Copy Markdown
Contributor Author

@matrei Thanks. #17088 was already merged, so the follow-ups went into new PRs:

  1. The Undertow completion fix for 8.0.x is 8.0 (fix): Complete asynchronous processing on Undertow once an error its request dispatch sent is answered #17100: AsyncErrorCompletionHandler, its factory line and its three tests. On 8.0.x, the two completion tests fail without the handler and all three pass with it. So the 9.0.x upgrade note needs no sentence for it.
  2. 9.0 (fix): Track the asynchronous failure Tomcat reports three times, and note what Spring Framework 7.1.1 makes stale #17101 (9.0.x) adds your row. On Jetty and Undertow it's a row of the "fails with a server error Spring MVC resolves" feature, and it passes. On Tomcat it's a separate feature with @PendingFeature(reason = 'https://bz.apache.org/bugzilla/show_bug.cgi?id=70362'), and it fails on the NOTIFIED assertion as expected. The description of 9.0 (fix): Report a failure an error handler answered once on Spring Framework 7.1 #17088 now names the case under "Limitation".
  3. 9.0 (fix): Track the asynchronous failure Tomcat reports three times, and note what Spring Framework 7.1.1 makes stale #17101 points the ERROR_EXCEPTION removal in FirstFailureListener.onComplete and the recordServerError Javadoc at ServerHttpObservationFilter reports the container's ServletException for failed async requests spring-projects/spring-framework#37731, as what goes with the 7.1.1 bump.
  4. The comment on recordHandledException(request, null) now says the call steps outside Spring's @NullMarked contract, and that ServerErrorObservationSpec fails if Spring starts rejecting null.
  5. The 168-character Javadoc line is wrapped at 120 (9.0 (fix): Track the asynchronous failure Tomcat reports three times, and note what Spring Framework 7.1.1 makes stale #17101).

This was referenced Nov 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

3 participants