Skip to content

Querier: initialise subservicesWatcher in blocksStoreReplicationSet - #7896

Open
yeya24 wants to merge 2 commits into
cortexproject:masterfrom
yeya24:fix/nil-subservices-watcher
Open

yeya24 wants to merge 2 commits into
cortexproject:masterfrom
yeya24:fix/nil-subservices-watcher

Conversation

@yeya24

@yeya24 yeya24 commented Oct 26, 2026

Copy link
Copy Markdown
Contributor

What this PR does:

blocksStoreReplicationSet declares a subservicesWatcher *services.FailureWatcher and calls WatchManager on it from starting(), but newBlocksStoreReplicationSet never assigns it.

WatchManager registers a manager listener whose callback sends on w.ch. Unlike Chan(), it has no nil receiver guard, so the first subservice failure dereferences a nil *FailureWatcher. Because Manager.AddListener runs each listener on a dedicated goroutine, the panic terminates the process rather than returning an error:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0]
goroutine 10 [running]:
...services.(*FailureWatcher).WatchManager.func1(...)
        pkg/util/services/failure_watcher.go:33
...services.funcBasedManagerListener.Failure(...)
        pkg/util/services/manager.go:321
...services.(*Manager).serviceStateChanged.func1(...)
        pkg/util/services/manager.go:185
...services.(*Manager).AddListener.func1()
        pkg/util/services/manager.go:245
created by ...services.(*Manager).AddListener in goroutine 8
        pkg/util/services/manager.go:243

The subservices here are the store-gateway ring and its client pool, so this path is reachable in practice. We hit it on a querier whose store-gateway ring failed to start: the panic's stack trace buried the error logged for the actual cause on the preceding line (module failed module=store-queryable err="..."), making a routine startup failure look like a crash inside Cortex.

The bug is masked at the other end. Chan() does guard a nil receiver and returns a nil channel, so the select in running() blocks forever on it rather than reporting the failure. A subservice failure is therefore either silently dropped or crashes the process, depending on which path observes it first.

Fixed by assigning services.NewFailureWatcher() in the constructor, matching BlocksStoreQueryable (blocks_store_queryable.go) and parquetQueryableWithFallback (parquet_queryable.go), which already do.

Note: Scheduler (pkg/scheduler) and Frontend (pkg/frontend/v1) have the same un-initialised field. I left them out of this PR because their subservices are requestQueue and activeUsers, both TimerServices whose iteration functions return nil unconditionally, so they have no path into Failed today and the nil watcher is never dereferenced. Happy to fix those here too if you'd prefer the pattern be consistent across the package.

Which issue(s) this PR fixes:

Checklist

  • Tests updated — regression test verified to fail before the fix and pass after
  • Documentation added — n/a
  • CHANGELOG.md updated
  • docs/configuration/v1-guarantees.md updated if this PR introduces experimental flags — n/a, no new flags

AI assistance: this change was prepared with the help of an AI coding assistant (investigation, patch and test). I have reviewed and verified it, and I take responsibility for its content, per the project's GenAI contribution policy.

blocksStoreReplicationSet declares a subservicesWatcher field and calls
WatchManager on it from starting(), but newBlocksStoreReplicationSet
never assigns it. WatchManager registers a manager listener whose
callback sends on w.ch, and unlike Chan() it has no nil receiver guard,
so the first subservice failure dereferences a nil *FailureWatcher.
Manager.AddListener runs each listener on its own goroutine, so the
resulting SIGSEGV is not recoverable by the caller and terminates the
process:

    panic: runtime error: invalid memory address or nil pointer dereference
    [signal SIGSEGV: segmentation violation]
    goroutine 10 [running]:
    ...services.(*FailureWatcher).WatchManager.func1(...)
            pkg/util/services/failure_watcher.go:33
    ...services.(*Manager).AddListener.func1()
            pkg/util/services/manager.go:245
    created by ...services.(*Manager).AddListener in goroutine 8
            pkg/util/services/manager.go:243

The subservices here are the store-gateway ring and its client pool, so
the failing path is reachable in practice: when the ring cannot be
started, the panic's stack trace buries the error logged for the actual
cause on the preceding line, making a routine startup failure look like
a crash inside Cortex.

The bug is masked at the other end. Chan() does guard a nil receiver and
returns a nil channel, so the select in running() blocks forever on it
rather than reporting the failure. A subservice failure is therefore
either silently dropped or crashes the process, depending on which path
observes it first.

Assign services.NewFailureWatcher() in the constructor, matching
BlocksStoreQueryable and parquetQueryableWithFallback, which already do.
Adds a regression test asserting the field is non-nil after construction.

Signed-off-by: Ben Ye <benye@amazon.com>
@yeya24
yeya24 requested a review from a team as a code owner October 26, 2026 22:06
@yeya24
yeya24 requested a review from danielblando October 26, 2026 22:06
Signed-off-by trailer added by git commit -s.

Signed-off-by: Ben Ye <benye@amazon.com>

@sandy2008 sandy2008 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. The new test fails without the fix, and with a ring whose KV Get fails at startup, the nil-pointer panic in the listener goroutine turns into a clean unable to start blocks store set subservices: ... error.

The test (arm64) failure is the TestPrometheusMetrics_IncrementedCorrectly flake in pkg/util/queryeviction, unrelated to this change.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants