Repository navigation
Conversation
blocksStoreReplicationSet declares a subservicesWatcher field and calls
WatchManager on it from starting(), but newBlocksStoreReplicationSet
never assigns it. WatchManager registers a manager listener whose
callback sends on w.ch, and unlike Chan() it has no nil receiver guard,
so the first subservice failure dereferences a nil *FailureWatcher.
Manager.AddListener runs each listener on its own goroutine, so the
resulting SIGSEGV is not recoverable by the caller and terminates the
process:
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation]
goroutine 10 [running]:
...services.(*FailureWatcher).WatchManager.func1(...)
pkg/util/services/failure_watcher.go:33
...services.(*Manager).AddListener.func1()
pkg/util/services/manager.go:245
created by ...services.(*Manager).AddListener in goroutine 8
pkg/util/services/manager.go:243
The subservices here are the store-gateway ring and its client pool, so
the failing path is reachable in practice: when the ring cannot be
started, the panic's stack trace buries the error logged for the actual
cause on the preceding line, making a routine startup failure look like
a crash inside Cortex.
The bug is masked at the other end. Chan() does guard a nil receiver and
returns a nil channel, so the select in running() blocks forever on it
rather than reporting the failure. A subservice failure is therefore
either silently dropped or crashes the process, depending on which path
observes it first.
Assign services.NewFailureWatcher() in the constructor, matching
BlocksStoreQueryable and parquetQueryableWithFallback, which already do.
Adds a regression test asserting the field is non-nil after construction.
Signed-off-by: Ben Ye <benye@amazon.com>
Signed-off-by trailer added by git commit -s. Signed-off-by: Ben Ye <benye@amazon.com>
sandy2008
approved these changes
Oct 27, 2026
sandy2008
left a comment
Contributor
There was a problem hiding this comment.
LGTM. The new test fails without the fix, and with a ring whose KV Get fails at startup, the nil-pointer panic in the listener goroutine turns into a clean unable to start blocks store set subservices: ... error.
The test (arm64) failure is the TestPrometheusMetrics_IncrementedCorrectly flake in pkg/util/queryeviction, unrelated to this change.
SungJin1212
approved these changes
Oct 27, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does:
blocksStoreReplicationSetdeclares asubservicesWatcher *services.FailureWatcherand callsWatchManageron it fromstarting(), butnewBlocksStoreReplicationSetnever assigns it.WatchManagerregisters a manager listener whose callback sends onw.ch. UnlikeChan(), it has no nil receiver guard, so the first subservice failure dereferences a nil*FailureWatcher. BecauseManager.AddListenerruns each listener on a dedicated goroutine, the panic terminates the process rather than returning an error:The subservices here are the store-gateway ring and its client pool, so this path is reachable in practice. We hit it on a querier whose store-gateway ring failed to start: the panic's stack trace buried the error logged for the actual cause on the preceding line (
module failed module=store-queryable err="..."), making a routine startup failure look like a crash inside Cortex.The bug is masked at the other end.
Chan()does guard a nil receiver and returns a nil channel, so theselectinrunning()blocks forever on it rather than reporting the failure. A subservice failure is therefore either silently dropped or crashes the process, depending on which path observes it first.Fixed by assigning
services.NewFailureWatcher()in the constructor, matchingBlocksStoreQueryable(blocks_store_queryable.go) andparquetQueryableWithFallback(parquet_queryable.go), which already do.Note:
Scheduler(pkg/scheduler) andFrontend(pkg/frontend/v1) have the same un-initialised field. I left them out of this PR because their subservices arerequestQueueandactiveUsers, bothTimerServices whose iteration functions returnnilunconditionally, so they have no path intoFailedtoday and the nil watcher is never dereferenced. Happy to fix those here too if you'd prefer the pattern be consistent across the package.Which issue(s) this PR fixes:
Checklist
CHANGELOG.mdupdateddocs/configuration/v1-guarantees.mdupdated if this PR introduces experimental flags — n/a, no new flagsAI assistance: this change was prepared with the help of an AI coding assistant (investigation, patch and test). I have reviewed and verified it, and I take responsibility for its content, per the project's GenAI contribution policy.