gprecoverseg: fix the defects the Behave suite exposes by MisterRaindrop · Pull Request #2183 · apache/cloudberry

MisterRaindrop · 2026-10-24T17:25:30Z

What does this PR do?

Fixes the gprecoverseg defects that the gpMgmt Behave suite (#2170) exposes, and brings back the two Greenplum guards whose absence makes gprecoverseg and gpstate misreport recoveries. Four commits; each commit message gives the full reasoning.

gprecoverseg: fix the defects the Behave suite exposes.
- A recovered segment now gets its own port in postgresql.conf before it starts. pg_basebackup and pg_rewind copy the primary's.
- The pg_basebackup/pg_rewind progress files a recovery writes are kept, and stale ones for the segments about to be recovered are removed first. Before, the files were deleted on success, and gpstate needs them.
- -F combined with -r or -p, and -o combined with -i or -r, are rejected.
- The "gprecoverseg failed" error and the note about the progress files left behind are logged again.
- A failed segment's summary line says why it failed, read back from its log.
gpstate: stop reporting recovery progress for a gprecoverseg that died. gpstate -e treated a leftover gprecoverseg.lock as a recovery in progress, so after a SIGKILL it reported "Segments in recovery" forever. It now checks whether the PID in the lock is alive, as Greenplum does.
gprecoverseg: skip segments that already have a recovery running. A second pg_basebackup or pg_rewind into the same segment leaves it inconsistent. Greenplum's check against gp_stat_replication and pg_stat_activity is ported.
gprecoverseg: accept a hostname in the -i recovery config file. Greenplum's hostname|address|port|datadir form is ported, with the hostname cross-checked against the address. Unlike upstream, the check resolves through getaddrinfo(), because upstream's gethostbyaddr() comparison calls localhost and 127.0.0.1 two different hosts on a dual-stack box.

Split out of #1639 (originally a3bfe47, 879b563, 7056e02 and 0aa2666, by @tuhaihe).

Changes from #1639, found by running the unit tests, which #1639 did not run:

The stale progress files were never removed. remove_existing_progress_files() came over from Greenplum, but Greenplum's matching change to _get_remove_cmd() did not. The pattern went into rm -f '*dbid5.out', quoted, and ran in the remote user's home directory, so nothing was removed. The remove command is now Greenplum's find <log dir> -name <pattern> -delete. I checked this on a cluster: I put an old pg_basebackup.20000101_000000.dbid5.out in gpAdminLogs, failed that mirror, and ran gprecoverseg -aF. With CI: add gpMgmt Behave test #1639's code both the old file and the new one were there afterwards; with this PR only the new one is.
Seven test_unit_buildMirrorSegments tests still asserted the old delete-on-success call. They now assert the single recovery call, and a test for remove_existing_progress_files() is added.
Four test_unit_gpsegrecovery tests failed on the line the port rewrite logs. The rewrite was not mocked, so those tests and the workflow tests ran a real perl -i on a postgresql.conf that does not exist; perl only warns about that, so nothing caught it. The tests now mock ModifyConfSetting, assert the port it writes, and expect the log line.
test_is_gprecoverseg_running_succeeds passed a bare MagicMock to os.path.join, which the mock version in requirements-dev.txt cannot turn into a path. It now gives the mock a path and asserts the PID file it opens.

Why one PR for four fixes: #1639 carried 19 fixes. Opening one PR per fix would leave a long queue of tiny PRs for reviewers, so the remaining ones are grouped by area into five PRs (this one, #2178, #2180, #2182 (gpcheckcat) and #2184 (gprecoverseg Greenplum ports), which builds on this one). The commits are kept separate on purpose, so please merge this with "Rebase and merge" rather than squash.

Test Plan

Rocky 9 build image, built from #2170's tree plus #2177, #2178 and #2180 (which the demo cluster and gpstop need), with a demo cluster created the way the Behave workflow creates it.

I removed the @needs_fix tags that point at these commits, including the ones on the scenarios that commits 3 and 4 make runnable.
I then ran each feature with CI's arguments.

Run A is without this PR; run B is with it installed into $GPHOME.

feature (CI arguments)	A: without	B: with this PR
`gpmovemirrors.feature` (`--tags ~@concourse_cluster --tags ~@extended`)	10 passed, 7 failed	17 passed, 0 failed
`gpaddmirrors.feature` (`--tags ~@concourse_cluster`)	10 passed, 2 failed	12 passed, 0 failed
`gprecoverseg.feature` (`--tags ~@concourse_cluster --tags ~@extended --tags ~@differential --tags ~@not_implemented`)	timed out after 60 minutes in its 10th scenario; 4 of the first 9 failed	28 passed, 0 failed

Without this PR, most failures come from the port defect: a recovered mirror keeps its primary's port in postgresql.conf.

gpaddmirrors: both scenarios fail on port value in postgresql.conf ... is incorrect.
gprecoverseg: three scenarios fail on that check. A fourth fails because the mirror cannot start on a port its primary already holds. The tenth scenario then waits forever for a cluster whose mirrors never came back. This is the hang that made CI: run the gpMgmt Behave suite #2170 skip gprecoverseg.feature as a whole.
gpmovemirrors: "can change the location / port of mirrors within a single host", the four examples of "user can rerun gpmovemirrors / run gprecoverseg if mirrors failed to move initially", and "can move mirrors even if start fails for some mirrors". Two fail on the port check, and one on the missing "gprecoverseg failed" line that gpmovemirrors scrapes. The other four find segments still down (all segments are not currently running).

The scenarios skipped in B are the multi-host (@concourse_cluster), @extended and @differential ones, which CI skips too, plus the five that #2184 makes runnable.

Unit tests (python3 -m unittest, with PYTHONPATH set the way make -C gpMgmt/bin unitdevel sets it): test_unit_buildMirrorSegments, test_unit_gpsegrecovery and test_unit_gp pass. Across the whole gppylib suite (690 tests without this PR, 693 with it), the same tests fail with and without this PR. One of them covers code this PR touches: test_gprecoverseg_with_mirrorless errors in both runs because its mock gpArray has no standbyCoordinator. I have not looked into the rest.

#2170 tags the scenarios these commits fix or make runnable @needs_fix, each pointing at its #1639 commit. Whichever PR lands second removes those tags.

Type of Change

Bug fix (non-breaking change)

Checklist

This PR contains AI-assisted code generation

leborchuk

LGTM, agreed with all the fixes and motivation

Alena0704 · 2026-10-29T08:22:26Z

Port rewriting can corrupt a valid hexadecimal setting - gp.py:867.
PostgreSQL accepts port = '0x9c40' as port 40000, but rewriting it to 50000 produces port=50000x9c40'. The command reports success, while PostgreSQL rejects the resulting configuration—even with -p 50000.
The regex matches only the initial decimal digits. The proposed patch fixes this by replacing the complete value and adds regression coverage for hexadecimal, signed, octal, and quoted scientific notation.
fix-numeric-port-rewrite.patch

MisterRaindrop · 2026-10-29T12:02:48Z

Thanks, confirmed, and applied your patch as is in 1bedf80, credited in the commit message.

On a demo cluster I wrote content 0's primary port as port = '0x1b5a' (7002) and ran gprecoverseg -F on its mirror. With this PR's previous code, the mirror's postgresql.conf came back as port=7005x1b5a' and gprecoverseg failed. With your patch the line is port=7005, gprecoverseg succeeds and the mirror is back in sync.

Rewriting a copy of the conf to 50000 and asking postgres -C port: '0x9c40', 0x9c40 and '4e4' used to give a file PostgreSQL rejects, and +40000 was left unchanged at 40000. Now every form, 0116100 and 40000.0 included, reads back as 50000, and port_range is still left alone. Your new tests fail 9 times against the previous code, and test_unit_gpsegrecovery.py (26) and test_unit_gp.py (19) all pass now.

Alena0704

I suggest two changes: handle CRLF line endings when rewriting the port, and prevent gpstate -e from failing when the PID file is empty.

Defects the Behave suite exposes in gprecoverseg: * gprecoverseg: point a recovered segment's conf at its own port pg_basebackup and pg_rewind both copy the source segment's postgresql.conf, port setting included, so a mirror recovered by gprecoverseg came back configured for its primary's port. On a multi-host cluster the wrong port is merely wrong; on a single host it collides with the primary that is still running there. Rewrite the port before starting the segment, on both the full and the incremental path, using the existing ModifyConfSetting command. Greenplum does the same in the same two places. ModifyConfSetting's perl expression only matched "port=N" and "port = N". postgresql.conf also accepts leading whitespace, tabs, an omitted '=', a quoted value and the name in any case, and a line in any of those forms kept the primary's port while the command still succeeded. An integer setting may also be written signed, octal, hexadecimal or, quoted, with a fraction or an exponent, and rewriting only its leading digits left the rest behind: port = '0x1b5a' became port=7005x1b5a', which the segment cannot start with. Replace the whole value, up to trailing whitespace, a comment, or the carriage return of a CRLF line ending, which PostgreSQL also accepts. The match stays anchored at the start of the line and needs '=' or whitespace after the name, so a commented-out port, or a setting whose name only begins with "port", is left alone. Alena Rybakina contributed the whole-value match, the CRLF handling and their unit tests in review. Its other caller, gpconfigurenewsegment for gpexpand, gets the same wider match. The gpsegrecovery unit tests mock the rewrite, assert it, and expect the line it logs; before, they ran perl on a postgresql.conf that does not exist. A new test runs the real rewrite on each of those forms, since only the file shows whether a line matched. Fixes the eight gprecoverseg Behave scenarios that assert "port value in postgresql.conf of <host> is incorrect". * gprecoverseg: keep the progress files a recovery writes Each recovering segment writes pg_basebackup or pg_rewind output to gpAdminLogs/<tool>.<timestamp>.dbid<N>.out, and gprecoverseg deleted that file as soon as the segment came back successfully. Deleting it on success is backwards: those files are what gpstate reads to report recovery progress, and what an operator reads afterwards to see what a recovery actually did. Remove stale progress files for the segments about to be recovered before starting, and leave the run's own files in place, the way Greenplum does. Clearing beforehand is also what makes the file a reliable record of the current run rather than an append of every run that came before. The stale files are matched by name ("*dbid<N>.out"), so the remove command becomes Greenplum's "find <log dir> -name <pattern> -delete". The old "rm -f" form would have quoted the pattern and run it in the remote user's home directory, removing nothing. Its unit tests now assert the single recovery call and the find commands. Fixes the Behave scenarios asserting gpAdminLogs holds "pg_basebackup*" and "pg_rewind*" files after a recovery. * gprecoverseg: reject -F combined with -r or -p A full resynchronisation rebuilds a mirror from its primary in place. That is not what -r asks for -- rebalance returns segments to their preferred roles -- nor what -p asks for, which is to rebuild onto a different host. gprecoverseg accepted both combinations and silently did something other than what was asked. Reject them, with the messages Greenplum uses. * gprecoverseg: report the failure and the progress files it leaves Two log lines Greenplum emits around recover_mirrors() were dropped: - the [ERROR] 'gprecoverseg failed. Please check the output for more details.' that follows a failed recovery. Without it a failed run ends on the per-segment errors alone, and callers that scrape the output -- gpmovemirrors among them -- cannot tell a failure from a partial success. - the notice that the pg_basebackup/pg_rewind progress files this run wrote are left behind and a later gprecoverseg may remove them. Asserted by gpmovemirrors.feature:136 and gprecoverseg.feature:58. * gprecoverseg: reject -o combined with -i or -r -o writes out a sample recovery configuration describing what would be recovered and recovers nothing. -i names a configuration to recover from and -r rebalances back to preferred roles, so either one together with -o is a contradiction: the user asked for a file and for a recovery at the same time, and gprecoverseg quietly honours one of them. Upstream rejects both combinations; the checks live in a method whose other validations are for options this tree does not have, so take just these two and put them beside the "Only one of -i, -p, and -r" check they belong with. They run before the cluster is contacted, so the error is immediate. gprecoverseg.feature has been asserting this and failing. * gprecoverseg: say why a segment failed to recover, not just where When a recovery fails, the summary names the segment and the log file and stops there: Failed to recover the following segments. You must run gprecoverseg -F for all incremental failures hostname: sdw1; port: 7000; logfile: ~/gpAdminLogs/pg_rewind...out; recoverytype: incremental so the one thing the reader wants -- what went wrong -- is the one thing they have to go and find. Upstream ends that line with "; error: {}" and fills it by reading the log back, and the printer for setup-recovery failures in this same file already does the equivalent. Only this one lost it. Bring over get_error_from_logfile, which takes the last ERROR, PANIC or FATAL line from the segment's log. It deliberately skips the line pg_rewind writes about itself after a failure, which is never the reason for it; upstream skips PostgreSQL 12's wording, and this tree needs its own, because pg_rewind.c has since been split into pg_log_error("postgres single-user mode in target cluster failed"); pg_log_error_detail("Command was: %s", ...); With the self-referential line out of the way what is left is the backend's own complaint, which is what a reader needs: for a mirror whose pg_wal was removed, 'required WAL directory "pg_wal" does not exist'. The behave step that checks this summary matches up to "recoverytype: X" with a regex search, so the added field does not disturb it. Split out of apache#1639 (originally a3bfe47).

gpstate -e decides a recovery is in progress by asking whether $COORDINATOR_DATA_DIRECTORY/gprecoverseg.lock exists. gprecoverseg removes that directory when it exits normally, but a process killed with SIGKILL never gets the chance, so from then on every gpstate -e reports Segments in recovery for a recovery that is not happening, and exits 1 while doing so. Upstream asks a different question: it reads the PID out of the lock directory and checks whether that process is still alive. Bring over is_gprecoverseg_running and use it, with the two unit tests that cover the pid-is-alive and no-pidfile cases. Unlike upstream, parse the PID first and take an empty, malformed or non-positive one to mean no recovery is running. The file is empty for a moment while gprecoverseg writes it, and passing such contents on made gpstate -e fail with "invalid literal for int() with base 10: ''" until check_pid() learned to parse its argument; a negative PID would still look alive, since os.kill() takes it as a process group. Alena Rybakina contributed this check and its unit tests in review. Found by gprecoverseg.feature:735 "SIGKILL on gprecoverseg should not display progress in gpstate -e" on a five-node cluster, where it was the first of 38 failures -- the other 37 were scenarios that inherited the half-recovered cluster it left behind. The get_coordinatordatadir import existed only for the directory check and is now unused, as it is upstream. Split out of apache#1639 (originally 879b563).

There must be one writer per target segment. If a pg_basebackup or a pg_rewind is already streaming into a segment and gprecoverseg starts a second one, the two races leave the segment inconsistent. Greenplum guards against this by asking gp_stat_replication which segments are already sourcing a pg_basebackup, and pg_stat_activity whether a pg_rewind is connected, then skipping those and saying so. Cloudberry carried over neither check, so a second recovery would just start. Port both. gpmovemirrors needs no change of its own: it moves mirrors by shelling out to gprecoverseg, which is why its three scenarios assert on what gprecoverseg printed. This lets seven scenarios run instead of being skipped -- four in gprecoverseg.feature and three in gpmovemirrors.feature, all of the form "gives warning if pg_basebackup already running for ... segments". cherry-pick from Greenplum Database, greenplum-db/gpdb-archive @ 482967c1b4, gpMgmt/bin/gppylib/programs/clsRecoverSegment_triples.py. Two deliberate departures from upstream. Greenplum skips a third case, a source segment already in backup mode, which only arises from its differential recovery. Cloudberry has no differential recovery, so that arm and its pg_is_in_backup() dependency are left out. And the skip for a failed segment on an unreachable host is taken before either check: is_pg_rewind_running() connects to the peer, and on a double fault that connection raises and would abort the whole run rather than skip the one segment. All three skips, like upstream's, come before the segment registered in gpArray is rewritten for a move. updateSystemConfig() writes every segment changed there to the catalog, so a segment skipped after that point is recorded at a location nothing was recovered to. Checked on a demo cluster with a pg_basebackup streaming from content 0's primary and "gprecoverseg -i" moving the mirrors of contents 0 and 1: with the skip after the rewrite, content 0 was skipped but gp_segment_configuration named its new, nonexistent data directory and port; with it before, content 0 keeps its old location and a later gprecoverseg recovers it. test_cluster_clsrecoversegment_triples.py mocks both checks -- unmocked, they tried to connect to the hosts in the test arrays -- and takes upstream's cases for them, plus cases where the skipped segment is being moved by -i or to a new host. run_pass_tests() compares gpArray with what the expected triplets alone would make of it, so those cases fail if a skipped segment is changed. Split out of apache#1639 (originally 7056e02).

A segment has both a hostname and an address, and they are not always the same -- the address is whichever interface the segment listens on, and a host with several interfaces has several of them. gprecoverseg's -i config file only understood "address|port|datadir", so on such a cluster it had to guess the hostname from the address, and a recovery to a new host could register the segment under the wrong one. Greenplum lets the line carry the hostname explicitly, as "hostname|address|port|datadir", and when it does: - the hostname is cross-checked against the address, by resolving both and requiring a common IP, so a typo is caught before anything is recovered rather than after; - the failed segment is matched on hostname as well, so a line naming the wrong host no longer silently recovers the right one; - the error naming an unfound segment quotes all four fields. Port that. Both groups on a line must use the same number of fields, which is what the "expected equal parts" error says. cherry-pick from Greenplum Database, greenplum-db/gpdb-archive @ 482967c1b4: gpMgmt/bin/gppylib/programs/clsRecoverSegment_triples.py gpMgmt/bin/gppylib/utils.py This lets nine scenarios run instead of being skipped, the "populates hostname and address from the config file correctly" group and the two that assert the cross-check warning. Two deliberate departures from upstream. Greenplum also accepts a fifth field holding a per-line recovery type (I/D/F), which exists for its differential recovery; Cloudberry has no differential recovery and no recovery-type field, so only the 3- and 4-part forms are accepted and the error message says so. And validateHostnameAddress() calls socket.gethostbyaddr() rather than importing the bare name, because gppylib/utils.py already imports the socket module. One further departure is a fix rather than a trim. Upstream resolves both names with gethostbyaddr() and requires their address lists to intersect. gethostbyaddr() answers for a single address family, so on a dual-stack host "localhost" comes back as ::1 while an address naming that same machine comes back as 127.0.0.1; the lists do not intersect and the check reports two different hosts. The consequence is not a failure but a silence -- the segment is skipped and gprecoverseg still exits 0, having recovered nothing. Resolve through getaddrinfo() instead and take every address of each name, so the comparison no longer depends on which family the resolver happens to answer with first. Observed on the box this was tested on: localhost -> 127.0.0.1, ::1 cdw -> 127.0.0.1 which upstream's check calls a mismatch. Greenplum also reworked how an unreachable failover host is detected while it was here, probing the hosts in parallel through a new _get_unreachable_failover_hosts() and threading paralleldegree and outputConfigFile down for it. That is a separate concern from reading a hostname out of the config file, and Cloudberry already has its own unreachable handling that the suite exercises, so this keeps that and leaves the rework alone. The -o sample config file still writes three fields, as upstream does. The parser tests in test_cluster_clsrecoversegment_triples.py expect the hostname fields and the new messages, and take upstream's four-part cases. Split out of apache#1639 (originally 0aa2666).

MisterRaindrop · 2026-11-05T03:43:23Z

I suggest two changes: handle CRLF line endings when rewriting the port, and prevent gpstate -e from failing when the PID file is empty.

Thanks, both confirmed, and all six suggestions applied as is. The port change is folded into 8cf4546 and the PID change into 90bf1a9, with you credited in both commit messages.

CRLF. On a demo cluster, PostgreSQL reads a CRLF postgresql.conf without complaint (postgres -C port gives 40000), but the rewrite left port = 40000 and PORT = '40000' \t untouched; only a line with a trailing comment matched. End to end, with content 0's primary conf converted to CRLF and its port line a plain port=7002, gprecoverseg -F succeeded and the recovered mirror's conf still said port=7002. With \r?$, each of those lines becomes port=50000 with its \r\n kept, and the same recovery leaves port=7005. This is a regression this PR would have introduced: main's expression has no end anchor, so it never cared about the line ending.

Empty PID file. With gprecoverseg.lock/PID holding '', ' ' or not-a-pid, gpstate -e exited 2 with gpstate failed. (Reason='invalid literal for int() with base 10: '''). With your change, all three, plus 0, -1 and a dead PID, report normally. Now that the branch is on main, check_pid() already parses its argument (since #2180), so the empty file alone would no longer fail. Your check still covers -1, which os.kill() treats as "every process" and so reports as alive, and it no longer depends on how check_pid() parses.

The new tests pass, and against the old gp.py they fail: the CRLF test, the updated succeeds assertion, and every invalid-PID subtest.

Rebased onto main (thanks @tuhaihe for the earlier rebase; the patches are the same apart from what follows). This push also removes the @needs_fix tags that this PR's fixes cover: 2 in gpaddmirrors.feature, 7 in gpmovemirrors.feature, and 13 scenarios plus the file-level tag in gprecoverseg.feature. The 5 left in gprecoverseg.feature belong to #2184. With CI's behave arguments, all three features pass on a demo cluster: gpaddmirrors 12/0, gpmovemirrors 17/0, gprecoverseg 28/0.

Alena0704 requested review from Alena0704 and tuhaihe October 24, 2026 17:52

leborchuk approved these changes Oct 26, 2026

View reviewed changes

Alena0704 reviewed Oct 26, 2026

View reviewed changes

Comment thread gpMgmt/bin/gppylib/programs/clsRecoverSegment_triples.py Resolved

Comment thread gpMgmt/sbin/gpsegrecovery.py Resolved

MisterRaindrop force-pushed the fix/gprecoverseg-behave-defects branch from d7665a6 to 64c8de4 Compare October 28, 2026 04:34

Alena0704 reviewed Oct 28, 2026

View reviewed changes

Comment thread gpMgmt/bin/gppylib/commands/gp.py Resolved

MisterRaindrop force-pushed the fix/gprecoverseg-behave-defects branch from 64c8de4 to b66af61 Compare October 28, 2026 11:46

Alena0704 reviewed Oct 28, 2026

View reviewed changes

Comment thread gpMgmt/bin/gppylib/programs/clsRecoverSegment_triples.py Resolved

MisterRaindrop force-pushed the fix/gprecoverseg-behave-defects branch from b66af61 to b2115c6 Compare October 29, 2026 06:45

MisterRaindrop requested a review from yjhjstz October 29, 2026 07:52

tuhaihe added this to the Behave tests milestone Oct 29, 2026

MisterRaindrop force-pushed the fix/gprecoverseg-behave-defects branch from b2115c6 to 75f1ccb Compare October 29, 2026 12:02

Alena0704 reviewed Oct 29, 2026

View reviewed changes

MisterRaindrop force-pushed the fix/gprecoverseg-behave-defects branch from 75f1ccb to d867a98 Compare November 3, 2026 08:20

tuhaihe approved these changes Nov 4, 2026

View reviewed changes

tuhaihe enabled auto-merge (rebase) November 4, 2026 09:52

tuhaihe disabled auto-merge November 5, 2026 03:32

tuhaihe added 4 commits November 5, 2026 11:42

MisterRaindrop force-pushed the fix/gprecoverseg-behave-defects branch from fdb4bf9 to 3f83cf1 Compare November 5, 2026 03:42

tuhaihe enabled auto-merge (rebase) November 5, 2026 03:47

tuhaihe merged commit 8101834 into apache:main Nov 5, 2026
143 checks passed

Navigation Menu

Repository navigation

gprecoverseg: fix the defects the Behave suite exposes - #2183

gprecoverseg: fix the defects the Behave suite exposes#2183
tuhaihe merged 4 commits into
apache:mainfrom
MisterRaindrop:fix/gprecoverseg-behave-defects

MisterRaindrop commented Oct 24, 2026 •

edited

Loading

Uh oh!

leborchuk left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Alena0704 commented Oct 29, 2026

Uh oh!

MisterRaindrop commented Oct 29, 2026

Uh oh!

Alena0704 left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

MisterRaindrop commented Nov 5, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

MisterRaindrop commented Oct 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Test Plan

Type of Change

Checklist

Uh oh!

leborchuk left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Alena0704 commented Oct 29, 2026

Uh oh!

MisterRaindrop commented Oct 29, 2026

Uh oh!

Alena0704 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

MisterRaindrop commented Nov 5, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

MisterRaindrop commented Oct 24, 2026 •

edited

Loading