Skip to content

gpinitsystem: fix the defects the Behave suite exposes - #2178

Merged
tuhaihe merged 1 commit into
apache:mainfrom
MisterRaindrop:fix/gpinitsystem-behave-defects
Nov 3, 2026
Merged

tuhaihe merged 1 commit into
apache:mainfrom
MisterRaindrop:fix/gpinitsystem-behave-defects

Conversation

@MisterRaindrop

Copy link
Copy Markdown
Contributor

What does this PR do?

Fixes six gpinitsystem defects that the gpMgmt Behave suite (#2170) exposes. Each is a line or two that Cloudberry lost relative to Greenplum. They are kept together because they share gpinitsystem / gp_bash_functions.sh and the same feature file; the commit message gives each one's full reasoning.

  1. Mirror port after pg_basebackup. CREATE_QES_MIRROR started the mirror before fixing the port in the postgresql.conf that pg_basebackup copied from its primary, so on a single host the mirror bound its primary's port, or collided with it. The port is now rewritten before the start. This change also writes explicit 127.0.0.1/32 and ::1/128 trust entries into the coordinator's pg_hba.conf, because a hostname entry alone breaks when the client picks ::1.
  2. FORCE_FTS_PROBE never asked for a probe. The gp_request_fts_probe_scan() call was missing, so the 60 iterations re-read an unchanged catalog in a couple of seconds and gave up. gpinitsystem then exited 0 with all mirrors still down.
  3. The default ETCD/FTS config was logged as [WARN]. gpinitsystem's own log scan made every run end with "warnings or errors were generated". These messages are now logged at INFO.
  4. gpinitstandby got a bare -f. On any multi-host cluster without external FTS hosts, gpinitsystem passed -f with no value, gpinitstandby failed parsing its options, and gpinitsystem still exited 0 with no standby. -f is now passed only when there is a value, and the list builder's always-false [ -z FTS_HOST_STRING_LIST ] is fixed. This was first seen on a five-node EC2 cluster.
  5. [ERROR] did not set the exit status in LOG_MSG. As a result, gpinitsystem --unknown-option exited 0. The scenario that should have caught this was itself passing --ignore-warnings, which is not a gpinitsystem option. That stray argument is dropped, and a scenario for the exit status is added.
  6. SED_PG_CONF substitutions were unanchored. gpinitsystem -p with work_mem=999 rewrote maintenance_work_mem. The anchors are restored. CHECK_FTS's argument-count guard (copied from CHECK_ETCD) and CLEANUP_ETCD's error message are also corrected, because once fix 5 lands, the wrong guard would make gpinitsystem -F report a failure after it succeeded.

Split out of #1639 (originally b971d48, by @tuhaihe).

Test Plan

Rocky 9 build image, built from #2170's tree plus #2177 (767f12685a4, absolute TRUSTED_SHELL), with a demo cluster created the way the Behave workflow creates it. I removed the @needs_fix tags that point at either fix, then ran each feature with CI's arguments. Run A is without this PR; run B is with it, with the three scripts installed into $GPHOME/bin.

A: without B: with this PR
gpinitsystem.feature (--tags ~@extended) 17 passed, 16 failed 33 passed, 0 failed
gpaddmirrors.feature (--tags ~@concourse_cluster) 10 passed, 2 failed 10 passed, 2 failed (same two; not this fix)

Why the scenarios fail in A:

  • "exits with status 1 when the user provides invalid option" (the new scenario) exits 0. Fix 5.
  • The two "should warn but not fail when standby cannot be instantiated" scenarios fail with "all segments are not currently running". Fix 2.
  • "should print FQDN in pg_hba.conf when HBA_HOSTNAMES=1": gpinitsystem's own gpstop is refused with no pg_hba.conf entry for host "::1". That is the loopback part of fix 1.
  • The 12 scenarios after it fail because each one starts by stopping that cluster. gpstop is refused the same way, and the Behave suite's pg_ctl fallback raises NameError (fixed in CI: run the gpMgmt Behave suite #2170), so all of them start from a cluster that can't be stopped.
    • The run I did for gpdemo: write an absolute TRUSTED_SHELL into the generated config #2177 applied only that fix and still skipped every scenario up to and including the DCA one. In it, the locale, banner, -D and both trusted_shell scenarios passed.
    • "consistent port entry on segments postgresql.conf" was the exception: it failed with expected 15437, got 15434, which fix 1 addresses.
    • I have no clean result without this PR for the HBA_HOSTNAMES=0, standby HBA_HOSTNAMES=1 and DCA scenarios. They pass with it.

#2170 tags these scenarios @needs_fix pointing at this change. Whichever PR lands second removes those tags.

Type of Change

  • Bug fix (non-breaking change)

Checklist

  • This PR contains AI-assisted code generation

@leborchuk leborchuk left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, old code just didn't work with some options. Of course we should fix it.

@Alena0704 Alena0704 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. I suggest adding regression test for SED_PG_CONF, particularly preserving maintenance_work_mem when changing work_mem.

test
diff --git a/gpMgmt/test/behave/mgmt_utils/gp_bash_functions.feature b/gpMgmt/test/behave/mgmt_utils/gp_bash_functions.feature
index 35e6629020e..bfecbbc4a89 100644
--- a/gpMgmt/test/behave/mgmt_utils/gp_bash_functions.feature
+++ b/gpMgmt/test/behave/mgmt_utils/gp_bash_functions.feature
@@ -1,5 +1,16 @@
 @gp_bash_functions.sh
 Feature: gp_bash_funtions.sh unit test cases
+    @sed_pg_conf
+    Scenario Outline: SED_PG_CONF changes only the requested parameter
+        Then SED_PG_CONF preserves unrelated settings on "<host>" with KEEP_PREV "<keep_prev>"
+
+        Examples:
+            | host      | keep_prev |
+            | local     | 0         |
+            | local     | 1         |
+            | localhost | 0         |
+            | localhost | 1         |
+
     Scenario: REMOTE_EXECUTE_AND_GET_OUTPUT returns proper output when no banner is set
         Given a standard local demo cluster is running
         Then source gp_bash_functions and run simple echo
diff --git a/gpMgmt/test/behave/mgmt_utils/steps/gp_bash_functions.py b/gpMgmt/test/behave/mgmt_utils/steps/gp_bash_functions.py
new file mode 100644
index 00000000000..084afc01c87
--- /dev/null
+++ b/gpMgmt/test/behave/mgmt_utils/steps/gp_bash_functions.py
@@ -0,0 +1,48 @@
+import os
+from pathlib import Path
+import subprocess
+import tempfile
+
+from behave import then
+
+
+@then('SED_PG_CONF preserves unrelated settings on "{host}" with KEEP_PREV "{keep_prev}"')
+def check_sed_pg_conf_parameter_boundary(context, host, keep_prev):
+    # Use an active work_mem setting so the replacement branch is exercised;
+    # a file containing only the default #work_mem would take the append path.
+    unrelated_lines = [
+        "maintenance_work_mem = 65536",
+        "# work_mem should not change inside a comment",
+    ]
+    original = "  work_mem = 4096\n" + "\n".join(unrelated_lines) + "\n"
+    functions = Path(os.environ["GPHOME"]) / "bin/lib/gp_bash_functions.sh"
+
+    # localhost uses the real SSH branch against the same temporary file.
+    # No database is required; SSH must be configured as for the other tests
+    # in gp_bash_functions.feature.
+    with tempfile.TemporaryDirectory(prefix="sed_pg_conf_") as temp_dir:
+        config = Path(temp_dir) / "postgresql.conf"
+        config.write_text(original)
+        script = '''
+source "$1" || exit $?
+LOG_FILE="$2"
+SED_PG_CONF "$3" work_mem work_mem=999 "$4" "$5"
+'''
+        result = subprocess.run(
+            ["bash", "-c", script, "sed_pg_conf_test", str(functions),
+             str(Path(temp_dir) / "test.log"), str(config), keep_prev,
+             "" if host == "local" else host],
+            stdout=subprocess.PIPE, stderr=subprocess.PIPE,
+            universal_newlines=True, timeout=60,
+        )
+        assert result.returncode == 0, (
+            "SED_PG_CONF failed: stdout={!r}, stderr={!r}".format(
+                result.stdout, result.stderr)
+        )
+        lines = config.read_text().splitlines()
+        assert lines[1:] == unrelated_lines, (
+            "SED_PG_CONF modified unrelated settings or comments: {!r}".format(lines)
+        )
+        assert lines[0].split("#", 1)[0].strip() == "work_mem=999", (
+            "SED_PG_CONF did not update work_mem: {!r}".format(lines)
+        )

@MisterRaindrop
MisterRaindrop force-pushed the fix/gpinitsystem-behave-defects branch from 5ed5bf5 to 8752fe3 Compare October 28, 2026 11:46
@MisterRaindrop

Copy link
Copy Markdown
Contributor Author

Thanks, added your scenario as is in 8752fe3, credited in the commit message. All four examples fail against main's gp_bash_functions.sh, where maintenance_work_mem and the comment are both rewritten, and pass with this PR.

@Alena0704

Copy link
Copy Markdown
Collaborator

No more issues. LGTM. Thank you!)

@tuhaihe tuhaihe added this to the Behave tests milestone Oct 29, 2026
MisterRaindrop added a commit to MisterRaindrop/cloudberry that referenced this pull request Oct 29, 2026
cluster config held a relative TRUSTED_SHELL, so gpinitsystem could not
find lalshell from the directory the scenarios run in. apache#2177 made it
absolute, so remove those tags.

Two of them, the banner scenarios, are tagged @backup_restore_bashrc
because they append to ~/.bashrc. before_scenario() returns early for
the gpinitsystem feature before it backs ~/.bashrc up, while
after_scenario() still restores it, and with no backup to restore it
deletes ~/.bashrc. Back it up before that early return. Alena Rybakina
found this in review.

The mirror port scenario also waits for the gpinitsystem fixes in apache#2178
and keeps its tag until that is merged.
MisterRaindrop added a commit to MisterRaindrop/cloudberry that referenced this pull request Oct 29, 2026
cluster config held a relative TRUSTED_SHELL, so gpinitsystem could not
find lalshell from the directory the scenarios run in. apache#2177 made it
absolute, so remove those tags.

Two of them, the banner scenarios, are tagged @backup_restore_bashrc
because they append to ~/.bashrc. before_scenario() returns early for
the gpinitsystem feature before it backs ~/.bashrc up, while
after_scenario() still restores it, and with no backup to restore it
deletes ~/.bashrc. Back it up before that early return. Alena Rybakina
found this in review.

The mirror port scenario also waits for the gpinitsystem fixes in apache#2178
and keeps its tag until that is merged.
MisterRaindrop added a commit to MisterRaindrop/cloudberry that referenced this pull request Oct 30, 2026
cluster config held a relative TRUSTED_SHELL, so gpinitsystem could not
find lalshell from the directory the scenarios run in. apache#2177 made it
absolute, so remove those tags.

Two of them, the banner scenarios, are tagged @backup_restore_bashrc
because they append to ~/.bashrc. before_scenario() returns early for
the gpinitsystem feature before it backs ~/.bashrc up, while
after_scenario() still restores it, and with no backup to restore it
deletes ~/.bashrc. Back it up before that early return. Alena Rybakina
found this in review.

The mirror port scenario also waits for the gpinitsystem fixes in apache#2178
and keeps its tag until that is merged.
tuhaihe pushed a commit to MisterRaindrop/cloudberry that referenced this pull request Oct 31, 2026
cluster config held a relative TRUSTED_SHELL, so gpinitsystem could not
find lalshell from the directory the scenarios run in. apache#2177 made it
absolute, so remove those tags.

Two of them, the banner scenarios, are tagged @backup_restore_bashrc
because they append to ~/.bashrc. before_scenario() returns early for
the gpinitsystem feature before it backs ~/.bashrc up, while
after_scenario() still restores it, and with no backup to restore it
deletes ~/.bashrc. Back it up before that early return. Alena Rybakina
found this in review.

The mirror port scenario also waits for the gpinitsystem fixes in apache#2178
and keeps its tag until that is merged.
tuhaihe pushed a commit that referenced this pull request Nov 1, 2026
cluster config held a relative TRUSTED_SHELL, so gpinitsystem could not
find lalshell from the directory the scenarios run in. #2177 made it
absolute, so remove those tags.

Two of them, the banner scenarios, are tagged @backup_restore_bashrc
because they append to ~/.bashrc. before_scenario() returns early for
the gpinitsystem feature before it backs ~/.bashrc up, while
after_scenario() still restores it, and with no backup to restore it
deletes ~/.bashrc. Back it up before that early return. Alena Rybakina
found this in review.

The mirror port scenario also waits for the gpinitsystem fixes in #2178
and keeps its tag until that is merged.
@MisterRaindrop
MisterRaindrop force-pushed the fix/gpinitsystem-behave-defects branch from 12566ed to 4b7cdf9 Compare November 1, 2026 05:57
Each of these is one or two lines Cloudberry lost against Greenplum,
invisible until the Behave suite exercises the feature:

* gpinitsystem: set the mirror's own port after pg_basebackup

  CREATE_QES_MIRROR seeds the mirror with pg_basebackup, which copies the
  primary's postgresql.conf verbatim -- including the primary's port. The
  mirror was then started with START_QE before that port was corrected, so
  on a single host it either bound the wrong port or collided with its own
  primary. Rewrite postgresql.conf before starting the segment.

  Greenplum has the same SED_PG_CONF call but places it after START_QE,
  which leaves the same window open there.

  Also emit explicit 127.0.0.1/32 and ::1/128 trust entries in the
  coordinator's pg_hba.conf. The hostname-based entry alone is not enough
  when the loopback address the client picks depends on the host's IPv4
  and IPv6 resolution order.

* gpinitsystem: make FORCE_FTS_PROBE actually request a probe

  The loop waits for every segment to leave mode 'n'/status 'd', but the
  call that asks FTS to look -- Greenplum ends each iteration with

      select gp_request_fts_probe_scan()

  -- is missing, so all sixty iterations re-read an unchanged
  gp_segment_configuration in a couple of seconds and give up long before
  gp_fts_probe_interval elapses. gpinitsystem then logs

      [WARN]:-Failed to start Cloudberry instance; please review
              gpinitsystem log to determine failure.

  and returns 0 with all three mirrors still down. That is also why the
  FATAL was commented out with a 'should support notify fts' TODO: the
  function could never succeed, so the error it raised had to go instead.

  Restore the probe call. Leaving the FATAL commented out for now -- that
  is a separate decision about how hard gpinitsystem should fail.

  Fixes the three gpinitsystem.feature scenarios that begin 'all the
  segments are running' after a gpinitsystem run.

* gpinitsystem: do not log the default ETCD/FTS config as a warning

  Every gpinitsystem run that is not given -e/-f logs

      [WARN]:-No ETCD cluster host config provided, use default configuration.
      [WARN]:-No FTS cluster host config provided, use default configuration.

  and its own SCAN_LOG_FILE then greps the log for \[WARN\], so every run
  ends with

      [WARN]:-Scan of log file indicates that some warnings or errors
      [WARN]:-were generated during the array creation

  'Log file scan check passed' is therefore unreachable, and the two
  gpinitsystem.feature scenarios that assert it can never pass. Neither
  message reports a problem -- they say which configuration is in use --
  so log them at INFO, which the scan already ignores.

* gpinitsystem: only pass -f to gpinitstandby when there are FTS hosts

  CREATE_STANDBY_QD branches on CLUSTER_BOOT_MODE and, on the PRODUCTION
  side, always appends

      -f $FTS_HOST_STRING_LIST

  CLUSTER_BOOT_MODE is PRODUCTION for every cluster whose coordinator and
  segments are on different hosts -- that is, every real one; DEMO is only
  reached when they are the same host. FTS_HOST_STRING_LIST, though, stays
  empty unless the cluster was given external FTS hosts. So on an ordinary
  multi-host cluster the call expands to a bare trailing -f and
  gpinitstandby dies on its own option parser:

      gpinitstandby: error: -f option requires 1 argument

  in the same second, before it opens a log file. gpinitsystem reports

      [WARN]:-Failed to complete standby coordinator initialization
      [WARN]:-Cluster setup finished, but Standby Coordinator failed to
              initialize.

  and still exits 0, so a cluster built with -s comes up with no standby
  and nothing but a warning to say so. Running gpinitstandby by hand
  afterwards works, which is what has been hiding this.

  Guard -f on having a value, the way -P, -S and --hba-hostnames already
  are, and drop the branch: the two sides are now the same statement.
  Greenplum has no -f here at all and a single unconditional invocation.

  Also fix the list builder feeding it: [ -z FTS_HOST_STRING_LIST ] tests
  a literal string, which is never empty, so the first host took the
  append branch and the list came out with a leading comma. That path
  needs external FTS to reach and is not exercised here.

  Found on a five-node cluster on EC2; it is also why gpinitsystem.feature's
  HBA_HOSTNAMES=1 standby scenario could never pass -- with -I the DEMO
  assignment is skipped too, so the demo cluster took the same branch.

* gp_bash_functions: let an [ERROR] message set a failing exit status

  LOG_MSG derives the log level from the message and raises EXIT_STATUS so
  that a script which ends in "exit $EXIT_STATUS" reports the failure it
  just logged. The case statement kept the FATAL arm but lost the ERROR
  one, so "[ERROR]" was logged and then thrown away.

  gpinitsystem is the only script that both logs [ERROR] and exits on
  EXIT_STATUS, and it is how the gap shows: an unrecognised long option
  logs "Unknown option --x", prints the usage text, and exits 0. A caller
  checking the return code is told the cluster was created.

  The behave suite could not catch it, because the scenario that passes an
  unknown option was itself asserting the wrong thing -- it ran
  gpinitsystem with --ignore-warnings, which is not an option of any
  version of this script, and expected 0. It passed on the bogus 0 without
  initialising anything, then satisfied its follow-up grep from an older
  log still matching the ~/gpAdminLogs/gpinitsystem*log glob. Drop the
  stray argument so that scenario tests TRUSTED_SHELL as its name says,
  and cover the exit status separately.

  The level is matched against the text up to the first ']', so an INFO
  line that merely mentions an error is still INFO.

* gp_bash_functions: anchor the substitutions SED_PG_CONF makes

  SED_PG_CONF decides between appending and replacing by grepping for
  "^[ ]*${SEARCH_TXT}[ ]*=", but the sed expressions it then runs match
  ${SEARCH_TXT} anywhere on a line, so they edit settings that merely
  contain the requested one in their name.

  gpinitsystem -p passes each parameter of the supplied file through here.
  Asking it for work_mem=999 rewrites

      maintenance_work_mem = 64MB

  into

      maintenance_work_mem=999 #work_mem = 64MB

  which is not a comment: the cluster comes up with maintenance_work_mem
  silently set to the value asked for work_mem. The same happens to
  autovacuum_work_mem and logical_decoding_work_mem, and shared_buffers
  reaches into the wal_buffers line.

  Restore the anchors so a substitution can only touch the assignment it
  names. The searches gpinitsystem and gpcreateseg.sh do for themselves
  are unaffected -- "#port" still resolves to an active port= line, which
  is what the gpinitsystem behave scenario checks on every segment.

  A gp_bash_functions scenario, contributed by Alena Rybakina in review,
  asks SED_PG_CONF for work_mem=999 on a file that also holds
  maintenance_work_mem and a comment naming work_mem, with KEEP_PREV 0
  and 1, locally and through ssh to localhost, and checks that only the
  work_mem line changes. All four fail without the anchors.

* gpinitsystem: correct CHECK_FTS's argument-count guard

  CHECK_FTS reads only $1, the FTS host, and both call sites pass exactly
  that, but its guard was copied from CHECK_ETCD -- which does take a
  second argument, the boot mode -- and still demands two. So every
  external-FTS cluster logged

      [ERROR]: CHECK_FTS invalid params...

  on a perfectly good check. It was invisible while an [ERROR] could not
  influence the exit status; now that it can, gpinitsystem -F would report
  failure after succeeding.

  CLEANUP_ETCD's guard names RETRIVE_ETCD_CONFIG_VAL in its message, which
  sends anyone reading the log after a real failure to a function that was
  not involved. Name the function that is.

  Every remaining $#-guard in the library and in gpinitsystem was checked
  against its call sites; these were the only two wrong.

Remove the @needs_fix tags apache#2170 put on the gpinitsystem scenarios that
wait for this fix, including the mirror port scenario, which also
needed the gpdemo TRUSTED_SHELL fix that is already in main.

Split out of apache#1639 (originally b971d48).
@tuhaihe
tuhaihe force-pushed the fix/gpinitsystem-behave-defects branch from 4b7cdf9 to 3eec440 Compare November 3, 2026 02:01
@tuhaihe
tuhaihe enabled auto-merge (rebase) November 3, 2026 02:01
@tuhaihe
tuhaihe merged commit b389b91 into apache:main Nov 3, 2026
143 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants