From 231b74658ab212dc59a2150f6b3c9d2b425ae87f Mon Sep 17 00:00:00 2001 From: Anthony Ettinger Date: Sat, 31 Oct 2026 12:13:56 +0000 Subject: [PATCH 1/2] Harden fleet deploy trust and audit readiness --- .github/workflows/ci.yml | 4 +- .github/workflows/ssh-deploy.yml | 8 +- README.md | 20 ++- bin/deploy-gate | 114 ++++++++++++---- bin/enroll-repo-openbao.sh | 12 +- bin/oidc-ssh-deploy.mjs | 22 +++- bin/pfdeploy-ssh-receiver | 28 +++- bin/setup-deploy-gate | 68 ++++++++++ docs/fleet-deployment-audit.md | 215 +++++++++++++++++++++++++++++++ docs/fleet-ssh-ca.md | 53 ++++++-- test/deploy.test.js | 80 ++++++++++-- test/fleet-shell.test.js | 31 ++++- test/oidc-ssh.test.js | 34 ++++- 13 files changed, 614 insertions(+), 75 deletions(-) mode change 100644 => 100755 bin/enroll-repo-openbao.sh create mode 100755 bin/setup-deploy-gate create mode 100644 docs/fleet-deployment-audit.md diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index b498c9d..aa2bb01 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -9,8 +9,8 @@ jobs: test: runs-on: ubuntu-latest steps: - - uses: actions/checkout@v4 - - uses: actions/setup-node@v4 + - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4 + - uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4 with: node-version: 22 - run: npm test diff --git a/.github/workflows/ssh-deploy.yml b/.github/workflows/ssh-deploy.yml index 912d934..056529e 100644 --- a/.github/workflows/ssh-deploy.yml +++ b/.github/workflows/ssh-deploy.yml @@ -19,14 +19,14 @@ jobs: cancel-in-progress: false steps: - name: Checkout central deployment tools, target registry and pinned SSH hosts - uses: actions/checkout@v4 + uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4 with: - repository: profullstack/deploy - ref: main + repository: ${{ job.workflow_repository }} + ref: ${{ job.workflow_sha }} path: .pfdeploy persist-credentials: false - name: Set up Node - uses: actions/setup-node@v4 + uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4 with: node-version: '22' - name: GitHub OIDC to OpenBao to ephemeral SSH certificate to deployed app diff --git a/README.md b/README.md index 53d846c..822dc03 100644 --- a/README.md +++ b/README.md @@ -2,16 +2,16 @@ Deploy in batches, **3 at a time**, so one box never builds everything at once. -On 2026-10-19 a change rolled out to 38 repos at once on our production box. Every repo installed, tested, type-checked and built together, then every merge's deploy built again; the box ran out of memory and the kernel killed the shared production Postgres. This package makes that impossible from both ends: +On 2026-10-19 a change rolled out to 38 repos at once on our production box. Every repo installed, tested, type-checked and built together, then every merge's deploy built again; the box ran out of memory and the kernel killed the shared production Postgres. When every production deployment entry point is wired through this package, two controls limit this failure mode: -- **`deploy-gate`** (bash + `flock`, on the box): every deploy takes one of N slots before it builds. Merges can land all at once; builds queue and run three at a time. +- **`deploy-gate`** (bash + `flock`, on the box): every wired deploy takes one of N slots before it builds. Merges can land all at once; builds queue and run three at a time. A deploy path that bypasses the gate is not covered. - **`pfdeploy batch`** (and `batch()`, where the rollout is driven): run one command across many repos, three at a time, stopping at the first failure. Zero dependencies. Linux (`flock`, `/proc/meminfo`); the batch runner works anywhere Node 18+ does. ## Fleet-wide deployments without per-site SSH keys -**One workflow, one centrally managed SSH certificate authority, zero permanent private deployment keys in app repositories.** On every merge to `main` or `master`, the repository's existing tests/typechecks/ThreatCrush scan run, then the same shared workflow deploys the actual commit. GitHub's signed OIDC assertion authenticates the repo to OpenBao, which returns a five-minute SSH certificate for that repo's restricted account. Target hosts and health endpoints live in `fleet/targets.json` keyed by stable GitHub repository ID; the single signing URL lives in `fleet/config.json`. `fleet/properties.txt` contains the currently supplied property inventory, pending verification of repo/host associations. +**One workflow, one centrally managed SSH certificate authority, zero permanent private deployment keys in app repositories.** Each app must retain its own tests, type checks, builds, and ThreatCrush scan, and make the deploy job depend on all of them. The shared workflow cannot verify those caller-side dependencies; protect the deployment branch with required checks and review rules as well. The workflow deploys only the push SHA on `main` or `master`. GitHub's signed OIDC assertion authenticates the repo to OpenBao, which issues a short-lived SSH certificate for that repo's restricted account. Target hosts and health endpoints live in `fleet/targets.json` keyed by stable GitHub repository ID; the single signing URL lives in `fleet/config.json`. `fleet/properties.txt` contains the currently supplied property inventory, pending verification of repo/host associations. See **[Full SSH CA rollout and provisioning guide](docs/fleet-ssh-ca.md)**. The new workflow at `.github/workflows/ssh-deploy.yml` is disabled for an app until its central target entry is explicitly `enabled` and OpenBao + the host trust have been bootstrapped. The existing deployment gate remains in place. **No live app deployment is claimed by this code alone.** @@ -19,10 +19,14 @@ See **[Full SSH CA rollout and provisioning guide](docs/fleet-ssh-ca.md)**. The ```sh npx @profullstack/deploy install /usr/local/bin # as root: one copy every user's deploys share +install -m 0755 bin/setup-deploy-gate /usr/local/sbin/setup-deploy-gate +sudo /usr/local/sbin/setup-deploy-gate pfdeploy pfdeploy-1423083790 # root: provision three shared slots and group access deploy-gate --name crawlproof -- ./deploy-app.sh "$SHA" deploy-gate status ``` +Provision the gate before creating an SSH deployment account or allowing it to deploy. The setup command creates a root-owned sticky directory, three root-owned group-writable slot files, and a `systemd-tmpfiles` rule so the locks return safely after reboot. Add every per-repository deploy account on that host as an argument. It refuses to adopt an unsafe existing directory; drain deployments and inspect old lock state before replacing an earlier world-writable setup. New SSH sessions pick up the shared group membership. + ``` slot 0 busy crawlproof pid 41233 since 2026-10-19T13:20:02Z slot 1 busy smshub pid 41290 since 2026-10-19T13:20:09Z @@ -33,15 +37,15 @@ slot 2 free How it decides: 1. Take one of `DEPLOY_SLOTS` (3) lock files in `/run/lock/profullstack-deploy` (shared by root and every user). All taken: wait, saying who holds them every 30 s. -2. Wait up to `DEPLOY_MEM_WAIT` (600 s) for `DEPLOY_MIN_FREE_GB` (8) of available memory, then go on. +2. Wait up to `DEPLOY_MEM_WAIT` (600 s) for `DEPLOY_MIN_FREE_GB` (8) of available memory, then fail with exit 75 if the host is still below the floor. 3. Run the deploy with the lock descriptor closed, so a server it starts never keeps the slot. Signals are forwarded; the exit code is the deploy's. -4. Still no slot after `DEPLOY_WAIT` (3600 s): exit 75 (try again), or run anyway with `DEPLOY_GATE_ON_TIMEOUT=run`. +4. Still no slot after `DEPLOY_WAIT` (3600 s): exit 75 (try again). Missing or unsafe shared lock files also fail closed; they never start an ungated deploy. -A deploy already inside the gate (`DEPLOY_GATED=1`) passes straight through, so a gated script that calls another gated script never waits on itself. Without `flock` the gate runs the deploy ungated rather than blocking it. +A deploy already inside the gate passes straight through, so a gated script that calls another gated script never waits on itself. The SSH receiver clears inherited `DEPLOY_GATED` values before entering the gate. ### Wiring a deploy script -Put this right after `set -euo pipefail`; if the gate is not installed the script runs exactly as before: +Put this right after `set -euo pipefail`. A deploy script wired to the shared gate fails closed if the gate has not been installed: ```sh # Three deploys at a time on this box (@profullstack/deploy). @@ -49,6 +53,8 @@ if [ -z "${DEPLOY_GATED:-}" ]; then for gate in /usr/local/bin/deploy-gate "$HOME/.local/bin/deploy-gate"; do [ -x "$gate" ] && exec "$gate" --name "" -- "$0" "$@" done + echo "deploy-gate is not installed; refusing to deploy ungated" >&2 + exit 75 fi ``` diff --git a/bin/deploy-gate b/bin/deploy-gate index 434bfbd..d35a08a 100755 --- a/bin/deploy-gate +++ b/bin/deploy-gate @@ -12,12 +12,13 @@ # the production database (dev2, 2026-10-19). # # The command runs with the lock's descriptor closed, so a daemon it starts never -# holds the slot. A deploy already inside the gate (DEPLOY_GATED=1) passes -# straight through, so nested scripts never wait on themselves. +# holds the slot. A deploy already inside the gate passes straight through, so +# nested scripts never wait on themselves. The default lock directory and its +# root-owned files must be provisioned before deploy users are admitted. # -# Environment (flags win): DEPLOY_SLOTS=3, DEPLOY_WAIT=3600 (seconds for a slot; -# then exit 75, or run anyway with DEPLOY_GATE_ON_TIMEOUT=run), DEPLOY_MIN_FREE_GB=8, -# DEPLOY_MEM_WAIT=600 (seconds to wait for memory, then go on), DEPLOY_POLL=5, DEPLOY_LOCK_DIR. +# Environment (flags win): DEPLOY_SLOTS=3, DEPLOY_WAIT=3600, DEPLOY_MIN_FREE_GB=8, +# DEPLOY_MEM_WAIT=600 (seconds before failing for low memory), DEPLOY_POLL=5, +# DEPLOY_LOCK_DIR (test/custom-user mode only; default host path is root provisioned). set -uo pipefail SLOTS=${DEPLOY_SLOTS:-3} @@ -26,29 +27,65 @@ MIN_FREE_GB=${DEPLOY_MIN_FREE_GB:-8} MEM_WAIT=${DEPLOY_MEM_WAIT:-600} POLL=${DEPLOY_POLL:-5} LOCK_DIR=${DEPLOY_LOCK_DIR:-/run/lock/profullstack-deploy} +DEFAULT_LOCK_DIR=/run/lock/profullstack-deploy NAME="" say() { printf '[deploy-gate] %s\n' "$*" >&2; } usage() { sed -n '4,6p' "$0" | sed 's/^# \{0,1\}//' >&2; exit 64; } lock_dir() { - if [ ! -d "$LOCK_DIR" ]; then - (umask 000; mkdir -p "$LOCK_DIR") 2>/dev/null || { say "cannot make $LOCK_DIR"; return 1; } - chmod 1777 "$LOCK_DIR" 2>/dev/null || true + local owner group mode + [ -d "$LOCK_DIR" ] && [ ! -L "$LOCK_DIR" ] || { say "$LOCK_DIR must be a pre-provisioned, non-symlink directory"; return 1; } + owner=$(stat -c %u -- "$LOCK_DIR" 2>/dev/null) || return 1 + group=$(stat -c %g -- "$LOCK_DIR" 2>/dev/null) || return 1 + mode=$(stat -c %a -- "$LOCK_DIR" 2>/dev/null) || return 1 + if [ "$LOCK_DIR" = "$DEFAULT_LOCK_DIR" ]; then + [ "$owner" = 0 ] && [ "$group" != 0 ] && [ "$mode" = 3770 ] || { + say "$LOCK_DIR must be root-owned, group-shared, sticky, and mode 3770" + return 1 + } + else + [ "$owner" = "$(id -u)" ] || { say "custom lock directory must be owned by this user"; return 1; } + [ "$mode" = 700 ] || { say "custom lock directory must have mode 700"; return 1; } fi } slot_file() { printf '%s/slot-%s' "$LOCK_DIR" "$1"; } +slot_file_ready() { + local f="$1" owner group mode dir_group + [ -f "$f" ] && [ ! -L "$f" ] || { say "$(basename "$f") must be a pre-provisioned regular file"; return 1; } + owner=$(stat -c %u -- "$f" 2>/dev/null) || return 1 + group=$(stat -c %g -- "$f" 2>/dev/null) || return 1 + mode=$(stat -c %a -- "$f" 2>/dev/null) || return 1 + if [ "$LOCK_DIR" = "$DEFAULT_LOCK_DIR" ]; then + dir_group=$(stat -c %g -- "$LOCK_DIR" 2>/dev/null) || return 1 + [ "$owner" = 0 ] && [ "$group" = "$dir_group" ] && [ "$mode" = 660 ] || { + say "$(basename "$f") must be root-owned by the lock group and mode 660" + return 1 + } + else + [ "$owner" = "$(id -u)" ] && [ "$mode" = 600 ] || { + say "custom slot files must be owned by this user and mode 600" + return 1 + } + fi +} + +validate_slots() { + local i + for ((i = 0; i < SLOTS; i++)); do slot_file_ready "$(slot_file "$i")" || return 1; done +} + # The holder line written into a slot: who, since when. holder() { head -c 200 "$(slot_file "$1")" 2>/dev/null | head -1; } status() { - lock_dir || exit 1 + command -v flock >/dev/null || { say "flock is unavailable"; exit 1; } + lock_dir && validate_slots || exit 1 local i f busy=0 for ((i = 0; i < SLOTS; i++)); do f=$(slot_file "$i") - [ -e "$f" ] || (umask 000; : > "$f") if flock -n "$f" true 2>/dev/null; then printf 'slot %d free\n' "$i" else @@ -61,9 +98,12 @@ status() { free_gb() { awk '/^MemAvailable:/ { printf "%d", $2 / 1048576 }' /proc/meminfo 2>/dev/null || echo 0; } -[ "${1:-}" = status ] && { shift; [ "${1:-}" = --slots ] && SLOTS=$2; status; exit 0; } +[ "${1:-}" = status ] && { shift; [ "${1:-}" = --slots ] && { [ "$#" -ge 2 ] || usage; SLOTS=$2; }; status; exit 0; } while [ $# -gt 0 ]; do + case "$1" in + --slots|--name|--wait|--min-free-gb) [ "$#" -ge 2 ] || usage ;; + esac case "$1" in --slots) SLOTS=$2; shift 2 ;; --name) NAME=$2; shift 2 ;; @@ -76,15 +116,44 @@ while [ $# -gt 0 ]; do done [ $# -gt 0 ] || usage case "$SLOTS" in ''|*[!0-9]*|0) say "--slots must be a positive number"; exit 64 ;; esac +SLOTS=$((10#$SLOTS)) +[ "$SLOTS" -le 256 ] || { say "--slots must not exceed 256"; exit 64; } +for value in "$WAIT" "$MIN_FREE_GB" "$MEM_WAIT"; do + [[ "$value" =~ ^[0-9]+$ ]] || { say "wait and memory limits must be non-negative integers"; exit 64; } +done +[[ "$POLL" =~ ^[0-9]+([.][0-9]+)?$ ]] || { say "poll interval must be a non-negative number"; exit 64; } +WAIT=$((10#$WAIT)); MIN_FREE_GB=$((10#$MIN_FREE_GB)); MEM_WAIT=$((10#$MEM_WAIT)) + +already_gated() { + local target="${DEPLOY_GATED:-}" pid parent command + [[ "$target" =~ ^[1-9][0-9]*$ ]] || return 1 + pid=$$ + while [[ "$pid" =~ ^[1-9][0-9]*$ ]]; do + if [ "$pid" = "$target" ]; then + command=$(tr '\0' ' ' < "/proc/$pid/cmdline" 2>/dev/null || true) + [[ "$command" == *deploy-gate* ]] + return + fi + parent=$(awk '/^PPid:/ { print $2 }' "/proc/$pid/status" 2>/dev/null) + [ -n "$parent" ] || return 1 + pid=$parent + done + return 1 +} # Already inside the gate: never take a second slot. -if [ "${DEPLOY_GATED:-}" = 1 ]; then exec "$@"; fi +if already_gated; then exec "$@"; fi +unset DEPLOY_GATED NAME=${NAME:-$(basename "$PWD")} -export DEPLOY_GATED=1 +export DEPLOY_GATED=$$ -if ! command -v flock >/dev/null || ! lock_dir; then - say "no flock or lock dir: running ungated" - exec "$@" +if ! command -v flock >/dev/null; then + say "flock is unavailable; refusing to run ungated" + exit 75 +fi +if ! lock_dir || ! validate_slots; then + say "shared deployment locks are unavailable; refusing to run ungated" + exit 75 fi # Take a slot. Start at a random slot so waiters spread out; poll every DEPLOY_POLL (5) s. @@ -96,7 +165,6 @@ while :; do for ((n = 0; n < SLOTS; n++)); do i=$(((first + n) % SLOTS)) f=$(slot_file "$i") - [ -e "$f" ] || (umask 000; : > "$f") 2>/dev/null { exec {fd}<>"$f"; } 2>/dev/null || continue if flock -n "$fd"; then slot=$i @@ -107,10 +175,6 @@ while :; do done now=$(date +%s) if [ $((now - start)) -ge "$WAIT" ]; then - if [ "${DEPLOY_GATE_ON_TIMEOUT:-fail}" = run ]; then - say "waited ${WAIT}s for a slot; running anyway (DEPLOY_GATE_ON_TIMEOUT=run)" - exec "$@" - fi say "waited ${WAIT}s for one of $SLOTS deploy slots; giving up (exit 75). Busy: $(for ((i = 0; i < SLOTS; i++)); do printf '%s; ' "$(holder "$i")"; done)" exit 75 fi @@ -130,8 +194,10 @@ say "$NAME took deploy slot $slot of $SLOTS$([ "$waited" -gt 0 ] && printf ' aft mem_start=$(date +%s) while [ "$(free_gb)" -lt "$MIN_FREE_GB" ]; do if [ $(($(date +%s) - mem_start)) -ge "$MEM_WAIT" ]; then - say "only $(free_gb) GB available after ${MEM_WAIT}s; going on" - break + say "only $(free_gb) GB available after ${MEM_WAIT}s; refusing to start a build" + : > "$(slot_file "$slot")" 2>/dev/null || true + exec {fd}>&- + exit 75 fi say "$(free_gb) GB available, want $MIN_FREE_GB; waiting" sleep 15 @@ -139,6 +205,8 @@ done # Run with the lock descriptor closed, forwarding signals, and release when it ends. child="" +# Signal traps invoke this handler by name. +# shellcheck disable=SC2329 forward() { [ -n "$child" ] && kill "-$1" "$child" 2>/dev/null; } trap 'forward TERM' TERM trap 'forward INT' INT diff --git a/bin/enroll-repo-openbao.sh b/bin/enroll-repo-openbao.sh old mode 100644 new mode 100755 index 04507c9..4c2375f --- a/bin/enroll-repo-openbao.sh +++ b/bin/enroll-repo-openbao.sh @@ -2,18 +2,21 @@ # Central operator enrollment. One OpenBao role and one policy per GitHub repo, # automatically generated; NEVER one long-lived SSH key per repo. # Run only from an authenticated OpenBao administrator workstation. -# Usage: ./bin/enroll-repo-openbao.sh OWNER/REPO GITHUB_REPO_ID main|master +# Usage: ./bin/enroll-repo-openbao.sh OWNER/REPO GITHUB_REPO_ID main|master WORKFLOW_SHA set -euo pipefail repo="${1:-}" id="${2:-}" branch="${3:-}" +workflow_sha="${4:-}" [[ "$repo" =~ ^[a-zA-Z0-9_.-]+/[a-zA-Z0-9_.-]+$ ]] || { echo "invalid owner/repo" >&2; exit 2; } [[ "$id" =~ ^[1-9][0-9]{0,19}$ ]] || { echo "GitHub repo ID must be numeric" >&2; exit 2; } [[ "$branch" = main || "$branch" = master ]] || { echo "branch must be main or master" >&2; exit 2; } +[[ "$workflow_sha" =~ ^[a-f0-9]{40}$ ]] || { echo "workflow SHA must be a full lowercase commit SHA" >&2; exit 2; } command -v bao >/dev/null || { echo "Install the OpenBao CLI first" >&2; exit 2; } policy="fleet-deploy-repo-$id" +workflow_ref="profullstack/deploy/.github/workflows/ssh-deploy.yml@$workflow_sha" tmp="$(mktemp)" trap 'rm -f "$tmp"' EXIT cat > "$tmp" < setTimeout(r, 3000)) + if (i < attempts - 1) await new Promise(r => setTimeout(r, retryDelayMs)) } throw new Error('Deployment health check failed: ' + (lastError?.message || 'unknown')) } @@ -132,13 +142,11 @@ export async function deploy(env = process.env) { vaultToken, body: { public_key: publicKey, valid_principals: cfg.username, ttl: '5m' }, }) - if (typeof signed?.data?.signed_key !== 'string' || !signed.data.signed_key.includes('-cert-v01@openssh.com')) { - throw new Error('OpenBao did not return an SSH user certificate') - } - writeFileSync(certificate, signed.data.signed_key + '\n', { mode: 0o600 }) + writeFileSync(certificate, normalizeSignedCertificate(signed?.data?.signed_key), { mode: 0o600 }) console.log('Deploying', cfg.repository, cfg.sha.slice(0, 12), 'to', cfg.host, 'with a short-lived SSH certificate') execFileSync('ssh', [ '-o', 'BatchMode=yes', '-o', 'IdentitiesOnly=yes', '-o', 'StrictHostKeyChecking=yes', + '-F', '/dev/null', '-o', 'UserKnownHostsFile=' + knownHosts, '-o', 'CertificateFile=' + certificate, '-o', 'ConnectTimeout=15', '-p', cfg.port, '-i', key, cfg.username + '@' + cfg.host, 'deploy ' + cfg.sha, diff --git a/bin/pfdeploy-ssh-receiver b/bin/pfdeploy-ssh-receiver index 2d5f397..4b87f04 100644 --- a/bin/pfdeploy-ssh-receiver +++ b/bin/pfdeploy-ssh-receiver @@ -9,10 +9,11 @@ account="$(id -un)" [[ "$account" =~ ^pfdeploy-[1-9][0-9]{0,19}$ ]] || deny "unrecognized deployment identity" id="${account#pfdeploy-}" cfg="/etc/profullstack/deploy/$account.conf" -[[ -f "$cfg" ]] || deny "repository not enrolled" +[[ -f "$cfg" && ! -L "$cfg" ]] || deny "repository not enrolled or enrollment is a symlink" [[ "$(stat -c %u "$cfg")" = 0 ]] || deny "enrollment must be owned by root" mode="$(stat -c %a "$cfg")" [[ "$mode" = 644 || "$mode" = 640 || "$mode" = 600 ]] || deny "unsafe configuration permissions" +[[ "$(realpath -e -- "$cfg")" = "$cfg" ]] || deny "enrollment path is not canonical" # Strict KEY=value reader; never evaluate shell code from config or SSH input. setting() { sed -n "s/^$1=//p" "$cfg" | head -n 1; } @@ -21,7 +22,21 @@ branch="$(setting BRANCH)" checkout="$(setting CHECKOUT)" [[ "$repo" =~ ^[a-zA-Z0-9_.-]+/[a-zA-Z0-9_.-]+$ ]] || deny "invalid repository enrollment" [[ "$branch" = main || "$branch" = master ]] || deny "invalid protected branch" -[[ "$checkout" = /* && -d "$checkout/.git" ]] || deny "missing dedicated checkout" +[[ "$checkout" = /* && "$checkout" != *$'\n'* && -d "$checkout/.git" && ! -L "$checkout/.git" ]] || deny "missing dedicated checkout or unsafe Git metadata" +[[ "$(realpath -e -- "$checkout")" = "$checkout" ]] || deny "checkout path contains symlinks or traversal" +parent="${checkout%/*}" +[[ -n "$parent" ]] || parent="/" +while [[ "$parent" != "/" ]]; do + [[ -d "$parent" && ! -L "$parent" && "$(realpath -e -- "$parent")" = "$parent" ]] || deny "checkout parent path is not canonical" + parent_owner="$(stat -c %u "$parent")" + parent_mode="$(stat -c %a "$parent")" + [[ "$parent_owner" = 0 ]] || deny "checkout parent must be root-owned" + (( (8#$parent_mode & 0022) == 0 )) || deny "checkout parent must not be group/world-writable" + parent="${parent%/*}" + [[ -n "$parent" ]] || parent="/" +done +[[ "$(stat -c %u "$checkout")" = "$(id -u)" && "$(stat -c %u "$checkout/.git")" = "$(id -u)" ]] || deny "checkout must be owned by its deployment account" +[[ "$(git -C "$checkout" rev-parse --show-toplevel)" = "$checkout" ]] || deny "Git worktree root does not match enrollment" [[ "${SSH_ORIGINAL_COMMAND:-}" =~ ^deploy[[:space:]]([a-f0-9]{40})$ ]] || deny "only deploy FULL_SHA is allowed" sha="${BASH_REMATCH[1]}" remote="$(git -C "$checkout" remote get-url origin)" @@ -39,7 +54,8 @@ old="$(git -C "$checkout" rev-parse HEAD)" git -C "$checkout" merge-base --is-ancestor "$old" "$sha" || deny "refusing rollback" git -C "$checkout" checkout --detach --force "$sha" cd "$checkout" -if [[ -x /usr/local/bin/deploy-gate ]]; then - exec /usr/local/bin/deploy-gate --name "$id" -- bash bin/install.sh -fi -exec bash bin/install.sh +PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin +export PATH +unset DEPLOY_GATED DEPLOY_LOCK_DIR DEPLOY_SLOTS DEPLOY_WAIT DEPLOY_MIN_FREE_GB DEPLOY_MEM_WAIT DEPLOY_POLL DEPLOY_GATE_ON_TIMEOUT +[[ -x /usr/local/bin/deploy-gate ]] || deny "shared deploy-gate is not installed" +exec /usr/local/bin/deploy-gate --name "$id" -- bash bin/install.sh diff --git a/bin/setup-deploy-gate b/bin/setup-deploy-gate new file mode 100755 index 0000000..83f9319 --- /dev/null +++ b/bin/setup-deploy-gate @@ -0,0 +1,68 @@ +#!/usr/bin/env bash +# Root-only host provisioning for the shared cross-user deployment semaphore. +set -euo pipefail + +LOCK_DIR=/run/lock/profullstack-deploy +GROUP="${1:-pfdeploy}" +shift || true +SLOTS="${DEPLOY_SLOTS:-3}" + +fail() { printf 'setup-deploy-gate: %s\n' "$*" >&2; exit 1; } +[[ "$(id -u)" = 0 ]] || fail 'run as root' +[[ "$GROUP" =~ ^[a-z_][a-z0-9_-]{0,30}$ ]] || fail 'invalid gate group name' +[[ "$SLOTS" =~ ^[1-9][0-9]{0,2}$ ]] || fail 'DEPLOY_SLOTS must be from 1 through 256' +((10#$SLOTS <= 256)) || fail 'DEPLOY_SLOTS must be from 1 through 256' +SLOTS=$((10#$SLOTS)) +command -v getent >/dev/null || fail 'getent is required' +command -v groupadd >/dev/null || fail 'groupadd is required' +command -v usermod >/dev/null || fail 'usermod is required' +command -v systemd-tmpfiles >/dev/null || fail 'systemd-tmpfiles is required for reboot recovery' + +if ! getent group "$GROUP" >/dev/null; then groupadd --system "$GROUP"; fi +GID="$(getent group "$GROUP" | cut -d: -f3)" + +if [ -L "$LOCK_DIR" ]; then fail "$LOCK_DIR is a symlink; inspect and remove it during a drained maintenance window"; fi +if [ ! -e "$LOCK_DIR" ]; then + install -d -o root -g "$GROUP" -m 0700 "$LOCK_DIR" +else + [[ -d "$LOCK_DIR" ]] || fail "$LOCK_DIR exists and is not a directory" + [[ "$(stat -c %u "$LOCK_DIR")" = 0 && "$(stat -c %g "$LOCK_DIR")" = "$GID" ]] || fail "$LOCK_DIR must be owned by root:$GROUP" + mode="$(stat -c %a "$LOCK_DIR")" + [[ "$mode" = 700 || "$mode" = 3770 ]] || fail "$LOCK_DIR has unsafe mode $mode; drain old deployments and inspect it manually" +fi + +for ((i = 0; i < SLOTS; i++)); do + slot="$LOCK_DIR/slot-$i" + if [ -L "$slot" ]; then fail "$slot is a symlink"; fi + if [ ! -e "$slot" ]; then + [[ "$(stat -c %a "$LOCK_DIR")" = 700 ]] || fail "cannot add $slot while the shared directory is open to deploy users" + install -o root -g "$GROUP" -m 0660 /dev/null "$slot" + else + [[ -f "$slot" ]] || fail "$slot is not a regular file" + [[ "$(stat -c %u "$slot")" = 0 && "$(stat -c %g "$slot")" = "$GID" ]] || fail "$slot must be owned by root:$GROUP" + [[ "$(stat -c %a "$slot")" = 660 ]] || fail "$slot must have mode 660" + fi +done + +for account in "$@"; do + getent passwd "$account" >/dev/null || fail "unknown deployment account: $account" + usermod -a -G "$GROUP" "$account" +done + +tmpfiles="/etc/tmpfiles.d/profullstack-deploy.conf" +tmp="$(mktemp /etc/tmpfiles.d/.profullstack-deploy.XXXXXX)" +trap 'rm -f "$tmp"' EXIT +{ + printf 'd %s 3770 root %s - -\n' "$LOCK_DIR" "$GROUP" + for ((i = 0; i < SLOTS; i++)); do + printf 'f %s/slot-%s 0660 root %s - -\n' "$LOCK_DIR" "$i" "$GROUP" + done +} >"$tmp" +if [ -e "$tmpfiles" ] && ! cmp -s "$tmp" "$tmpfiles"; then + fail "$tmpfiles already exists with different contents; review it before updating" +fi +install -o root -g root -m 0644 "$tmp" "$tmpfiles" +chmod 3770 "$LOCK_DIR" +systemd-tmpfiles --create "$tmpfiles" + +printf 'Provisioned %s slots in %s for group %s. New SSH sessions pick up account group membership.\n' "$SLOTS" "$LOCK_DIR" "$GROUP" diff --git a/docs/fleet-deployment-audit.md b/docs/fleet-deployment-audit.md new file mode 100644 index 0000000..fb8c010 --- /dev/null +++ b/docs/fleet-deployment-audit.md @@ -0,0 +1,215 @@ +# Fleet deployment security and readiness audit + +**Audit date:** 2026-10-31 +**Implementation reviewed:** `profullstack/deploy` merge commit `d91e84f93230232410c231d48e47df122631fbf5` (PR #2), plus the hardening changes on branch `audit/fleet-deployment-readiness`. +**Verdict:** **NOT PRODUCTION READY.** The software path passed isolated end-to-end tests after fixes, but the production trust boundaries and infrastructure have not been provisioned or verified. Do not enable the PIT target or migrate another application yet. + +## Executive summary + +The design can remove permanent per-application SSH private keys: each runner creates an ephemeral Ed25519 key, obtains a short-lived OpenSSH user certificate through GitHub OIDC and OpenBao, and logs in as a repository-specific unprivileged account. The receiver enforces a forced deploy command, checks the repository and branch, rejects rollbacks, runs install behind a host-wide semaphore, and the runner checks HTTPS health. + +The central implementation was **not merely syntax checked**. In a disposable environment, OpenBao 2.7.2 issued a certificate from a synthetic OIDC JWT; a local OpenSSH server authenticated it; the real client and receiver deployed an approved test commit as a non-root user; invalid principals, commands, branches, SHA values, and rollback attempts were rejected; and the test health endpoint passed. Separate tests covered role and policy revocation behavior, certificate expiry, gate concurrency and failure paths. + +That is not production evidence. No production OpenBao, production CA, real GitHub-issued OIDC token, dev2 SSHD configuration, deploy account, known host key, or production deployment was available to verify. PIT remains disabled in the registry and still uses its old root/static-key workflow. The merged implementation authorized the mutable central `main` ref without binding `job_workflow_sha`. This audit branch fixes that path: callers must pin the central workflow to a commit, OpenBao roles bind both `job_workflow_ref` and `job_workflow_sha`, and the called workflow checks out its own helper code from `${{ job.workflow_sha }}`. Central `main` still lacks branch protection and should be protected before rollout. + +## Architecture verification + +### Verified in code and isolated tests + +- The reusable workflow is declared with `workflow_call`, restricts its job to `push` on `main` or `master`, grants only `contents: read` and `id-token: write`, and groups deploy jobs by caller repository ID. +- The client resolves the host and health URL from `fleet/targets.json` by runner-provided `GITHUB_REPOSITORY_ID`; it verifies the owner/repo and ref too. Unknown, disabled, or mismatched targets fail closed. +- GitHub's OIDC request endpoint must be HTTPS. The token audience is `profullstack-deploy`; OpenBao login and sign API failures return nonzero without printing response bodies or tokens. +- A fresh Ed25519 key is generated in a mode-restricted temporary directory. The private key is used only by that run and removed in `finally`. SSH is invoked with `-F /dev/null`, strict host-key checking, the central known-hosts file, and no key-scan or insecure fallback. +- The enrollment helper creates one JWT role, one signer policy, and one SSH signer role per repository ID. The JWT role binds repository name, immutable repository ID, branch ref, push event, audience, exact reusable workflow ref, and exact reusable workflow SHA. The signer policy can sign only that repository's signer role; that role allows only `pfdeploy-REPO_ID`. +- GitHub documents `job.workflow_repository` and `job.workflow_sha` for checking out a reusable workflow's own source. The shared workflow now uses those fields so the helper checkout matches the exact workflow code. This context is documented for GitHub.com; it is not available on GitHub Enterprise Server. +- The receiver is configured as an SSH `ForceCommand`. A local disposable `sshd -T` check confirmed `TrustedUserCAKeys`, `AuthorizedPrincipalsFile`, `ForceCommand`, no TTY, and no TCP forwarding. The receiver rejected arbitrary shell commands and accepts only a full 40-character lowercase commit SHA. +- The receiver checks a root-owned non-symlink enrollment file, canonical checkout path, root-owned/non-writable parent chain, non-symlink `.git` metadata, checkout ownership, expected Git remote, enrolled branch ancestry, and forward-only progression before checking out the SHA. The local certificate principal was tied to the matching unprivileged Unix account. The app install command ran as UID 1000 in the fixture; no sudo was granted. +- The receiver clears inherited deploy-gate overrides, requires the root-provisioned shared gate, and fails closed if it is missing or unsafe. The hardened gate uses three cross-user root-owned lock files by default, refuses unsafe/symlinked files, waits for the configured memory floor, releases its slot on timeout, and has no timeout bypass. +- Health checks reject redirects, retry transient failures, and fail the deployment if the endpoint never returns success. A regression test now covers an HTTP 503 response. + +### Concurrency and failure reporting + +The reusable workflow's concurrency group is repository-specific and uses `cancel-in-progress: false`. GitHub allows one running and one pending run per group by default; a newer pending run replaces the older pending run, so this is not a FIFO deploy queue. The running run is not cancelled, and the newest surviving run deploys its own immutable `GITHUB_SHA`. The host-wide `deploy-gate` is a separate semaphore with a default of three slots shared by every migrated deployment that uses the same provisioned lock directory. Its timeout and memory failures return exit 75; SSH, install, and final health-check failures also leave the job failed. Existing deployments that bypass the gate are outside that limit. + +### Not verified against real GitHub + +GitHub's current OIDC documentation lists `repository`, `repository_id`, `ref`, `event_name`, `job_workflow_ref`, and `job_workflow_sha` among the relevant claims for reusable workflows. Its documentation also requires `id-token: write` in the calling workflow/job and permits public reusable workflows across organizations subject to each caller organization's Actions policy. The claim names and intended format were checked against the docs, and the isolated role matched synthetic claims. **No token from a real GitHub Actions run was exchanged with OpenBao.** Therefore the production issuer, audience, exact claim values, cross-organization policy, and organization-specific workflow access remain unverified. + +Current official references: + +- GitHub OIDC: https://docs.github.com/en/actions/reference/security/oidc +- Reusable workflows: https://docs.github.com/en/actions/how-tos/reuse-automations/reuse-workflows +- GitHub job context for reusable workflow source: https://docs.github.com/en/actions/reference/workflows-and-actions/contexts#job-context +- Workflow concurrency: https://docs.github.com/en/actions/concepts/workflows-and-actions/concurrency +- OpenBao JWT auth: https://openbao.org/docs/auth/jwt/ +- OpenBao JWT API: https://openbao.org/docs/api/auth/jwt/ +- OpenBao SSH signed certificates: https://openbao.org/docs/secrets/ssh/signed-ssh-certificates/ +- OpenBao audit configuration: https://openbao.org/docs/configuration/audit/ + +## Findings + +### High: central branch protection is absent; immutable role binding must be used + +The merged implementation authorized `profullstack/deploy/.github/workflows/ssh-deploy.yml@refs/heads/main`, did not bind `job_workflow_sha`, and checked helper code out from mutable `main`. The central GitHub API check returned “branch not protected” for `main`. On this audit branch, callers pin a commit, the enrollment helper binds the exact `job_workflow_ref` and `job_workflow_sha`, and the called workflow checks out its own source using GitHub's `job.workflow_repository` and `job.workflow_sha` context. A changed central commit therefore fails OpenBao authentication until an operator reviews and updates the repo role. The branch protection gap remains a governance risk and must be closed before rollout. + +**Fix before production:** protect central `main` with required review and checks; use the same reviewed commit SHA in each caller and OpenBao role; verify the OIDC `job_workflow_sha` from a real Actions run; and document how approved revisions are rolled out. Pin third-party actions to full commit SHAs as part of the same review. + +### High: tests and security scans are caller-controlled + +The shared workflow cannot inspect the caller's `needs` graph. A repository can call the deploy workflow without waiting for tests, builds, type checks, or ThreatCrush. The central CI workflow at the reviewed merge commit ran `npm test` only; its green run does not establish application gates. The shared example contains placeholder test/scan commands. + +**Fix before each app migration:** preserve the app's real test/build/typecheck/security jobs, make the reusable deploy job depend on all required jobs, and configure required branch checks/rulesets. Audit each workflow individually; do not infer gates from the shared workflow. + +### High: production identity, host and account provisioning is absent + +`fleet/config.json` has an empty OpenBao URL, `ssh/known_hosts` has no pinned host key, PIT's registry target is disabled, and no production CA, OpenBao JWT role, SSHD CA trust, account, principal file, or receiver enrollment was verified. There was no production read or write to dev2 and no SSHD change was made. The implementation is deliberately unable to deploy until bootstrap completes. + +**Fix:** provision and test the production control plane and a canary host using the runbook below, with operator-controlled backups and recovery. Keep PIT disabled until all prerequisites pass. + +### High: PIT still requires a non-root deployment design + +The existing PIT deployment runs as root. The new receiver correctly executes app installation as `pfdeploy-1423083790`; it does not grant sudo. PIT's installer needs dependencies, runtime/service setup, environment files, Nginx/TLS integration, and restart permissions. Those filesystem/service boundaries have not been migrated or tested. Granting blanket passwordless sudo would defeat the isolation. + +**Fix:** define service ownership and secrets access for the dedicated account, pre-provision runtime and service units, and use only narrowly scoped root-owned helpers where necessary. Prove restart and rollback in a disposable PIT-like host before production. + +### High: revoking a role alone leaves existing token signing access + +In OpenBao 2.7.2 testing, deleting a JWT role prevented new logins, but a token already issued by that role could still call the signer while its policy remained. Deleting the repo signing policy then blocked that token. Disabling a fleet registry entry only blocks the central client; it does not invalidate a previously issued certificate or OpenBao token. + +**Fix:** the runbook now requires disabling the target, deleting the JWT role and removing its signing policy; to block an already-issued certificate immediately, remove the user's authorized principal or disable the account on the host. Existing certs otherwise remain valid until their expiry. + +### Medium: real OpenBao audit, CA recovery, and rotation are not commissioned + +The disposable OpenBao test validated the JWT and SSH signing API formats and the enrollment helper. It did not validate production TLS, HA, unseal/recovery, durable audit delivery, backup restore, or rotation. OpenBao 2.7.2 rejected enabling an audit device through the API in the isolated instance; its audit device must be configured declaratively. No production audit configuration or restore test is available. + +**Fix:** configure persistent audit devices declaratively on all HA nodes, test both login and signing events plus failure alerting, and run a restore and dual-trust CA rotation rehearsal before production enrollment. + +### Medium: registry coverage is not established + +`fleet/targets.json` contains exactly one entry: PIT, disabled. `fleet/properties.txt` is an unverified inventory and is not a mapping source. No domains were mapped to GitHub repository IDs, protected branches, hosts, health checks, or server accounts. Cross-organization claim behavior is documented but not tested with `profullstack` and `moshcoder` production org settings. + +**Fix:** add a target only after independently verifying repository ID, branch, DNS/host ownership, health URL, server, service user, owner approval, and authorized principal. Unknown IDs must remain disabled; do not infer names from domains. + +### Medium: runtime and checkout assumptions require per-host review + +The receiver requires a dedicated checkout and runs its `bin/install.sh` under the per-repository account. It does not itself establish checkout ownership, service permissions, environment-file protection, per-app systemd policy, or that every other deploy path on the host shares the gate. The gate caps concurrency only for deploy paths that use it; legacy or operator scripts can still bypass it. + +**Fix:** inventory all deployment entry points on each host, migrate them to the shared gate, validate account/group/file permissions, and test service start/stop under the real service manager. Keep app code and secrets isolated per account. + +## Bugs fixed on the audit branch + +- OpenBao's signed certificate response ended in newlines; the client appended another newline, creating an invalid extra line that caused `ssh-keygen -L` to warn. The client now trims and validates a single certificate line before writing it. Regression tests cover valid and malformed responses. +- `deploy-gate` now refuses unsafe lock directories, symlinked slot files, missing lock files, invalid bounds, and low-memory continuation. It releases the slot on low-memory timeout, ignores the old timeout bypass, and validates nested gate markers against the process ancestry. +- Added root-only `bin/setup-deploy-gate` to provision root-owned shared slot files and reboot-persistent tmpfiles configuration. +- The SSH receiver clears gate environment overrides and fails closed if the shared gate is unavailable; it no longer runs install ungated. +- The OpenSSH call ignores ambient client configuration with `-F /dev/null`. +- The receiver now rejects symlinked or traversal checkout paths, unsafe parent directories, symlinked Git metadata, and checkouts not owned by the dedicated deployment account. +- `bin/enroll-repo-openbao.sh` is now executable as its documented `./bin/...` command. +- OpenBao enrollment now requires a full workflow SHA and binds both `job_workflow_ref` and `job_workflow_sha`; the reusable workflow checks out its own tools at `job.workflow_sha`. +- Deploy scripts wired to `deploy-gate` now fail closed if the gate executable is missing. +- Documentation now states the caller-side CI requirement, host-gate provisioning, OpenBao revocation semantics, declarative audit setup, and a CA rotation/recovery procedure. + +## Tests executed and results + +### Repository and workflow checks + +- Reviewed merge commit `d91e84f93230232410c231d48e47df122631fbf5`, PR #2, current workflow, registry, scripts, and tests. +- Merge CI run `41192340522` succeeded on that commit. Its only job ran `npm test`; it did not run security scanning, a real GitHub OIDC exchange, or deployment. +- `npm test` on the hardened audit branch: **31 tests pass** (local final run; hosted PR CI is still required). +- `node --check bin/oidc-ssh-deploy.mjs`: passed. +- `bash -n bin/enroll-repo-openbao.sh`, `bash -n bin/pfdeploy-ssh-receiver`, `bash -n bin/deploy-gate`, and `bash -n bin/setup-deploy-gate`: passed. The scripts are also covered by the repository test suite. +- ShellCheck: passed with no findings. The signal handler is called by a trap and is documented as such. +- `actionlint` was not installed in the audit environment. The GitHub workflow was reviewed manually and will be exercised by the PR's hosted check; no claim is made that it passed actionlint. + +### Disposable OpenBao and SSH environment + +- OpenBao 2.7.2 ran locally in dev mode with an in-memory disposable CA; a synthetic RSA JWT issuer and test-only claims were used. No production token, CA, or endpoint was used. +- The actual enrollment helper configured a repo role and policy bound to `job_workflow_ref` and `job_workflow_sha` for `d91e84f93230232410c231d48e47df122631fbf5`. A synthetic JWT with that exact pinned SHA logged in; changing only the workflow SHA was rejected with HTTP 400. JWT login returned a five-minute token; SSH signing returned a certificate for only `pfdeploy-1423083790` with the requested five-minute lifetime. OpenBao's `valid_after` was 30 seconds before issuance and `valid_before` five minutes after issuance. +- A disposable Debian/OpenSSH server trusted the test CA with `TrustedUserCAKeys`, used `AuthorizedPrincipalsFile` and `ForceCommand`, and authenticated the valid cert. The receiver deployed test commit `dae587c57da41454868ca6e977833d37e3506041`; the fixture's install process ran as UID 1000, and the health endpoint returned success. +- The fixture rejected arbitrary shell input, malformed SHA input, a principal for a different repo, a server enrollment with a mismatched Git remote, a symlink checkout, a traversal path, a feature-branch commit, and a rollback. A short-lived certificate authenticated before expiry and failed after expiry. +- Unauthorized repository and branch JWTs and a foreign principal signing request were rejected by OpenBao. Role deletion blocked a new login; policy deletion blocked an already-issued token from signing. +- An isolated declarative audit configuration wrote request and response records. Attempting to enable the audit device through the API returned HTTP 400 on OpenBao 2.7.2. +- The gate tests launch six jobs against three slots and assert the observed peak is exactly three. Tests also cover queued timeout, nested invocations, a daemon not retaining the lock, memory timeout/release, unsafe lock directories, symlink slots, and rejection of the old bypass flag. +- A disposable full client path used a local HTTPS OIDC fixture, test OpenBao, pinned test SSH host key, the actual client, the actual SSH receiver, and the health endpoint. It deployed the approved commit and reported `Health check passed` both before and after the path-safety changes. The real GitHub reusable-workflow context was not available in that fixture. + +These tests validate the software behavior in isolation. They do **not** substitute for a GitHub-issued OIDC token, production OpenBao, dev2, or a production service health check. + +## Explicit answers to the ten questions + +1. **Functional or only syntactically valid?** More than syntax: the core issuance-to-SSH-to-deploy-to-health path worked in a disposable test. Production functionality remains unverified. +2. **OpenBao APIs and role options correct?** JWT login, signer role, policy, and SSH signing request/response formats worked against OpenBao 2.7.2. Production issuer discovery, token claims, HA, audit, and recovery remain unverified. +3. **Can one compromised repository deploy another app?** The tested role/policy/principal design denied a foreign repo/principal. Each repository has a separate signing policy and principal. The audit branch also requires a pinned central workflow SHA in both caller and OpenBao role, and checks out matching helper code; central branch protection and host account isolation remain required before production. +4. **Can a workflow trick the receiver into an unauthorized commit?** The receiver rejected feature-branch and rollback SHAs. The client deploys runner `GITHUB_SHA`, while the receiver verifies that SHA is on the enrolled branch and forward from current HEAD. It does not cryptographically bind the SSH certificate to a commit; branch protection, trusted shared workflow code, and the receiver's ancestry checks are required controls. +5. **Can app deploy scripts compromise root?** The new receiver itself runs app code under a dedicated non-root account and does not grant sudo. Root compromise is not demonstrated by the code path. Production safety depends on not adding broad sudo and on isolating service secrets and host processes. +6. **Does it support multiple GitHub organizations?** The stable repository ID and standard GitHub claims are organization-neutral, and a public reusable workflow can be called cross-org if the caller's Actions policy permits. Real org policy and OIDC were not tested. Only PIT is mapped. +7. **Will the server workflow work with existing app permissions?** Not established. PIT currently runs deployments as root; its runtime, service, Nginx/TLS, environment file, and checkout permissions need migration. No production account was tested. +8. **Does it remove permanent per-app SSH keys?** Yes, the new design requires no application SSH private key. During migration, existing keys must remain until the new path succeeds; PIT still uses `PIT_DEPLOY_KEY` in its legacy workflow. +9. **Can the CA rotate without breaking deploys?** OpenBao supports multiple issuers, and dual public-key trust can enable overlap. A production rotation was not performed. A staged, documented runbook was added; rehearse it before rollout. +10. **Can failures be diagnosed and recovered?** Test errors fail closed and include service status/receiver/gate errors; the new docs cover logs, revocation and forward-revert recovery. Production audit persistence, restore, alerting, access retention, and recovery were not tested. + +## Original PIT SSH failure + +GitHub run `41184952985` confirms the legacy workflow attempted `root@dev2.profullstack.com` with `deploy $SHA` and received `Permission denied (publickey)` (SSH exit 255). The reported command `grep -n 'pit-world-deploy' /root/.ssh/authorized_keys` found no matching comment/string. That establishes only that no matching line contained that string; it does not establish whether the configured key was absent, mismatched, revoked, sourced from another `AuthorizedKeysFile`/command, or whether another SSH authorization mechanism applied. + +The audit credentials could not inspect PIT repository Actions secrets or the production host's SSH configuration/logs. Therefore the exact cause is **undetermined**. Do not remove or rotate the old key during migration; inspect the secret name/value metadata without exposing the value, effective SSHD config, root's `AuthorizedKeysFile`, key fingerprints, and server auth logs with the host operator. + +## Unverified infrastructure assumptions and production blockers + +- OpenBao production installation, TLS endpoint, HA/storage, init/unseal/recovery controls, backup restore, audit sink, GitHub OIDC trust, repo roles, signer mount, and central CA do not exist or were not verified. +- GitHub's actual OIDC claims for a reusable workflow were not captured. No production `id-token: write` execution has authenticated to OpenBao. +- No verified host keys exist in `ssh/known_hosts`; no target server certificate auth, user account, principal file, restricted SSHD block, or receiver install was checked on dev2. +- PIT's target is disabled. No real deployment, service restart, health failure, or rollback recovery has been tested. +- The fleet inventory is unverified and contains no authoritative mappings. Do not generate roles/accounts or enable targets from the properties list alone. +- Central `main` is unprotected, though the audit branch requires exact workflow SHA pinning; caller CI dependencies are not enforced by the central workflow. +- No production evidence confirms all existing deployment entry points use the host-wide gate. Only gated receiver deployments are bounded by this implementation. +- CA backup/restore, revocation, audit retention/alerting and rotation need operator-owned runbooks and rehearsal. + +## Operational runbook + +### Before enrolling any repository + +1. Protect the central deployment branch and app deploy branches. Pin the reviewed reusable workflow to an approved immutable SHA; enroll that same SHA in OpenBao. Confirm the helper checkout uses the called workflow's own repository and SHA. +2. Provision OpenBao with supported persistent storage, TLS, separated unseal/recovery control, encrypted snapshots, restore testing, monitoring, and declarative durable audit sinks. Keep CA private material inside OpenBao. +3. Configure the GitHub OIDC issuer/audience and a repository-specific JWT role. Capture a real test-run claim set securely and verify exact `repository`, `repository_id`, `ref`, `event_name`, `job_workflow_ref`, `job_workflow_sha`, and `aud` before issuing any production signing token. +4. Verify the repository, branch protection, target host, DNS, health URL, app owner, service account, runtime, secrets, restart, and rollback plan. Create an explicit disabled target and review it. Never infer mappings from a domain name. +5. Provision the root-owned deploy gate with `bin/setup-deploy-gate` and every account on that host. Confirm every legacy and new deploy path shares it. Test simultaneous runs and low-memory behavior. +6. Provision one non-root per-repository account, root-owned enrollment and principal files, restricted `ForceCommand` configuration, checkout ownership, and only narrowly scoped service helpers. Preserve admin access and all current working authorization. +7. Pin and verify server host keys out-of-band. Never use `ssh-keyscan` in GitHub Actions or disable checking. +8. Test login, certificate principal, forced command, approved commit, wrong repo, wrong branch, rollback, arbitrary command, gate failure, install failure, health failure, and recovery using isolated credentials and hosts first. + +### Revoke one repository + +Set its central target to disabled; delete `auth/github-actions/role/repo-ID`; remove `fleet-deploy-repo-ID`'s signer policy (or its signer capability); then remove the host principal or disable the account if immediate cert rejection is required. Verify new login and signing fail, existing cert behavior is understood, and record the operator/action in durable audit logs. Do not assume the registry flag revokes already-issued authority. + +### Rotate the CA + +Create a new issuer; distribute and verify the new public CA alongside the old CA; validate SSHD config and test both issuers on a canary; switch signing to the new issuer; monitor; retain old trust for the maximum certificate lifetime and rollback window; then remove old trust and verify rejection. Keep an administrative session and tested old-signer rollback throughout. Do not rotate production credentials as part of ordinary tests. + +### Recover a failed deploy + +Use the failed GitHub job SHA and timestamp to correlate OpenBao audit, host auth, receiver, deploy-gate, app installer/service logs, and health result. Restore service with a reviewed forward revert commit that passes the same gates. If forward deployment is impossible, follow an explicitly authorized operator procedure, preserve audit evidence, and do not bypass the gate or grant the app user unrestricted sudo. Verify the health endpoint from a fresh external request after recovery. + +## PIT migration plan + +Keep the target disabled until each step is signed off by the operator: + +1. Back up the current deployment configuration and verify administrative recovery access. +2. Provision production OpenBao securely, validate real GitHub OIDC claims, configure durable audit, and create a disposable test CA before production CA setup. +3. Create the central CA and distribute only its public key after fingerprint verification. Configure dev2 trust only with explicit operator authorization; retain existing root SSH authorization and keep an admin session open. +4. Provision PIT's restricted account and service ownership. Migrate the root-owned installer operations without broad sudo; test restart, secrets, logs, health, and rollback on an isolated host. +5. Pin verified dev2 host keys. Add the approved PIT mapping and OpenBao role only after repo ID, `main`, host, principal, health endpoint, central immutable workflow SHA, and app CI rules are reviewed. +6. Replace only PIT's legacy deploy job with the shared workflow; retain existing tests and ThreatCrush gates and require them on protected `main`. +7. Enable PIT after all checks. Deploy a reviewed commit, verify `https://pit.world/health` and the PIT Pro category/company metadata from a fresh external request, inspect service restart and logs, and execute the rollback/recovery rehearsal. +8. Revoke `PIT_DEPLOY_KEY` only after successful verification and operator sign-off. Keep a securely controlled recovery record until the migration is accepted. + +## Staged fleet rollout + +1. **Phase 1 — Isolated OIDC + OpenBao + SSH certificate tests:** use a disposable OpenBao, CA, GitHub test repository/workflow, SSH server and application; verify real GitHub claims, issue/expire/revoke certs, unauthorized repos/branches, failure cases, gate concurrency and health. +2. **Phase 2 — PIT pilot:** complete the migration steps above; use PIT as the single production canary and observe through a defined stability window. +3. **Phase 3 — Three low-risk applications:** select three properties only after owners, repository IDs, hosts, branch protections, service permissions and health endpoints are verified. Migrate and observe one at a time while preserving the shared per-host three-slot gate. +4. **Phase 4 — Controlled fleet batches:** build an authoritative repository-to-host registry. Roll out in small reviewed batches, stop on any failed check, verify each external health path, and reconcile inventory against GitHub and actual hosts. Do not assume all 88+ properties are mapped. +5. **Phase 5 — Remove obsolete per-app SSH keys:** after each app's new path is verified and recovery access tested, revoke its old static key. Track remaining legacy keys until none are in use, then remove obsolete secrets and document closure. + +## Final production-readiness verdict + +The hardened implementation is **ready for additional isolated testing**, not for fleet rollout. A successful central CI job or local OpenBao/SSH fixture is insufficient. Production readiness requires central branch protection and use of the immutable trust binding, required caller CI protections, a real GitHub OIDC exchange, production OpenBao audit/backup/recovery and CA rotation rehearsal, verified host trust and non-root service provisioning, unauthorized access tests, gate verification on the actual hosts, and a successful PIT pilot followed by fresh external health checks. diff --git a/docs/fleet-ssh-ca.md b/docs/fleet-ssh-ca.md index 3e1c7ca..e1ba29b 100644 --- a/docs/fleet-ssh-ca.md +++ b/docs/fleet-ssh-ca.md @@ -11,7 +11,7 @@ This design covers **all properties** across `profullstack`, `moshcoder` and add - `bin/oidc-ssh-deploy.mjs`: creates a fresh Ed25519 key, requests GitHub OIDC proof, exchanges it with OpenBao, gets a five-minute certificate restricted to the repository's login principal, deploys only the GitHub push SHA and checks health. - `bin/pfdeploy-ssh-receiver`: root-owned SSHD ForceCommand for dedicated unprivileged `pfdeploy-REPO_ID` users. The server verifies the Git remote and that the SHA belongs to the enrolled branch before checking it out and running the app's existing `bin/install.sh` through the existing `deploy-gate`. -**One CA private key** stays in OpenBao; each GitHub job generates a throwaway, independently signed SSH key, not a shared reusable credential. Revocation is via disabling its JWT role or repo registry entry (certificates already issued expire in five minutes). One central SSH CA means rotation requires dual-trusting the old and new public CA briefly on servers; OpenBao supports CA issuers for rotation. +**One CA private key** stays in OpenBao; each GitHub job generates a throwaway, independently signed SSH key, not a shared reusable credential. The certificate's `valid_before` is five minutes after issuance; OpenBao sets `valid_after` 30 seconds earlier to allow for clock skew. To revoke an app, disable its registry entry, delete its JWT role **and** delete its repo signing policy. Deleting only the JWT role blocks new logins but an already-issued OpenBao token can still sign until its policy is removed or the token expires. Existing SSH certificates remain valid until their short expiry; for immediate host-side revocation, remove that account's authorized principal or disable its SSH account. One central SSH CA means rotation requires dual-trusting the old and new public CA briefly on servers; use the OpenBao issuer mechanism and the rotation runbook below. ## 1. One-time OpenBao control plane (operator only) @@ -27,7 +27,23 @@ bao write ssh-client-signer/config/ca generate_signing_key=true bao read -field=public_key ssh-client-signer/config/ca > profullstack-user-ca.pub ``` -The `auth/github-actions/role/repo-ID` and `ssh-client-signer/roles/repo-ID` records must be created **only by an operator** using `bin/enroll-repo-openbao.sh OWNER/REPO REPO_ID main|master`. The script generates roles and policies from identity facts; nobody must hand-manage credentials. An authenticated OpenBao operator may later automate the script from an approved registry sync. Every role binds GitHub `repository`, immutable `repository_id`, exact `ref`, `event_name=push`, audience `profullstack-deploy` and exact central `job_workflow_ref`. Each policy can sign **only** its own SSH role, and each SSH role only its account principal. +OpenBao 2.7.2 rejects enabling an audit device through the API. Configure the audit device declaratively in the server configuration, with persistent storage and monitoring; do not treat an unaudited dev server or an API `bao audit enable` attempt as production setup. In HA, keep the audit configuration consistent on every node. Prefer two independently operated durable audit sinks so a single sink failure does not remove the only audit trail. Test that both authentication and SSH signing events reach the sinks and that audit failure alerts fire before opening GitHub OIDC access. + +This HCL shape was accepted by the disposable OpenBao 2.7.2 test. Set `file_path` to a protected persistent location and ship/monitor the resulting audit records in production: + +```hcl +audit "file" "local" { + description = "OpenBao audit trail" + options = { + file_path = "/var/log/openbao/audit.log" + log_raw = "false" + } +} +``` + +The `auth/github-actions/role/repo-ID` and `ssh-client-signer/roles/repo-ID` records must be created **only by an operator** using `bin/enroll-repo-openbao.sh OWNER/REPO REPO_ID main|master WORKFLOW_SHA`. The workflow SHA must be the reviewed full commit SHA that callers pin in `uses: profullstack/deploy/.github/workflows/ssh-deploy.yml@WORKFLOW_SHA`. The enrollment script binds both exact `job_workflow_ref` and `job_workflow_sha`, in addition to GitHub `repository`, immutable `repository_id`, exact app `ref`, `event_name=push`, and audience `profullstack-deploy`. Each policy can sign **only** its own SSH role, and each SSH role only its account principal. + +The original implementation bound only `job_workflow_ref` to `@refs/heads/main`, allowing workflow code on that mutable branch to change without changing the role. The audit branch fixes this: callers must pin a SHA, roles bind the ref and SHA, and the reusable workflow checks out its own helper code at `${{ job.workflow_sha }}`. Protect the central deployment branch as an additional review control before provisioning roles. Caller repositories must protect their deploy branch with required test and security checks; a SHA claim does not prove branch protections are enabled. **Do not** configure a single wildcard JWT policy that can sign arbitrary SSH principals. That would turn one compromised repo into access to the entire fleet. @@ -47,8 +63,13 @@ printf 'REPOSITORY=moshcoder/pit.world\nBRANCH=main\nCHECKOUT=/opt/pit.world\n' sudo tee /etc/profullstack/deploy/pfdeploy-1423083790.conf >/dev/null sudo chown root:root /etc/ssh/auth_principals/pfdeploy-1423083790 /etc/profullstack/deploy/pfdeploy-1423083790.conf sudo chmod 644 /etc/ssh/auth_principals/pfdeploy-1423083790 /etc/profullstack/deploy/pfdeploy-1423083790.conf +sudo install -m 755 bin/deploy-gate /usr/local/bin/deploy-gate +sudo install -m 755 bin/setup-deploy-gate /usr/local/sbin/setup-deploy-gate +sudo /usr/local/sbin/setup-deploy-gate pfdeploy pfdeploy-1423083790 ``` +The gate provisioning command creates a root-owned sticky directory, three group-shared lock files and a reboot-persistent `systemd-tmpfiles` rule. Run it before enabling the account. Include every deploy account on that host as an argument. It fails closed if the old lock path is unsafe; drain existing deployments and inspect that path before replacing it. Do not grant deploy users ownership of the lock directory or its slot files. + Configure OpenSSH, keeping these global directives **outside** of any Match section, and the Match block last in `sshd_config`. Preserve your existing admin SSH access, do not disable it during migration: ```sshconfig @@ -70,13 +91,27 @@ Match User pfdeploy-* Before reload, verify `sudo sshd -t` and `sudo sshd -T -C user=pfdeploy-1423083790,host=dev2.profullstack.com,addr=127.0.0.1`. **Do not** overwrite an existing SSHD configuration blindly. Keep an administrative shell open during the transition. Provision the checkout for the unprivileged deploy account and audit `bin/install.sh` and any necessary **narrow sudo permissions**; the current PIT installation runs as root, so **migration of its filesystem/service permissions is required** before enabling this pilot. Do not grant general passwordless root or a shell to deploy accounts. -Each SSH certificate principal is `pfdeploy-REPO_ID`, and only the matching Unix account accepts it. After authentication, the `ForceCommand` ignores arbitrary commands and accepts only `deploy FULL_40_CHARACTER_SHA`. It checks the Git remote and that the target commit is on the approved `origin/main` or `origin/master`. This keeps repo permissions separate without separate private keys. +Each SSH certificate principal is `pfdeploy-REPO_ID`, and only the matching Unix account accepts it. The checkout and its `.git` directory must be owned by that account; every parent directory must be canonical, root-owned, and not group/world-writable. After authentication, the `ForceCommand` ignores arbitrary commands and accepts only `deploy FULL_40_CHARACTER_SHA`. It checks the Git remote and that the target commit is on the approved `origin/main` or `origin/master`. This keeps repo permissions separate without separate private keys. + +## 3. CA rotation, revocation and recovery + +1. Create a second OpenBao SSH CA issuer and record its public key and fingerprint through the operator's authenticated management channel. Keep the current issuer active. +2. Install the new **public** CA alongside the old CA on every enrolled host. Validate file ownership, `sshd -t`, and the effective `TrustedUserCAKeys` configuration before any reload. Keep a tested administrative session open; production SSHD changes require operator authorization. +3. Reload SSHD in a controlled canary, then test new certificates from the new issuer and existing certificates from the old issuer. Do not delete the old trust yet. +4. Switch the OpenBao signing role/workflow to the new issuer. Monitor audit records, certificate authentication and app health. Keep a documented rollback that restores the old signer while both public keys remain trusted. +5. After the maximum outstanding certificate lifetime plus the rollback window, remove the old public CA from hosts and retire the old issuer. Verify the old CA can no longer authenticate. + +For repo revocation, set `enabled=false`, delete `auth/github-actions/role/repo-ID`, and delete `fleet-deploy-repo-ID` (or remove its signer capability). Deleting the role alone does not invalidate a token already issued from that role. Removing the policy blocks that token from signing; already-issued SSH certificates remain usable until expiry unless the account principal is also removed or disabled on the host. Record the reason, operator, timestamps and affected repo in the audit trail. + +Recovery readiness requires an encrypted, access-controlled OpenBao storage snapshot, protected recovery/unseal material with separated custodians, and a tested restore that proves the SSH CA private material, JWT configuration, roles, policies and audit configuration all survive recovery. Do not assume a public CA key backup is enough: it cannot sign certificates. Never export the private CA as part of ordinary backup or rotation. + +If deployment fails, inspect the GitHub job summary and exit code, OpenBao audit/auth logs, SSHD authentication logs and receiver/deploy-gate logs. A missing or invalid host key, unavailable signing API, rejected certificate, unsafe gate, low memory, failed install, or failed health check must leave the workflow failed. The receiver rejects rollbacks; use a reviewed forward revert commit on the approved branch after CI, or follow an explicitly authorized operator recovery procedure. Do not work around a failed gate or grant broad sudo to recover service. -## 3. Host-key trust is separate from user CA +## 4. Host-key trust is separate from user CA SSH **user certificates** authenticate GitHub to your server. SSH **host keys** authenticate the server to GitHub. Do not confuse them. Pin the servers' verified host keys centrally in `ssh/known_hosts`, or deploy a separate SSH host CA and trust it there with an `@cert-authority` entry. Verify host CA fingerprints out-of-band. Never disable SSH host-key checking or run unauthenticated `ssh-keyscan` inside CI. -## 4. Every application has the same tiny workflow call +## 5. Every application has the same tiny workflow call Your app workflow retains its existing project-specific test and security jobs, and gates the shared deploy on them: @@ -100,20 +135,20 @@ jobs: deploy: if: github.event_name == 'push' needs: [test, security] - uses: profullstack/deploy/.github/workflows/ssh-deploy.yml@main + uses: profullstack/deploy/.github/workflows/ssh-deploy.yml@0123456789abcdef0123456789abcdef01234567 # replace with the reviewed commit SHA permissions: contents: read id-token: write ``` -The `echo` commands in the **example above are placeholders**. Do not replace any real CI with those placeholders. Keep your existing tests, build, type checks, and ThreatCrush scan. In production pin the reusable workflow to a reviewed release commit/tag and protect it from unreviewed changes. If deploying across organizations, grant access to the public shared workflow; its OpenBao OIDC policy independently approves the actual calling repository and workflow. +The `echo` commands and SHA in the **example above are placeholders**. Do not replace any real CI with those placeholders. Keep your existing tests, build, type checks, and ThreatCrush scan. Use the same reviewed commit SHA in the caller workflow and in the repo's OpenBao enrollment; changing the central SHA requires a reviewed role update. Protect central and app deploy branches with review/check rules. If deploying across organizations, ensure the caller organization's Actions policy permits the public shared workflow; its OpenBao OIDC policy independently approves the actual calling repository and workflow. There are **no repository secrets**, no PEM files in CI, and no target host settings in the individual application's workflow. Only `needs` and the reusable workflow reference. GitHub issues OIDC credentials that expire and OpenBao signs the generated public SSH key. -## 5. Migrating PIT and then the fleet +## 6. Migrating PIT and then the fleet 1. Confirm a backed-up, TLS-protected OpenBao and GitHub OIDC verification work. -2. Bind `moshcoder/pit.world` repo ID `1423083790` with `bin/enroll-repo-openbao.sh moshcoder/pit.world 1423083790 main`. +2. Select the reviewed immutable central workflow commit SHA and use that same SHA in the PIT caller workflow. Bind PIT repo ID `1423083790` with `bin/enroll-repo-openbao.sh moshcoder/pit.world 1423083790 main WORKFLOW_SHA`. 3. Install the one public SSH user CA on dev2, provision the dedicated PIT account, repository checkout ownership and any minimal service permissions. Check SSHD and audit logs. 4. Pin the host key in `ssh/known_hosts` and set `fleet/config.json`'s OpenBao URL. 5. Change PIT's central `fleet/targets.json` entry to `enabled=true` only **after** all checks pass. Change only PIT's deploy job to the reusable workflow; tests and ThreatCrush continue to gate the deployment. diff --git a/test/deploy.test.js b/test/deploy.test.js index 876a5cd..340da87 100644 --- a/test/deploy.test.js +++ b/test/deploy.test.js @@ -1,14 +1,28 @@ import { test } from 'node:test'; import assert from 'node:assert/strict'; import { spawn, spawnSync } from 'node:child_process'; -import { mkdtempSync, readFileSync, writeFileSync } from 'node:fs'; +import { chmodSync, existsSync, mkdirSync, mkdtempSync, readFileSync, symlinkSync, unlinkSync, writeFileSync } from 'node:fs'; import { tmpdir } from 'node:os'; import { join } from 'node:path'; import { batch, freeGb, gatePath, runCommand, shellQuote } from '../src/index.js'; const scratch = () => mkdtempSync(join(tmpdir(), 'pfdeploy-')); -const run = (args, env = {}) => - new Promise((resolve) => { +const prepareLocks = (dir, slots = 3) => { + if (!existsSync(dir)) mkdirSync(dir, { mode: 0o700 }); + for (let i = 0; i < slots; i++) { + const path = join(dir, `slot-${i}`); + if (!existsSync(path)) writeFileSync(path, '', { mode: 0o600, flag: 'wx' }); + } + chmodSync(dir, 0o700); +}; +const run = (args, env = {}) => { + const lockDir = env.DEPLOY_LOCK_DIR; + if (lockDir) { + const slotIndex = args.indexOf('--slots'); + const slots = slotIndex >= 0 ? Number(args[slotIndex + 1]) : Number(env.DEPLOY_SLOTS || 3); + prepareLocks(lockDir, slots); + } + return new Promise((resolve) => { const child = spawn('bash', [gatePath, ...args], { env: { ...process.env, DEPLOY_MIN_FREE_GB: '0', DEPLOY_POLL: '0.2', ...env }, stdio: ['ignore', 'pipe', 'pipe'] }); let out = ''; let err = ''; @@ -16,6 +30,7 @@ const run = (args, env = {}) => child.stderr.on('data', (d) => (err += d)); child.on('close', (code) => resolve({ code, out, err })); }); +}; test('the gate never lets more than --slots deploys run at once', async () => { const dir = scratch(); @@ -24,7 +39,7 @@ test('the gate never lets more than --slots deploys run at once', async () => { // Each job appends +1 when it starts and -1 when it ends; we replay the log for the peak. const job = `echo +1 >> ${log}; sleep 0.6; echo -1 >> ${log}`; const started = Date.now(); - const results = await Promise.all(Array.from({ length: 6 }, (_, i) => run(['--slots', '2', '--name', `site${i}`, '--', 'bash', '-c', job], { DEPLOY_LOCK_DIR: join(dir, 'locks') }))); + const results = await Promise.all(Array.from({ length: 6 }, (_, i) => run(['--slots', '3', '--name', `site${i}`, '--', 'bash', '-c', job], { DEPLOY_LOCK_DIR: join(dir, 'locks') }))); assert.ok(results.every((r) => r.code === 0), results.map((r) => r.err).join('\n')); let now = 0; let peak = 0; @@ -32,10 +47,10 @@ test('the gate never lets more than --slots deploys run at once', async () => { now += Number(line); peak = Math.max(peak, now); } - assert.equal(peak, 2); - // Six jobs, two at a time, 0.6 s each: at least three rounds. - assert.ok(Date.now() - started >= 1700, `took ${Date.now() - started} ms`); - assert.ok(results.every((r) => /took deploy slot [01] of 2/.test(r.err)), results.map((r) => r.err).join('|')); + assert.equal(peak, 3); + // Six jobs against the production default of three slots run in two rounds. + assert.ok(Date.now() - started >= 1000, `took ${Date.now() - started} ms`); + assert.ok(results.every((r) => /took deploy slot [012] of 3/.test(r.err)), results.map((r) => r.err).join('|')); }) test('exit codes pass through; a nested gate never waits on itself; status names the holder', async () => { @@ -66,7 +81,7 @@ test('a daemon the deploy starts does not keep the slot', async () => { assert.ok(Date.now() - t < 1500, 'the second deploy got the slot at once'); }) -test('a slot that never frees times out with 75, or runs anyway when asked', async () => { +test('a slot that never frees times out with 75 even when the old bypass flag is set', async () => { const dir = scratch(); const env = { DEPLOY_LOCK_DIR: join(dir, 'locks') }; const holder = run(['--slots', '1', '--', 'sleep', '8'], env); @@ -75,11 +90,56 @@ test('a slot that never frees times out with 75, or runs anyway when asked', asy assert.equal(late.code, 75); assert.match(late.err, /giving up/); const forced = await run(['--slots', '1', '--wait', '1', '--', 'echo', 'ran'], { ...env, DEPLOY_GATE_ON_TIMEOUT: 'run' }); - assert.equal(forced.out.trim(), 'ran'); + assert.equal(forced.code, 75); + assert.equal(forced.out.trim(), ''); spawnSync('pkill', ['-f', 'sleep 8']); await holder; }) +test('unsafe lock paths and symlinked slots fail closed', async () => { + const dir = scratch(); + const occupied = join(dir, 'not-a-directory'); + writeFileSync(occupied, 'preserve'); + const badPath = spawnSync('bash', [gatePath, '--slots', '1', '--', 'echo', 'ran'], { + env: { ...process.env, DEPLOY_LOCK_DIR: occupied, DEPLOY_MIN_FREE_GB: '0' }, encoding: 'utf8', + }); + assert.equal(badPath.status, 75); + assert.equal(badPath.stdout.trim(), ''); + assert.equal(readFileSync(occupied, 'utf8'), 'preserve'); + + const lockDir = join(dir, 'links'); + prepareLocks(lockDir, 1); + const target = join(dir, 'root-owned-target'); + writeFileSync(target, 'preserve'); + unlinkSync(join(lockDir, 'slot-0')); + symlinkSync(target, join(lockDir, 'slot-0')); + const linked = spawnSync('bash', [gatePath, '--slots', '1', '--', 'echo', 'ran'], { + env: { ...process.env, DEPLOY_LOCK_DIR: lockDir, DEPLOY_MIN_FREE_GB: '0' }, encoding: 'utf8', + }); + assert.equal(linked.status, 75); + assert.equal(linked.stdout.trim(), ''); + assert.equal(readFileSync(target, 'utf8'), 'preserve'); +}); + +test('low-memory timeout rejects the build and releases its slot', async () => { + const dir = scratch(); + const env = { DEPLOY_LOCK_DIR: join(dir, 'locks'), DEPLOY_MIN_FREE_GB: '999999', DEPLOY_MEM_WAIT: '0' }; + const r = await run(['--slots', '1', '--', 'echo', 'ran'], env); + assert.equal(r.code, 75); + assert.equal(r.out.trim(), ''); + assert.match(r.err, /refusing to start a build/); + const status = spawnSync('bash', [gatePath, 'status', '--slots', '1'], { env: { ...process.env, ...env }, encoding: 'utf8' }); + assert.match(status.stdout, /slot 0 {2}free/); +}); + +test('a caller cannot spoof the nested-gate marker', async () => { + const dir = scratch(); + const r = await run(['--slots', '1', '--', 'echo', 'ran'], { DEPLOY_LOCK_DIR: join(dir, 'locks'), DEPLOY_GATED: '1' }); + assert.equal(r.code, 0, r.err); + assert.equal(r.out.trim(), 'ran'); + assert.match(r.err, /took deploy slot/); +}); + test('batch: at most `concurrency` at once, in order, and stops starting after a failure', async () => { let running = 0; let peak = 0; diff --git a/test/fleet-shell.test.js b/test/fleet-shell.test.js index ab1cbb4..d580017 100644 --- a/test/fleet-shell.test.js +++ b/test/fleet-shell.test.js @@ -1,11 +1,38 @@ import { describe, it } from 'node:test' import { execFileSync } from 'node:child_process' -import { resolve } from 'node:path' +import assert from 'node:assert/strict' +import { mkdtempSync, readFileSync, rmSync, writeFileSync } from 'node:fs' +import { tmpdir } from 'node:os' +import { join, resolve } from 'node:path' describe('shared fleet deployment scripts', () => { - for (const name of ['bin/enroll-repo-openbao.sh', 'bin/pfdeploy-ssh-receiver']) { + for (const name of ['bin/enroll-repo-openbao.sh', 'bin/pfdeploy-ssh-receiver', 'bin/setup-deploy-gate', 'bin/deploy-gate']) { it('parses safely in bash: ' + name, () => { execFileSync('bash', ['-n', resolve(name)]) }) } + + it('binds OpenBao enrollment to the pinned workflow SHA and rejects malformed SHAs', () => { + const dir = mkdtempSync(join(tmpdir(), 'pfdeploy-enroll-test-')) + const bao = join(dir, 'bao') + const log = join(dir, 'bao.log') + const workflowSha = '0123456789abcdef0123456789abcdef01234567' + writeFileSync(bao, `#!/usr/bin/env bash +set -euo pipefail +printf '%s\\n' "$*" >> "$BAO_TEST_LOG" +if [[ "$1" == write && "$2" == "auth/github-actions/role/repo-1423083790" ]]; then cat >> "$BAO_TEST_LOG"; fi +`, { mode: 0o755 }) + const env = { ...process.env, PATH: `${dir}:${process.env.PATH}`, BAO_TEST_LOG: log } + try { + execFileSync('bash', [resolve('bin/enroll-repo-openbao.sh'), 'moshcoder/pit.world', '1423083790', 'main', workflowSha], { env }) + const output = readFileSync(log, 'utf8') + assert.ok(output.includes(`"job_workflow_ref": ["profullstack/deploy/.github/workflows/ssh-deploy.yml@${workflowSha}"]`)) + assert.ok(output.includes(`"job_workflow_sha": ["${workflowSha}"]`)) + rmSync(log) + assert.throws(() => execFileSync('bash', [resolve('bin/enroll-repo-openbao.sh'), 'moshcoder/pit.world', '1423083790', 'main', 'main'], { env, stdio: 'ignore' })) + assert.throws(() => readFileSync(log, 'utf8')) + } finally { + rmSync(dir, { recursive: true, force: true }) + } + }) }) diff --git a/test/oidc-ssh.test.js b/test/oidc-ssh.test.js index 4177f98..ea37cf3 100644 --- a/test/oidc-ssh.test.js +++ b/test/oidc-ssh.test.js @@ -1,6 +1,8 @@ import { describe, it } from 'node:test' import assert from 'node:assert/strict' -import { validateContext, resolveFleetTarget, AUDIENCE } from '../bin/oidc-ssh-deploy.mjs' +import { createServer } from 'node:http' +import { once } from 'node:events' +import { validateContext, resolveFleetTarget, normalizeSignedCertificate, verifyHealth, AUDIENCE } from '../bin/oidc-ssh-deploy.mjs' const valid = { GITHUB_REPOSITORY_ID: '1423083790', @@ -38,6 +40,17 @@ describe('central fleet OIDC deployment context', () => { } }) +describe('OpenBao certificate response handling', () => { + it('writes one complete certificate line even when OpenBao returns trailing newlines', () => { + const line = 'ssh-ed25519-cert-v01@openssh.com AAAATEST' + assert.equal(normalizeSignedCertificate(`${line}\n\n`), `${line}\n`) + }) + it('rejects a multiline or non-certificate signing response', () => { + assert.throws(() => normalizeSignedCertificate('ssh-ed25519-cert-v01@openssh.com AAAA\nssh-ed25519 AAAA')) + assert.throws(() => normalizeSignedCertificate('ssh-ed25519 AAAA')) + }) +}) + describe('central repository-to-deployment mapping', () => { const targets = { '1423083790': { repository: 'moshcoder/pit.world', ref: 'refs/heads/main', host: 'dev2.profullstack.com', port: 22, health_url: 'https://pit.world/health', enabled: true } } const fleet = { openbao_url: 'https://openbao.example.net' } @@ -54,3 +67,22 @@ describe('central repository-to-deployment mapping', () => { assert.throws(() => resolveFleetTarget({ ...valid, GITHUB_REF: 'refs/heads/master' }, targets, fleet)) }) }) + +describe('post-deployment health checks', () => { + it('fails the deployment when the health endpoint is not successful', async () => { + const server = createServer((_request, response) => { + response.writeHead(503).end('unhealthy') + }) + server.listen(0, '127.0.0.1') + await once(server, 'listening') + const address = server.address() + try { + await assert.rejects( + verifyHealth(`http://127.0.0.1:${address.port}/health`, { attempts: 1 }), + /Deployment health check failed: HTTP 503/, + ) + } finally { + await new Promise(resolve => server.close(resolve)) + } + }) +}) From 1d6c2dc05a61eb556b9fe53422a75a4c2849511c Mon Sep 17 00:00:00 2001 From: Anthony Ettinger Date: Sat, 31 Oct 2026 12:14:51 +0000 Subject: [PATCH 2/2] Record hosted deployment audit checks --- docs/fleet-deployment-audit.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/fleet-deployment-audit.md b/docs/fleet-deployment-audit.md index fb8c010..721c893 100644 --- a/docs/fleet-deployment-audit.md +++ b/docs/fleet-deployment-audit.md @@ -115,11 +115,12 @@ The receiver requires a dedicated checkout and runs its `bin/install.sh` under t - Reviewed merge commit `d91e84f93230232410c231d48e47df122631fbf5`, PR #2, current workflow, registry, scripts, and tests. - Merge CI run `41192340522` succeeded on that commit. Its only job ran `npm test`; it did not run security scanning, a real GitHub OIDC exchange, or deployment. -- `npm test` on the hardened audit branch: **31 tests pass** (local final run; hosted PR CI is still required). +- Audit PR #3 hosted CI run `41200192304` passed; it ran `npm test`. Both Socket Security checks (Project Report and Pull Request Alerts) also passed. No ThreatCrush scan ran in this central repository. +- `npm test` on the hardened audit branch: **31 tests pass** locally and in the hosted PR check. - `node --check bin/oidc-ssh-deploy.mjs`: passed. - `bash -n bin/enroll-repo-openbao.sh`, `bash -n bin/pfdeploy-ssh-receiver`, `bash -n bin/deploy-gate`, and `bash -n bin/setup-deploy-gate`: passed. The scripts are also covered by the repository test suite. - ShellCheck: passed with no findings. The signal handler is called by a trap and is documented as such. -- `actionlint` was not installed in the audit environment. The GitHub workflow was reviewed manually and will be exercised by the PR's hosted check; no claim is made that it passed actionlint. +- `actionlint` was not installed in the audit environment. The reusable workflow was reviewed manually; hosted CI validates the repository's test workflow and does not exercise the deploy workflow against real GitHub OIDC or production infrastructure. ### Disposable OpenBao and SSH environment