@profullstack/deploy 0.1.0: deploy gate + batch runner

.github/workflows/ci.yml

@@ -0,0 +1,16 @@
+name: CI
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-node@v4
+        with:
+          node-version: 22
+      - run: npm test

.gitignore

@@ -0,0 +1,2 @@
+.claude/
+node_modules/

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Profullstack, Inc.
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README.md

@@ -0,0 +1,69 @@
+# @profullstack/deploy
+
+Deploy in batches, **3 at a time**, so one box never builds everything at once.
+
+On 2026-10-19 a change rolled out to 38 repos at once on our production box. Every repo installed, tested, type-checked and built together, then every merge's deploy built again; the box ran out of memory and the kernel killed the shared production Postgres. This package makes that impossible from both ends:
+
+- **`deploy-gate`** (bash + `flock`, on the box): every deploy takes one of N slots before it builds. Merges can land all at once; builds queue and run three at a time.
+- **`pfdeploy batch`** (and `batch()`, where the rollout is driven): run one command across many repos, three at a time, stopping at the first failure.
+
+Zero dependencies. Linux (`flock`, `/proc/meminfo`); the batch runner works anywhere Node 18+ does.
+
+## On the box: the gate
+
+```sh
+npx @profullstack/deploy install /usr/local/bin     # as root: one copy every user's deploys share
+deploy-gate --name crawlproof -- ./deploy-app.sh "$SHA"
+deploy-gate status
+```
+
+```
+slot 0  busy  crawlproof pid 41233 since 2026-10-19T13:20:02Z
+slot 1  busy  smshub pid 41290 since 2026-10-19T13:20:09Z
+slot 2  free
+2 of 3 busy, 31 GB available
+```
+
+How it decides:
+
+1. Take one of `DEPLOY_SLOTS` (3) lock files in `/run/lock/profullstack-deploy` (shared by root and every user). All taken: wait, saying who holds them every 30 s.
+2. Wait up to `DEPLOY_MEM_WAIT` (600 s) for `DEPLOY_MIN_FREE_GB` (8) of available memory, then go on.
+3. Run the deploy with the lock descriptor closed, so a server it starts never keeps the slot. Signals are forwarded; the exit code is the deploy's.
+4. Still no slot after `DEPLOY_WAIT` (3600 s): exit 75 (try again), or run anyway with `DEPLOY_GATE_ON_TIMEOUT=run`.
+
+A deploy already inside the gate (`DEPLOY_GATED=1`) passes straight through, so a gated script that calls another gated script never waits on itself. Without `flock` the gate runs the deploy ungated rather than blocking it.
+
+### Wiring a deploy script
+
+Put this right after `set -euo pipefail`; if the gate is not installed the script runs exactly as before:
+
+```sh
+# Three deploys at a time on this box (@profullstack/deploy).
+if [ -z "${DEPLOY_GATED:-}" ]; then
+  for gate in /usr/local/bin/deploy-gate "$HOME/.local/bin/deploy-gate"; do
+    [ -x "$gate" ] && exec "$gate" --name "<site>" -- "$0" "$@"
+  done
+fi
+```
+
+## Where the rollout runs: batches
+
+```sh
+# one change, many repos, three at a time; {} is the item, shell-quoted
+pfdeploy batch -c 3 -- 'cd ~/src/profullstack/{} && bun test && gh pr merge --squash --auto' crawlproof smshub ugig.net
+ls ~/www | pfdeploy batch -- '~/www/{}/deploy-app.sh origin/main'
+pfdeploy batch -c 3 --keep-going --min-free-gb 12 -- 'make -C {} build' < repos.txt
+```
+
+Output lines are prefixed with the item. The first failure stops new starts (running ones finish); the summary names what failed and what never started. `--keep-going` runs everything.
+
+```js
+import { batch } from '@profullstack/deploy';
+
+const results = await batch(repos, (repo) => rollOut(repo), { concurrency: 3, minFreeGb: 8 });
+// [{ item, ok, value | error, skipped }]
+```
+
+## License
+
+MIT

bin/deploy-gate

@@ -0,0 +1,156 @@
+#!/usr/bin/env bash
+# deploy-gate: at most N deploys build at once on this box, whoever starts them.
+#
+#   deploy-gate [--slots 3] [--name site] [--wait 3600] [--min-free-gb 8] -- command [args...]
+#   deploy-gate status
+#
+# Every deploy path on the box (cli-tools deploy-app.sh, OpenInstall bin/install.sh,
+# a hand-run build) wraps itself in this. A deploy takes one of N slots, a file
+# lock under DEPLOY_LOCK_DIR shared by every user, waits while all are taken,
+# then waits a little for free memory, then runs. Thirty-eight merges at once
+# become three builds at a time and a queue, instead of an out-of-memory kill of
+# the production database (dev2, 2026-10-19).
+#
+# The command runs with the lock's descriptor closed, so a daemon it starts never
+# holds the slot. A deploy already inside the gate (DEPLOY_GATED=1) passes
+# straight through, so nested scripts never wait on themselves.
+#
+# Environment (flags win): DEPLOY_SLOTS=3, DEPLOY_WAIT=3600 (seconds for a slot;
+# then exit 75, or run anyway with DEPLOY_GATE_ON_TIMEOUT=run), DEPLOY_MIN_FREE_GB=8,
+# DEPLOY_MEM_WAIT=600 (seconds to wait for memory, then go on), DEPLOY_POLL=5, DEPLOY_LOCK_DIR.
+set -uo pipefail
+
+SLOTS=${DEPLOY_SLOTS:-3}
+WAIT=${DEPLOY_WAIT:-3600}
+MIN_FREE_GB=${DEPLOY_MIN_FREE_GB:-8}
+MEM_WAIT=${DEPLOY_MEM_WAIT:-600}
+POLL=${DEPLOY_POLL:-5}
+LOCK_DIR=${DEPLOY_LOCK_DIR:-/run/lock/profullstack-deploy}
+NAME=""
+
+say() { printf '[deploy-gate] %s\n' "$*" >&2; }
+usage() { sed -n '4,6p' "$0" | sed 's/^# \{0,1\}//' >&2; exit 64; }
+
+lock_dir() {
+  if [ ! -d "$LOCK_DIR" ]; then
+    (umask 000; mkdir -p "$LOCK_DIR") 2>/dev/null || { say "cannot make $LOCK_DIR"; return 1; }
+    chmod 1777 "$LOCK_DIR" 2>/dev/null || true
+  fi
+}
+
+slot_file() { printf '%s/slot-%s' "$LOCK_DIR" "$1"; }
+
+# The holder line written into a slot: who, since when.
+holder() { head -c 200 "$(slot_file "$1")" 2>/dev/null | head -1; }
+
+status() {
+  lock_dir || exit 1
+  local i f busy=0
+  for ((i = 0; i < SLOTS; i++)); do
+    f=$(slot_file "$i")
+    [ -e "$f" ] || (umask 000; : > "$f")
+    if flock -n "$f" true 2>/dev/null; then
+      printf 'slot %d  free\n' "$i"
+    else
+      busy=$((busy + 1))
+      printf 'slot %d  busy  %s\n' "$i" "$(holder "$i")"
+    fi
+  done
+  printf '%d of %d busy, %s GB available\n' "$busy" "$SLOTS" "$(free_gb)"
+}
+
+free_gb() { awk '/^MemAvailable:/ { printf "%d", $2 / 1048576 }' /proc/meminfo 2>/dev/null || echo 0; }
+
+[ "${1:-}" = status ] && { shift; [ "${1:-}" = --slots ] && SLOTS=$2; status; exit 0; }
+
+while [ $# -gt 0 ]; do
+  case "$1" in
+    --slots) SLOTS=$2; shift 2 ;;
+    --name) NAME=$2; shift 2 ;;
+    --wait) WAIT=$2; shift 2 ;;
+    --min-free-gb) MIN_FREE_GB=$2; shift 2 ;;
+    --) shift; break ;;
+    -h|--help) usage ;;
+    *) break ;;
+  esac
+done
+[ $# -gt 0 ] || usage
+case "$SLOTS" in ''|*[!0-9]*|0) say "--slots must be a positive number"; exit 64 ;; esac
+
+# Already inside the gate: never take a second slot.
+if [ "${DEPLOY_GATED:-}" = 1 ]; then exec "$@"; fi
+NAME=${NAME:-$(basename "$PWD")}
+export DEPLOY_GATED=1
+
+if ! command -v flock >/dev/null || ! lock_dir; then
+  say "no flock or lock dir: running ungated"
+  exec "$@"
+fi
+
+# Take a slot. Start at a random slot so waiters spread out; poll every DEPLOY_POLL (5) s.
+fd=""
+start=$(date +%s)
+last_note=0
+while :; do
+  first=$((RANDOM % SLOTS))
+  for ((n = 0; n < SLOTS; n++)); do
+    i=$(((first + n) % SLOTS))
+    f=$(slot_file "$i")
+    [ -e "$f" ] || (umask 000; : > "$f") 2>/dev/null
+    { exec {fd}<>"$f"; } 2>/dev/null || continue
+    if flock -n "$fd"; then
+      slot=$i
+      break 2
+    fi
+    exec {fd}>&-
+    fd=""
+  done
+  now=$(date +%s)
+  if [ $((now - start)) -ge "$WAIT" ]; then
+    if [ "${DEPLOY_GATE_ON_TIMEOUT:-fail}" = run ]; then
+      say "waited ${WAIT}s for a slot; running anyway (DEPLOY_GATE_ON_TIMEOUT=run)"
+      exec "$@"
+    fi
+    say "waited ${WAIT}s for one of $SLOTS deploy slots; giving up (exit 75). Busy: $(for ((i = 0; i < SLOTS; i++)); do printf '%s; ' "$(holder "$i")"; done)"
+    exit 75
+  fi
+  if [ $((now - last_note)) -ge 30 ]; then
+    say "$NAME waiting for a deploy slot ($SLOTS busy): $(for ((i = 0; i < SLOTS; i++)); do printf '%s; ' "$(holder "$i")"; done)"
+    last_note=$now
+  fi
+  sleep "$POLL"
+done
+
+# Say who holds it (truncate, then write: the lock is on the descriptor, not the content).
+printf '%s pid %s since %s\n' "$NAME" "$$" "$(date -u +%FT%TZ)" > "$(slot_file "$slot")" 2>/dev/null || true
+waited=$(($(date +%s) - start))
+say "$NAME took deploy slot $slot of $SLOTS$([ "$waited" -gt 0 ] && printf ' after %ss' "$waited")"
+
+# Then a little room: never start a build into a box that is already out of memory.
+mem_start=$(date +%s)
+while [ "$(free_gb)" -lt "$MIN_FREE_GB" ]; do
+  if [ $(($(date +%s) - mem_start)) -ge "$MEM_WAIT" ]; then
+    say "only $(free_gb) GB available after ${MEM_WAIT}s; going on"
+    break
+  fi
+  say "$(free_gb) GB available, want $MIN_FREE_GB; waiting"
+  sleep 15
+done
+
+# Run with the lock descriptor closed, forwarding signals, and release when it ends.
+child=""
+forward() { [ -n "$child" ] && kill "-$1" "$child" 2>/dev/null; }
+trap 'forward TERM' TERM
+trap 'forward INT' INT
+trap 'forward HUP' HUP
+"$@" {fd}>&- &
+child=$!
+while :; do
+  wait "$child"
+  rc=$?
+  # wait returns early (>128) when a trapped signal arrives; keep waiting for the child.
+  kill -0 "$child" 2>/dev/null || break
+done
+: > "$(slot_file "$slot")" 2>/dev/null || true
+exec {fd}>&-
+exit "$rc"

bin/pfdeploy.js

@@ -0,0 +1,102 @@
+#!/usr/bin/env node
+import { spawnSync } from 'node:child_process';
+import { copyFileSync, chmodSync, readFileSync } from 'node:fs';
+import { join } from 'node:path';
+import { batch, DEFAULT_CONCURRENCY, gatePath, runCommand } from '../src/index.js';
+
+const HELP = `pfdeploy: deploy in batches, ${DEFAULT_CONCURRENCY} at a time.
+
+  pfdeploy batch [-c 3] [--keep-going] [--min-free-gb 8] -- '<command with {}>' item1 item2 ...
+      run the command once per item, at most -c at once ({} is the quoted item;
+      without {} the item is appended). Items may also come on stdin, one per line.
+      Stops starting new items after the first failure unless --keep-going.
+  pfdeploy gate [--slots 3] [--name site] -- <command...>
+      take one of the box's deploy slots, then run the command (bin/deploy-gate).
+  pfdeploy status [--slots 3]     which deploy slots are busy, and by whom
+  pfdeploy install [dir]          copy deploy-gate to dir (default /usr/local/bin)
+
+Examples:
+  pfdeploy batch -c 3 -- 'cd ~/src/profullstack/{} && bun test && gh pr merge --squash' crawlproof smshub ugig
+  ls ~/www | pfdeploy batch -c 3 -- '~/www/{}/deploy-app.sh origin/main'
+`;
+
+const args = process.argv.slice(2);
+const command = args.shift();
+
+const gate = (rest) => {
+  const r = spawnSync('bash', [gatePath, ...rest], { stdio: 'inherit' });
+  process.exit(r.status ?? 1);
+};
+
+if (!command || command === '-h' || command === '--help' || command === 'help') {
+  process.stdout.write(HELP);
+  process.exit(command ? 0 : 64);
+}
+if (command === 'version' || command === '--version') {
+  process.stdout.write(`${JSON.parse(readFileSync(new URL('../package.json', import.meta.url), 'utf8')).version}\n`);
+  process.exit(0);
+}
+if (command === 'gate') gate(args);
+if (command === 'status') gate(['status', ...args]);
+if (command === 'install') {
+  const dir = args[0] ?? '/usr/local/bin';
+  const to = join(dir, 'deploy-gate');
+  try {
+    copyFileSync(gatePath, to);
+    chmodSync(to, 0o755);
+    process.stdout.write(`installed ${to}\n`);
+    process.exit(0);
+  } catch (e) {
+    process.stderr.write(`pfdeploy install: ${e.message}${e.code === 'EACCES' ? ' (run with sudo)' : ''}\n`);
+    process.exit(1);
+  }
+}
+if (command !== 'batch') {
+  process.stderr.write(`pfdeploy: unknown command "${command}"\n\n${HELP}`);
+  process.exit(64);
+}
+
+let concurrency = DEFAULT_CONCURRENCY;
+let keepGoing = false;
+let minFreeGb = 0;
+while (args.length && args[0] !== '--') {
+  const a = args.shift();
+  if (a === '-c' || a === '--concurrency') concurrency = Number(args.shift());
+  else if (a === '--keep-going' || a === '-k') keepGoing = true;
+  else if (a === '--min-free-gb') minFreeGb = Number(args.shift());
+  else {
+    process.stderr.write(`pfdeploy batch: unknown option ${a}\n`);
+    process.exit(64);
+  }
+}
+if (args[0] === '--') args.shift();
+const template = args.shift();
+if (!template || !(concurrency >= 1)) {
+  process.stderr.write(HELP);
+  process.exit(64);
+}
+let items = args;
+if (!items.length && !process.stdin.isTTY) items = readFileSync(0, 'utf8').split('\n').map((s) => s.trim()).filter(Boolean);
+if (!items.length) {
+  process.stderr.write('pfdeploy batch: no items\n');
+  process.exit(64);
+}
+
+const log = (s) => process.stderr.write(`[pfdeploy] ${s}\n`);
+log(`${items.length} item${items.length === 1 ? '' : 's'}, ${concurrency} at a time`);
+const results = await batch(items, (item) => runCommand(template, item), {
+  concurrency,
+  keepGoing,
+  minFreeGb,
+  onEvent: (e) => {
+    if (e.type === 'start') log(`start ${e.item}`);
+    if (e.type === 'done') log(`${e.ok ? 'ok   ' : 'FAIL '} ${e.item}${e.ok ? '' : `: ${e.error.message}`}`);
+    if (e.type === 'memory') log(`${e.freeGb.toFixed(1)} GB free, want ${e.wantGb}: holding ${e.item}`);
+  },
+});
+const failed = results.filter((r) => !r.ok && !r.skipped);
+const skipped = results.filter((r) => r.skipped);
+log(`done: ${results.length - failed.length - skipped.length} ok, ${failed.length} failed${skipped.length ? `, ${skipped.length} not started (after a failure; --keep-going runs them)` : ''}`);
+for (const r of failed) log(`failed: ${r.item}`);
+for (const r of skipped) log(`not started: ${r.item}`);
+process.exit(failed.length || skipped.length ? 1 : 0);