[mattj-monad]

Hardening from Node rotation left an agent permanently Stopped (ENG-11690). This is not a confirmed fix for that outage; its cause is still unknown, and the ticket stays open.

Background

On 2026-11-02 a node rotation left oncall-triage recorded as STOPPED for about 8 hours. Its Sandbox was Suspended with no sandbox-runtime-suspension-cause annotation, and no driver operation was pending. That state is what a FailClosed suspension produces: permanent, and indistinguishable from a deliberate stop. The log line that would say which path did it was on a gateway that no longer exists.

Changes

Both are in reap_stale_sandbox_runtime_bootstrap and the suspension helper in the Kubernetes driver.

1. A stale restart bootstrap is now a lost runtime. Until now, a non-create bootstrap unfinished after the 5-minute grace was suspended as FailClosed. A restart is meant to end up running, so it now suspends as RuntimeLost, and the gateway starts a fresh generation under its existing restart backoff. The workload is suspended in both cases, so an agent still never runs without its supervisor. A stale stop, or a bootstrap with no recorded operation, stays fail-closed.
2. Every fail-closed suspension logs at ERROR, with the sandbox and the bootstrap operation, so the next one says what happened. The stale-bootstrap warning also names the operation, and its text changes from suspending stale sandbox-runtime restart bootstrap to suspending stale sandbox-runtime bootstrap.

What this does and doesn't cover

This closes one path to a permanent stop. It may not be the path taken on 11-02:

• On kind, on monad.18, a restart held Pending for 330s by a cordon did not go Stopped. The gateway regenerates the restart every 2 minutes, which resets the 5-minute timer, and pm was Ready 4s after the uncordon. So this path needs regeneration to be blocked as well, for example by a pending operation or a lease gap, and that combination hasn't been reproduced.
• The other fail-closed sites (admission revalidation, an altered workload fence) are unchanged and remain candidates. They now log at ERROR.

A restart that can never become ready now retries indefinitely under backoff instead of stopping. That's the behaviour we want for agents, but it is a change from upstream's fail-closed choice for this path.

Testing

• New test: stale_restart_bootstrap_is_a_lost_runtime_and_a_stale_stop_stays_stopped.
• cargo test -p openshell-driver-kubernetes --lib: 297 passed.
• cargo clippy -p openshell-driver-kubernetes --all-targets -D warnings and cargo fmt --check: clean.
• Not tested live: the new code path hasn't fired on a cluster, because nothing has reproduced a stale restart bootstrap.

🤖 Generated with Claude Code

https://claude.ai/code/session_01XY4mbnSV5mNkx9Hjg4qhdn