bug(podman): polled snapshot moves a restarting sandbox from Starting to Error (ContainerExited) #5258

User Story

I run OpenShell's release qualification, which exercises the Podman driver in CI. A stop-then-start of a sandbox failed there with ContainerExited even though the restarted containers came up. I need sandbox start to be reliable on Podman so that qualification stays green and users are not told a healthy sandbox is in Error.

Problem Statement

StartSandbox on the Podman driver is a multi-step sequence: stop the supervisor, copy restart metadata and bootstrap archives, start the workload, then start the supervisor. In the failing run this took about 500ms. During it, the workload container still shows its previous exited state.

A gateway reconcile poll that lands in that window gets a snapshot that maps to Ready=False, reason=ContainerExited. The server's phase merge lets an Error snapshot replace Starting, so the sandbox moves to Error. The SDK's wait_ready stops on the first Error it sees, so the caller gets sandbox '<name>' entered error phase.

start_sandbox already records a fence for the previous run's FinishedAt (crates/openshell-driver-podman/src/driver.rs, record_previous_exit). Only the watcher's event path consults it (watcher.rs, matches_previous_exit). get_sandbox and list_sandboxes, which the reconcile loop uses, never check it.

This is the polling counterpart of the event-path race fixed for #3776. Related: #4129 (Docker, delayed exit snapshot regresses a restarted sandbox) and #4115 (lifecycle coordination across drivers).

Impact / Why This Matters

• sandbox start after sandbox stop fails intermittently on Podman. Retrying works.
• The sandbox is reported as Error while its containers are starting, so exec and other gateway-managed operations refuse it.
• It failed Pre-release qualification for v0.1.8-pre.1. The same lane passed on recent earlier runs, so the failure is intermittent and depends on when a reconcile tick lands inside the start sequence.

Acceptance Criteria

• A polled snapshot taken during start_sandbox that still shows the previous run's exit (same FinishedAt as the recorded fence) does not move the sandbox from Starting to Error.
• A genuine exit of the restarted workload is still reported.
• A unit test covers the polled path next to the existing fence tests in watcher.rs.
• conformance::persistent_workspace persistent_workspace_survives_stop_start passes repeatedly on fedora-podman-rootless.

Reproduction Steps

The failure is intermittent and timing dependent.

1. Run the gateway with the Podman driver (rootless on Fedora in CI).
2. openshell sandbox create --detach -- sleep infinity and wait for Ready.
3. openshell sandbox stop <name>, then openshell sandbox start <name>.
4. Occasionally start fails because the sandbox enters the Error phase with ContainerExited.

CI reference: Pre-release run 41843596010, job 126635171437 (conformance-integration, fedora-podman-rootless, rpm). The test failed 3.0s in; the other 14 conformance tests passed.

Environment

• OpenShell: v0.1.8-pre.1 (7cfa14f5f)
• OS: Fedora tmachine guest on a GitHub-hosted runner
• Runtime: rootless Podman, RPM installer

Logs

Redacted excerpt, sandbox test-izen9c79-kw, 2026-11-04 (UTC):

14:33:18.083 podman.stop_sandbox: Stopping sandbox container
14:33:18.178 StopSandbox request completed successfully
14:33:18.197 podman.start_sandbox: Starting sandbox container
14:33:18.411 reconcile.sandboxes: Sandbox phase changed old_phase=Starting new_phase=Error
14:33:18.411 reconcile.sandboxes: Sandbox failed to become ready reason=ContainerExited
14:33:18.707 StartSandbox request completed successfully
14:33:18.750 supervisor PushSandboxLogs allowed (replacement supervisor connected)

Test output:

wait for sandbox test-izen9c79-kw to be ready: connect error: sandbox 'test-izen9c79-kw' entered error phase