Skip to content

fix(pi): keep one pi.wake while Pi work is active - #402

Merged
eersnington merged 3 commits into
mainfrom
fix/pi-recovery-wake
Nov 17, 2026
Merged

eersnington merged 3 commits into
mainfrom
fix/pi-recovery-wake

Conversation

@eersnington

@eersnington eersnington commented Nov 16, 2026 •

Copy link
Copy Markdown
Member

Pi Durable asks its host to keep a recovery wakeup armed while harness.wake() is active, so work a crash cut off resumes (spec, Wakeups; the Durable Object host in the README). pi() scheduled pi.wake only for sleeping tasks. Keep-awake ends with the process, so an actor that crashed during a run stayed asleep until a client called it.

wakeActor now keeps one pi.wake:

const heartbeat = active ? Date.now() + RECOVERY_WAKE_MS : undefined; // 15 s
const target = min(heartbeat, at); // never later than the earliest sleep
Before After
Work is active keep-awake only, no wake one pi.wake, at most 15 s ahead
A task sleeps pi.wake at its time the same wake, at the earlier time
The process dies during a run asleep until a call the wake fires, and Pi resumes the run
A stop cuts work off resumes on the next call resumes on the wake
Work ends the wake is cancelled
  • While work runs, the wake fires on the live actor and pi.wake arms the next one.
  • A new wake is written before the old one is cancelled, so active work always has one.
  • An action waits until the wake for the work it started is written, so a crash right after the reply still resumes that work.

Supersedes #399. It keeps that PR's crash and failed-commit tests, and replaces its interval with the one wake above.


This is part 1 of 6 in a stack:

Nathaneil Flurry and others added 3 commits November 16, 2026 15:45
RivetKit's default crash policy leaves a crashed actor asleep, and
keep-awake only lasts while the process does, so an actor that crashed
mid-run stayed asleep until something called it. While Pi Durable's wake
is active, the actor now keeps a pi.wake scheduled about 15 seconds ahead
and re-arms it every 7.5 seconds, cancelling the previous one. When work
ends the wake is cancelled; a stop with work still active keeps it, so the
actor wakes by itself and Pi resumes the run.

A new test runs the agent in a child process, kills it with its engine
while a tool runs, and checks that the run finishes with no call to the
actor.
A commit that fails while a run is going, with no call in flight, must
reopen Pi Durable at once so the run finishes. In 0.5.7-rc.1 closing the
failed harness asked it for its wake, which it refuses, so the reopen
failed and the run waited for the recovery wake about 15 seconds later.
The new test has a replay-safe tool whose first run commits a document
that storage rejects, and checks that the tool reruns within 10 seconds
and the run ends done. It fails on rc.1's runtime and passes on this
stack.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Pi Durable's host contract asks a host to keep a recovery wakeup armed while
harness.wake() is active. The actor now keeps one pi.wake at the earliest
sleep, and while work is active no later than 15 s ahead. It fires on the live
actor and is armed again, so a crash leaves a wake that resumes the run.

- One scheduled wake replaces the interval and the second recovery wake.
- A call waits until the wake for the work it started is written.
- The crash test kills the child's engine by PID, so it never reuses it.
- Sleep and crash tests make no call before they check the outcome.
@github-actions

Copy link
Copy Markdown

📟 Pip-Boy · d4945d3

📦 Packages · fresh npm install resolve, no lockfile · this PR vs base

No notable install size, dependency, or entrypoint changes across 10 packages.

All 10 packages · linux-x64-gnu
package (linux-x64-gnu) @latest base this PR Δ vs base packages
@rivet-dev/pi 392.5 MB 392.5 MB 392.5 MB +5.9 KB (+0.0%) 220 (0)
@rivet-dev/sandbox-adapter 114.5 KB 114.5 KB 114.5 KB 0 1 (0)
@sandbox-agent/cli-shared 27.2 KB 27.2 KB 27.2 KB 0 1 (0)
@sandbox-agent/persist-indexeddb 12.4 KB 12.4 KB 12.4 KB 0 1 (0)
@sandbox-agent/persist-postgres 12.3 KB 12.3 KB 12.3 KB 0 1 (0)
@sandbox-agent/persist-rivet 12.2 KB 12.2 KB 12.2 KB 0 1 (0)
@sandbox-agent/persist-sqlite 12.3 KB 12.3 KB 12.3 KB 0 1 (0)
@sandbox-agent/react 41.6 MB 41.9 MB 41.9 MB 0 15 (0)
acp-http-client 6.7 MB 6.7 MB 6.7 MB 0 3 (0)
sandbox-agent 31.0 MB 31.0 MB 31.0 MB 0 8 (0)

🧹 Rust dependencies

Skipped: this PR doesn't change any Rust code or dependencies.

🦀 Binaries

Skipped: this PR doesn't change any Rust code.

⏱ Timing

job wall time slowest step
changes 19s checkout · 14s
zizmor 15s checkout · 9s
footprint 3m32s Run pnpm --dir sandbox-agent install --frozen-lockfile · 1m13s
⏭ self-test — skipped
⏭ hygiene — skipped
⏭ native · ${{ matrix.platform }} — skipped
report 21s checkout · 9s
🔭 Full CI trace for this commit

One OpenTelemetry trace of CI and Pip-Boy for d4945d3: every job and step, with the sizes above attached. Run after CI finishes.

pnpm install --frozen-lockfile --filter 'pipboy...'
pnpm --filter pipboy pipboy trace --sha d4945d31472981cbe811f2099b4b6860ad0da405

Add --otlp <url> to send it to any OTLP/HTTP collector.

Agent instruction: From a rivet-dev/agents checkout, run the commands above, then read the trace file the last command prints. List the slowest jobs and steps and explain where CI time went.

@eersnington
eersnington merged commit 2c2a64a into main Nov 17, 2026
10 checks passed
@eersnington
eersnington deleted the fix/pi-recovery-wake branch November 17, 2026 14:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants