Skip to content

[2/3] [core] Resume a stale parked VM over another orchestrator's events - #5470

Draft
benpankow wants to merge 4 commits into
ben/park-retained-sessionfrom
ben/park-resume-stale
Draft

benpankow wants to merge 4 commits into
ben/park-retained-sessionfrom
ben/park-resume-stale

Conversation

@benpankow

@benpankow benpankow commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Description

Stacked on #5467. There, a parked VM is dropped when another orchestrator wrote in-band since it parked (the read's seqInBand moved past the parked count), and the delivery loads and replays the whole log from cold.

That case is common once one run's messages land on different instances. I hit it with eve on Vercel running deliveries inline (#5365): each message runs in whichever server instance received it. In one 8-follow-up run, 2 of the 4 parked VMs found were stale and fell back.

The events that orchestrator wrote are in the incremental read, after the parked cursor. This PR resumes the parked VM on those events too, and falls back only when the read could not continue from the cursor (reloadedFromStart). The fence is unchanged: the delivery adopts the read's snapshot, so its next in-band write still has to match the World's count, and an orchestrator that overlaps after the read is refused there, as before.

Replaying another orchestrator's events isn't a new kind of trust. They go through the same events consumer a cold replay uses, and a parked VM that diverges on them falls back to a cold replay of the log, which decides; only that replay's divergence counts against WORKFLOW_REPLAY_DIVERGENCE_MAX_RETRIES (workflow.parked_session_diverged, with crossedWrites on the warning). The unchecked part is a warm VM writing new events from state a cold replay didn't build, and #5467 already has that whenever it resumes on a new hook_received.

Worth flagging, though not new here: a stale VM can resume while the orchestrator that wrote past it is still lingering with a newer VM. The fence lets one of them write and refuses the other, but with run-ahead on, a retriable step's body can start before its start commits, so the refused racer has run it too. That's the overlap any redelivery makes (see "The in-band fence" in the spec 9 README), and a maxRetries: 0 step never runs ahead. eve doesn't run ahead today; it would matter if it did.

How did you test your changes?

parked-session.runtime.test.ts replaces "loads and replays the whole log when another orchestrator wrote in-band meanwhile", which only bumped the World's count, with a real overlap: the test takes the VM a parked aside, so b is handled from cold, then puts the stale VM back for c. The run finishes with each step run once and no further full load. New tests:

  • the stale VM races the orchestrator still lingering past it: one wins, the other is refused, and each body runs once (a maxRetries: 0 step, and a retriable one with run-ahead off);
  • two deliveries finish out of order after a crossed resume, and the stale VM doesn't replace the newer entry (newer_parked);
  • a new deployment handles b with a different step, so the old VM diverges on its events and falls back to one cold replay, without spending the divergence budget.

src in core passes locally.

On Vercel, with eve on #5365 + #5467 + this, 8 follow-ups 35s apart on a 20-turn session resumed 7 parked VMs (4 of them stale) with no errors. The server's post.start → turn step was 402ms median, against 568ms with queued delivery and no parking. Client first output was ~760ms against ~860ms on #5365 + #5467 without this change.

PR Checklist - Required to merge

  • 📦 pnpm changeset was run to create a changelog for this PR
    • Use the correct semver bump type: patch for bug fixes, minor for new features, major for breaking changes.
    • Use pnpm changeset --empty if you are changing documentation or workbench apps
  • 🔒 DCO sign-off passes (run git commit --signoff on your commits)
  • 📝 Ping @vercel/workflow in a comment once the PR is ready, and the above checklist is complete

🤖

@changeset-bot

changeset-bot Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 0aac38f

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 18 packages
Name Type
@workflow/core Patch
@workflow/builders Patch
@workflow/cli Patch
@workflow/next Patch
@workflow/nitro Patch
@workflow/vitest Patch
@workflow/web-shared Patch
@workflow/web Patch
workflow Patch
@workflow/world-cloudflare Patch
@workflow/world-testing Patch
@workflow/astro Patch
@workflow/nest Patch
@workflow/rollup Patch
@workflow/sveltekit Patch
@workflow/vite Patch
@workflow/nuxt Patch
@workflow/world-vercel Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@vercel

vercel Bot commented Nov 16, 2026

Copy link
Copy Markdown
Contributor

You must have Developer access to commit code to Vercel Labs on Vercel. If you contact an administrator and receive Developer access, commit again to see your changes.

Learn more: https://vercel.com/docs/accounts/team-members-and-roles/access-roles#team-level-roles

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

🧪 E2E Test Results

✅ All tests passed

🛠 Infra Events (absorbed by the harness)

Platform anomalies the e2e harness detected and worked around (e.g. a run the queue never picked up, replaced by a fresh run). Clustered timestamps indicate a backend blip; a steady drip indicates a platform issue worth escalating.

  • run-pickup-stall · runs app-generated source against a registered step (tanstack-start) · at 21:53:56Z · abandoned wrun_01M801KMAKMFGWTBMXFV6KQPEM
  • cold-start-warmup · suite warmup (tanstack-start) · at 21:54:04Z · abandoned wrun_01M801KMB0CT8AZEBX5KC65XM1

E2E Test Summary

Summary
Passed Failed Skipped Total
✅ 💻 Local Development 5272 0 731 6003
✅ 📦 Local Production 5272 0 731 6003
✅ 🐘 Local Postgres 5417 0 586 6003
✅ 🪟 Windows 401 0 13 414
✅ 🌐 Cross-language Conformance 68 0 114 182
✅ dynamic-runs 0 0 0 0
✅ local-postgres-spec9 0 0 364 364
✅ local-spec9 409 0 5 414
✅ vercel-http-transport 1054 0 188 1242
✅ vercel-spec9 395 0 19 414
✅ vercel-ws-transport 712 0 116 828
Total 19000 0 2867 21867
Details by Category

✅ 💻 Local Development

App Passed Failed Skipped
✅ astro-stable-node 172 0 35
✅ astro-stable-quickjs 171 0 36
✅ express-stable-node 172 0 35
✅ express-stable-quickjs 171 0 36
✅ fastify-stable-node 172 0 35
✅ fastify-stable-quickjs 171 0 36
✅ hono-stable-node 172 0 35
✅ hono-stable-quickjs 171 0 36
✅ nest-stable-node 172 0 35
✅ nest-stable-quickjs 171 0 36
✅ nextjs-turbopack-canary-node 201 0 6
✅ nextjs-turbopack-canary-quickjs 200 0 7
✅ nextjs-turbopack-quickjs-snapshot 200 0 7
✅ nextjs-turbopack-stable-node 201 0 6
✅ nextjs-turbopack-stable-quickjs 200 0 7
✅ nextjs-webpack-canary-node 201 0 6
✅ nextjs-webpack-canary-quickjs 200 0 7
✅ nextjs-webpack-stable-node 201 0 6
✅ nextjs-webpack-stable-quickjs 200 0 7
✅ nitro-stable-node 172 0 35
✅ nitro-stable-quickjs 171 0 36
✅ nuxt-stable-node 172 0 35
✅ nuxt-stable-quickjs 171 0 36
✅ sveltekit-stable-node 191 0 16
✅ sveltekit-stable-quickjs 190 0 17
✅ tanstack-start-node 172 0 35
✅ tanstack-start-quickjs 171 0 36
✅ vite-stable-node 172 0 35
✅ vite-stable-quickjs 171 0 36

✅ 📦 Local Production

App Passed Failed Skipped
✅ astro-stable-node 172 0 35
✅ astro-stable-quickjs 171 0 36
✅ express-stable-node 172 0 35
✅ express-stable-quickjs 171 0 36
✅ fastify-stable-node 172 0 35
✅ fastify-stable-quickjs 171 0 36
✅ hono-stable-node 172 0 35
✅ hono-stable-quickjs 171 0 36
✅ nest-stable-node 172 0 35
✅ nest-stable-quickjs 171 0 36
✅ nextjs-turbopack-canary-node 201 0 6
✅ nextjs-turbopack-canary-quickjs 200 0 7
✅ nextjs-turbopack-quickjs-snapshot 200 0 7
✅ nextjs-turbopack-stable-node 201 0 6
✅ nextjs-turbopack-stable-quickjs 200 0 7
✅ nextjs-webpack-canary-node 201 0 6
✅ nextjs-webpack-canary-quickjs 200 0 7
✅ nextjs-webpack-stable-node 201 0 6
✅ nextjs-webpack-stable-quickjs 200 0 7
✅ nitro-stable-node 172 0 35
✅ nitro-stable-quickjs 171 0 36
✅ nuxt-stable-node 172 0 35
✅ nuxt-stable-quickjs 171 0 36
✅ sveltekit-stable-node 191 0 16
✅ sveltekit-stable-quickjs 190 0 17
✅ tanstack-start-node 172 0 35
✅ tanstack-start-quickjs 171 0 36
✅ vite-stable-node 172 0 35
✅ vite-stable-quickjs 171 0 36

✅ 🐘 Local Postgres

App Passed Failed Skipped
✅ astro-stable-node 177 0 30
✅ astro-stable-quickjs 176 0 31
✅ express-stable-node 177 0 30
✅ express-stable-quickjs 176 0 31
✅ fastify-stable-node 177 0 30
✅ fastify-stable-quickjs 176 0 31
✅ hono-stable-node 177 0 30
✅ hono-stable-quickjs 176 0 31
✅ nest-stable-node 177 0 30
✅ nest-stable-quickjs 176 0 31
✅ nextjs-turbopack-canary-node 206 0 1
✅ nextjs-turbopack-canary-quickjs 205 0 2
✅ nextjs-turbopack-quickjs-snapshot 205 0 2
✅ nextjs-turbopack-stable-node 206 0 1
✅ nextjs-turbopack-stable-quickjs 205 0 2
✅ nextjs-webpack-canary-node 206 0 1
✅ nextjs-webpack-canary-quickjs 205 0 2
✅ nextjs-webpack-stable-node 206 0 1
✅ nextjs-webpack-stable-quickjs 205 0 2
✅ nitro-stable-node 177 0 30
✅ nitro-stable-quickjs 176 0 31
✅ nuxt-stable-node 177 0 30
✅ nuxt-stable-quickjs 176 0 31
✅ sveltekit-stable-node 196 0 11
✅ sveltekit-stable-quickjs 195 0 12
✅ tanstack-start-node 177 0 30
✅ tanstack-start-quickjs 176 0 31
✅ vite-stable-node 177 0 30
✅ vite-stable-quickjs 176 0 31

✅ 🪟 Windows

App Passed Failed Skipped
✅ nextjs-turbopack-node 201 0 6
✅ nextjs-turbopack-quickjs 200 0 7

✅ 🌐 Cross-language Conformance

App Passed Failed Skipped
✅ python 68 0 114

✅ dynamic-runs

App Passed Failed Skipped
✅ example-vercel 0 0 0
✅ express-vercel 0 0 0
✅ hono-vercel 0 0 0
✅ nextjs-turbopack-vercel 0 0 0
✅ nitro-vercel 0 0 0
✅ vite-vercel 0 0 0

✅ local-postgres-spec9

App Passed Failed Skipped
✅ nextjs-turbopack-stable-node 0 0 182
✅ nextjs-turbopack-stable-quickjs 0 0 182

✅ local-spec9

App Passed Failed Skipped
✅ nextjs-turbopack-node 205 0 2
✅ nextjs-turbopack-quickjs 204 0 3

✅ vercel-http-transport

App Passed Failed Skipped
✅ example 171 0 36
✅ express 171 0 36
✅ hono 171 0 36
✅ nextjs-turbopack 199 0 8
✅ nitro 171 0 36
✅ vite 171 0 36

✅ vercel-spec9

App Passed Failed Skipped
✅ nextjs-turbopack-node 198 0 9
✅ nextjs-turbopack-quickjs 197 0 10

✅ vercel-ws-transport

App Passed Failed Skipped
✅ example 171 0 36
✅ express 171 0 36
✅ nextjs-turbopack 199 0 8
✅ vite 171 0 36

📋 View full workflow run

@vercel

vercel Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
workflow-docs Ready Ready Preview, v0 Nov 21, 2026 9:49pm UTC

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Sim World

Simulated world deterministic testing for races. Traces

🟢 world-sim scenario book — 0 fail of 45 total
scenario outcome events virt replay violations
✅ smoke-no-steps completed 3 0ms ok 0
✅ smoke-one-step completed 5 0ms ok 0
✅ hook-at-step-started completed 9 0ms ok 0
✅ hook-at-step-completed completed 9 0ms ok 0
✅ hook-at-hook-created completed 9 0ms ok 0
✅ claim-hook-at-registration completed 9 0ms ok 0
✅ claim-hook-at-step-started completed 9 0ms ok 0
✅ deadline-hook-wins completed 6 1.0h ok 0
✅ deadline-expires completed 6 1.0h ok 0
✅ step-vs-timer-early-settlement completed 6 1.0h ok 0
✅ long-sleep completed 9 30.0d ok 0
✅ hook-never-arrives stalled 3 0ms skipped 0
✅ step-retries-twice completed 9 2.0s ok 0
✅ parallel-steps completed 7 0ms ok 0
✅ hook-on-execution-state completed 9 0ms ok 0
✅ peek-hook-before-branch completed 9 0ms ok 0
✅ peek-hook-after-branch completed 9 0ms ok 0
✅ peek-hook-at-registration completed 9 0ms ok 0
✅ race-hook-before-probe completed 9 0ms ok 0
✅ race-hook-after-probe completed 9 0ms ok 0
✅ race-duplicate-delivery completed 10 0ms ok 0
✅ attr-hook-before-step completed 9 0ms ok 0
✅ attr-hook-after-step completed 9 0ms ok 0
✅ attr-from-step-body completed 10 0ms ok 0
✅ fork-hook-after-timeout completed 11 1.0m ok 0
✅ fork-hook-before-timeout completed 11 1.0m ok 0
✅ count-hook-after-timeout completed 13 1.0m ok 0
✅ count-hook-before-timeout completed 15 1.0m ok 0
✅ stale-read-step-count-fork completed 15 1.0m ok 0
✅ stale-read-equal-step-counts completed 11 1.0m ok 0
✅ step-vs-step-fork completed 9 0ms ok 0
✅ step-vs-step-fork-fenced completed 9 1.0s ok 0
✅ fence-catches-benign-direction completed 9 1.0s ok 0
✅ run-ahead-fenced-overlap completed 10 1.0s ok 0
✅ in-flight-before-decision completed 15 1.0m ok 0
✅ in-flight-before-decision-counted completed 13 1.0m ok 0
✅ in-flight-after-decision completed 15 2.0m ok 0
✅ stale-read-step-count-fork-fenced completed 15 1.0m ok 0
✅ fork-hook-wins completed 10 1.0m ok 0
✅ fork-timeout-wins completed 10 1.0m ok 0
✅ unclaimed-payload-under-fork completed 14 1.0m ok 0
✅ claimed-payload-under-fork completed 14 1.0m ok 0
✅ writers-independent-step-bodies completed 9 0ms ok 0
✅ writers-scripted-tempo completed 9 0ms ok 0
✅ cancel-mid-step cancelled 6 0ms skipped 0

Full trace: world-sim.txt

@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

📊 Workflow Benchmarks

❌ The benchmark run for 0aac38f failed. See the run logs for details.

No benchmark results were produced.

ℹ️ Metric definitions & methodology

Metrics —

All timestamps are deployment-side; runs are triggered in-deployment, so the CI runner and api.vercel.com sit outside every measured window. TTFS = start() → first step body (includes dispatch + any cold start); Fan-out TTFS/TTLS = first/last step completion of one Promise.all from the same anchor (the gap is the runtime’s fan-out spread); STSO/WO between step bodies. Streaming delivery performance is measured in durabench.

Cold starts stay in the numbers (real bursty-workload latency, inflates P75+); Best is the warm floor.

@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor
Framework Flow route Step reg. Framework output
hono 306.6 KiB (±0) 126.1 KiB (±0) 2.43 MiB (+4.5 KiB)
nextjs-turbopack 398.2 KiB (±0) 491 B (±0) 1.14 MiB (+338 B)
About these numbers

Sizes are gzip; parentheses show the change against main.
Flow route and Step reg. gate this job, on raw bytes rather than the gzip shown, at max(2%, 50.0 KiB). Framework output is informational.

0aac38f · run

@benpankow
benpankow force-pushed the ben/park-resume-stale branch from 60ece23 to 7c64607 Compare November 18, 2026 19:24
@benpankow
benpankow force-pushed the ben/park-resume-stale branch from 7c64607 to affdd51 Compare November 21, 2026 01:08
@benpankow benpankow changed the title [core] Resume a stale parked VM over another orchestrator's events [2/3] [core] Resume a stale parked VM over another orchestrator's events Nov 21, 2026
Signed-off-by: benpankow <ben.pankow@vercel.com>
The divergence handling landed with the rebase onto the drain fix: a parked VM
that diverges on its first pass is replayed from cold without spending the
divergence budget. Its test runs with that budget at zero. The stale-resume
test now also asserts that no cold replay ran.

Signed-off-by: benpankow <ben.pankow@vercel.com>
…s crossed

Signed-off-by: benpankow <ben.pankow@vercel.com>
…-of-order park, and a crossed divergence

Signed-off-by: benpankow <ben.pankow@vercel.com>

This branch was successfully deployed

1 active deployment
Preview – workflow-docs — 0aac38fd Deployed Nov 21, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant