Skip to content

[1/3] [core] Resume a spec 9 run's parked VM on its next delivery - #5467

Draft
benpankow wants to merge 13 commits into
mainfrom
ben/park-retained-session
Draft

benpankow wants to merge 13 commits into
mainfrom
ben/park-retained-session

Conversation

@benpankow

@benpankow benpankow commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Description

A spec 9 run that loops on a hook gets every payload that arrives after its orchestrator stopped lingering on a new delivery. On the node engine that delivery reads the log tail, loads the whole log, and replays the workflow from cold in a fresh VM before it can act on the one new event. The load and the replay both grow with the run's history.

I hit this in eve, where each chat session is one such run and each user message is a hook resume. People rarely reply within the 5s hook linger, so almost every follow-up takes this path. On Vercel, for a 30-turn session whose next delivery landed in the same process, the delivery spent ~350ms between starting and the turn's step: tail read 40–104ms, full load beside runs.get ~100ms, cold replay ~100ms, then step_started.

This PR keeps the VM of a delivery that suspends cleanly (only when getRetentionDecision already allows retention), together with its log, cursor, writer's in-band count and payload cache, per World instance. The run's next plain wake in the same process takes it, reads only what landed since, and resumes the VM when the read's snapshot still has the parked seqInBand. Otherwise it does the full load and cold replay as today. That read also replaces the no-op check: nothing new and no due timer means it parks the session again and acknowledges. Every other orchestrator delivery (a start, a timer, wakeUp()) drops the entry, so no two deliveries resume one VM. The snapshot check is not a lease: an orchestrator that overlaps after the read is caught by the fence on the next in-band write, as today. QuickJS is unchanged.

Parking is opt-in, and bounded:

WORKFLOW_PARKED_VM_MAX_BYTES=8388608   # per-World budget; 0 (the default) turns parking off
WORKFLOW_PARKED_VM_MAX_AGE_MS=600000   # how long an entry is kept (default 10 minutes)

An entry is estimated at 256 KiB plus 3× the log's serialized bytes (from Peter's measurements). One over half the budget isn't parked, and parking evicts oldest-first, with at most 64 entries. A parked VM holds the run's decrypted payloads until its next delivery or expiry, so for an encrypted run (nearly every run on Vercel) plaintext outlives the delivery that decrypted it, in the same process. Setting the budget accepts that; the docs say so. Opt-in is the last commit, so it can be dropped if we'd rather have parking on by default. Spans record each park or skip with its reason, evictions, and what a taken entry's read found (workflow.parked_session.*).

A wake that names an event the VM already consumed (the queued copy of a wake this process acted on while lingering) almost always finds nothing new, so it reads the run only once the read finds something. Every other wake still reads the run beside the log, so a resume never waits for it.

This also fixes run-ahead's rebase, which parking depends on. A rebased session mapped the ids it consumed ahead of their writes to their committed ids by id, and followed those aliases through each other. The event that displaced a speculative one holds that one's old id, and a repair that shifts adjacent speculative events reports a → b and b → c, so every later resume of a rebased session declined and replayed from cold. That's common: a step body that calls setAttributes writes an attr_set into the slot its step's speculative outcome was placed at. Aliases now map the consumed event, once, and a rebased session parks and resumes like any other: a later read only brings events above the tail it consumed.

loadWorkflowRunEvents can keep an incremental load's first-page snapshot (incrementalSnapshot) and reports when a refused cursor made it read from the start (reloadedFromStart). The spec 9 README has a "Parked sessions" section.

Docs Preview

Page v5
Runtime tuning WORKFLOW_PARKED_VM_MAX_BYTES, WORKFLOW_PARKED_VM_MAX_AGE_MS

How did you test your changes?

New parked-session.test.ts (estimate, budget, eviction, age) and parked-session.runtime.test.ts, which runs a hook loop with an inline step per payload and no linger. It checks that later payloads resume with no full load or cold replay, including a session whose step bodies set attributes and one rebased over displaced speculative writes. It also covers the paths that drop or decline an entry:

  • a timer or wakeUp() delivery drops it and loads and replays itself;
  • a VM that declines its resume falls back to a cold replay and the run finishes;
  • a run cancelled while parked ends without resuming the VM into user code;
  • a resumed VM's end-of-run drain runs in the resuming delivery's AsyncLocalStorage store.

New workflow.test.ts cases cover the alias mapping, including the shifted chain. src in core passes locally.

On Vercel I ran a spike of the same change, patched into the @workflow/core@5.5.0 that eve bundles, against a deployed eve agent. Three follow-ups 40s apart on a 30-turn session all resumed the parked VM, and the time from delivery start to the turn's step went from ~350ms to ~146ms (an incremental read of 66ms beside runs.get). In a longer run (8 follow-ups 35s apart on a 20-turn session), all 8 resumed with no errors. From the resume's post.sent to the turn's step was 247ms median against 417ms without the change, queue hop included, and client-side first output was ~870ms against ~1,000ms.

PR Checklist - Required to merge

  • 📦 pnpm changeset was run to create a changelog for this PR
    • Use the correct semver bump type: patch for bug fixes, minor for new features, major for breaking changes.
    • Use pnpm changeset --empty if you are changing documentation or workbench apps
  • 🔒 DCO sign-off passes (run git commit --signoff on your commits)
  • 📝 Ping @vercel/workflow in a comment once the PR is ready, and the above checklist is complete

🤖

@vercel

vercel Bot commented Nov 16, 2026

Copy link
Copy Markdown
Contributor

You must have Developer access to commit code to Vercel Labs on Vercel. If you contact an administrator and receive Developer access, commit again to see your changes.

Learn more: https://vercel.com/docs/accounts/team-members-and-roles/access-roles#team-level-roles

@changeset-bot

changeset-bot Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: be53c54

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 18 packages
Name Type
@workflow/core Patch
@workflow/builders Patch
@workflow/cli Patch
@workflow/next Patch
@workflow/nitro Patch
@workflow/vitest Patch
@workflow/web-shared Patch
@workflow/web Patch
workflow Patch
@workflow/world-cloudflare Patch
@workflow/world-testing Patch
@workflow/astro Patch
@workflow/nest Patch
@workflow/rollup Patch
@workflow/sveltekit Patch
@workflow/vite Patch
@workflow/nuxt Patch
@workflow/world-vercel Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

📊 Workflow Benchmarks

❌ The benchmark run for be53c54 failed. See the run logs for details.

No benchmark results were produced.

ℹ️ Metric definitions & methodology

Metrics —

All timestamps are deployment-side; runs are triggered in-deployment, so the CI runner and api.vercel.com sit outside every measured window. TTFS = start() → first step body (includes dispatch + any cold start); Fan-out TTFS/TTLS = first/last step completion of one Promise.all from the same anchor (the gap is the runtime’s fan-out spread); STSO/WO between step bodies. Streaming delivery performance is measured in durabench.

Cold starts stay in the numbers (real bursty-workload latency, inflates P75+); Best is the warm floor.

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

🧪 E2E Test Results

✅ All tests passed

⚠️ Flaky E2E Tests (passed on retry)

These tests failed at least once and passed on a retry. A recurring entry here is a real race worth investigating.

  • a chain of takeovers: each victim ends with HookForceClaimedError, deliveries follow the current owner (nextjs-webpack · local-dev / local / node / canary)
  • payloads delivered before the takeover stay with the victim; the iterator then throws (nextjs-webpack · local-dev / local / node / canary)
  • promiseRaceStressTestWorkflow (nextjs-webpack · local-dev / local / node / stable)

🛠 Infra Events (absorbed by the harness)

Platform anomalies the e2e harness detected and worked around (e.g. a run the queue never picked up, replaced by a fresh run). Clustered timestamps indicate a backend blip; a steady drip indicates a platform issue worth escalating.

  • run-pickup-stall · runs app-generated source against a registered step (tanstack-start) · at 21:53:06Z · abandoned wrun_01M801J3FTH2MAVD94F64WVVKZ
  • cold-start-warmup · suite warmup (tanstack-start) · at 21:53:19Z · abandoned wrun_01M801J3GNDF1KB2AE3NN4Y9VM

E2E Test Summary

Summary
Passed Failed Skipped Total
✅ 💻 Local Development 5272 0 731 6003
✅ 📦 Local Production 5272 0 731 6003
✅ 🐘 Local Postgres 5417 0 586 6003
✅ 🪟 Windows 401 0 13 414
✅ 🌐 Cross-language Conformance 68 0 114 182
✅ dynamic-runs 0 0 0 0
✅ local-postgres-spec9 0 0 364 364
✅ local-spec9 409 0 5 414
✅ vercel-http-transport 1054 0 188 1242
✅ vercel-spec9 395 0 19 414
✅ vercel-ws-transport 712 0 116 828
Total 19000 0 2867 21867
Details by Category

✅ 💻 Local Development

App Passed Failed Skipped
✅ astro-stable-node 172 0 35
✅ astro-stable-quickjs 171 0 36
✅ express-stable-node 172 0 35
✅ express-stable-quickjs 171 0 36
✅ fastify-stable-node 172 0 35
✅ fastify-stable-quickjs 171 0 36
✅ hono-stable-node 172 0 35
✅ hono-stable-quickjs 171 0 36
✅ nest-stable-node 172 0 35
✅ nest-stable-quickjs 171 0 36
✅ nextjs-turbopack-canary-node 201 0 6
✅ nextjs-turbopack-canary-quickjs 200 0 7
✅ nextjs-turbopack-quickjs-snapshot 200 0 7
✅ nextjs-turbopack-stable-node 201 0 6
✅ nextjs-turbopack-stable-quickjs 200 0 7
✅ nextjs-webpack-canary-node 201 0 6
✅ nextjs-webpack-canary-quickjs 200 0 7
✅ nextjs-webpack-stable-node 201 0 6
✅ nextjs-webpack-stable-quickjs 200 0 7
✅ nitro-stable-node 172 0 35
✅ nitro-stable-quickjs 171 0 36
✅ nuxt-stable-node 172 0 35
✅ nuxt-stable-quickjs 171 0 36
✅ sveltekit-stable-node 191 0 16
✅ sveltekit-stable-quickjs 190 0 17
✅ tanstack-start-node 172 0 35
✅ tanstack-start-quickjs 171 0 36
✅ vite-stable-node 172 0 35
✅ vite-stable-quickjs 171 0 36

✅ 📦 Local Production

App Passed Failed Skipped
✅ astro-stable-node 172 0 35
✅ astro-stable-quickjs 171 0 36
✅ express-stable-node 172 0 35
✅ express-stable-quickjs 171 0 36
✅ fastify-stable-node 172 0 35
✅ fastify-stable-quickjs 171 0 36
✅ hono-stable-node 172 0 35
✅ hono-stable-quickjs 171 0 36
✅ nest-stable-node 172 0 35
✅ nest-stable-quickjs 171 0 36
✅ nextjs-turbopack-canary-node 201 0 6
✅ nextjs-turbopack-canary-quickjs 200 0 7
✅ nextjs-turbopack-quickjs-snapshot 200 0 7
✅ nextjs-turbopack-stable-node 201 0 6
✅ nextjs-turbopack-stable-quickjs 200 0 7
✅ nextjs-webpack-canary-node 201 0 6
✅ nextjs-webpack-canary-quickjs 200 0 7
✅ nextjs-webpack-stable-node 201 0 6
✅ nextjs-webpack-stable-quickjs 200 0 7
✅ nitro-stable-node 172 0 35
✅ nitro-stable-quickjs 171 0 36
✅ nuxt-stable-node 172 0 35
✅ nuxt-stable-quickjs 171 0 36
✅ sveltekit-stable-node 191 0 16
✅ sveltekit-stable-quickjs 190 0 17
✅ tanstack-start-node 172 0 35
✅ tanstack-start-quickjs 171 0 36
✅ vite-stable-node 172 0 35
✅ vite-stable-quickjs 171 0 36

✅ 🐘 Local Postgres

App Passed Failed Skipped
✅ astro-stable-node 177 0 30
✅ astro-stable-quickjs 176 0 31
✅ express-stable-node 177 0 30
✅ express-stable-quickjs 176 0 31
✅ fastify-stable-node 177 0 30
✅ fastify-stable-quickjs 176 0 31
✅ hono-stable-node 177 0 30
✅ hono-stable-quickjs 176 0 31
✅ nest-stable-node 177 0 30
✅ nest-stable-quickjs 176 0 31
✅ nextjs-turbopack-canary-node 206 0 1
✅ nextjs-turbopack-canary-quickjs 205 0 2
✅ nextjs-turbopack-quickjs-snapshot 205 0 2
✅ nextjs-turbopack-stable-node 206 0 1
✅ nextjs-turbopack-stable-quickjs 205 0 2
✅ nextjs-webpack-canary-node 206 0 1
✅ nextjs-webpack-canary-quickjs 205 0 2
✅ nextjs-webpack-stable-node 206 0 1
✅ nextjs-webpack-stable-quickjs 205 0 2
✅ nitro-stable-node 177 0 30
✅ nitro-stable-quickjs 176 0 31
✅ nuxt-stable-node 177 0 30
✅ nuxt-stable-quickjs 176 0 31
✅ sveltekit-stable-node 196 0 11
✅ sveltekit-stable-quickjs 195 0 12
✅ tanstack-start-node 177 0 30
✅ tanstack-start-quickjs 176 0 31
✅ vite-stable-node 177 0 30
✅ vite-stable-quickjs 176 0 31

✅ 🪟 Windows

App Passed Failed Skipped
✅ nextjs-turbopack-node 201 0 6
✅ nextjs-turbopack-quickjs 200 0 7

✅ 🌐 Cross-language Conformance

App Passed Failed Skipped
✅ python 68 0 114

✅ dynamic-runs

App Passed Failed Skipped
✅ example-vercel 0 0 0
✅ express-vercel 0 0 0
✅ hono-vercel 0 0 0
✅ nextjs-turbopack-vercel 0 0 0
✅ nitro-vercel 0 0 0
✅ vite-vercel 0 0 0

✅ local-postgres-spec9

App Passed Failed Skipped
✅ nextjs-turbopack-stable-node 0 0 182
✅ nextjs-turbopack-stable-quickjs 0 0 182

✅ local-spec9

App Passed Failed Skipped
✅ nextjs-turbopack-node 205 0 2
✅ nextjs-turbopack-quickjs 204 0 3

✅ vercel-http-transport

App Passed Failed Skipped
✅ example 171 0 36
✅ express 171 0 36
✅ hono 171 0 36
✅ nextjs-turbopack 199 0 8
✅ nitro 171 0 36
✅ vite 171 0 36

✅ vercel-spec9

App Passed Failed Skipped
✅ nextjs-turbopack-node 198 0 9
✅ nextjs-turbopack-quickjs 197 0 10

✅ vercel-ws-transport

App Passed Failed Skipped
✅ example 171 0 36
✅ express 171 0 36
✅ nextjs-turbopack 199 0 8
✅ vite 171 0 36

📋 View full workflow run

@vercel

vercel Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
workflow-docs Ready Ready Preview, v0 Nov 21, 2026 9:50pm UTC

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Sim World

Simulated world deterministic testing for races. Traces

🟢 world-sim scenario book — 0 fail of 45 total
scenario outcome events virt replay violations
✅ smoke-no-steps completed 3 0ms ok 0
✅ smoke-one-step completed 5 0ms ok 0
✅ hook-at-step-started completed 9 0ms ok 0
✅ hook-at-step-completed completed 9 0ms ok 0
✅ hook-at-hook-created completed 9 0ms ok 0
✅ claim-hook-at-registration completed 9 0ms ok 0
✅ claim-hook-at-step-started completed 9 0ms ok 0
✅ deadline-hook-wins completed 6 1.0h ok 0
✅ deadline-expires completed 6 1.0h ok 0
✅ step-vs-timer-early-settlement completed 6 1.0h ok 0
✅ long-sleep completed 9 30.0d ok 0
✅ hook-never-arrives stalled 3 0ms skipped 0
✅ step-retries-twice completed 9 2.0s ok 0
✅ parallel-steps completed 7 0ms ok 0
✅ hook-on-execution-state completed 9 0ms ok 0
✅ peek-hook-before-branch completed 9 0ms ok 0
✅ peek-hook-after-branch completed 9 0ms ok 0
✅ peek-hook-at-registration completed 9 0ms ok 0
✅ race-hook-before-probe completed 9 0ms ok 0
✅ race-hook-after-probe completed 9 0ms ok 0
✅ race-duplicate-delivery completed 10 0ms ok 0
✅ attr-hook-before-step completed 9 0ms ok 0
✅ attr-hook-after-step completed 9 0ms ok 0
✅ attr-from-step-body completed 10 0ms ok 0
✅ fork-hook-after-timeout completed 11 1.0m ok 0
✅ fork-hook-before-timeout completed 11 1.0m ok 0
✅ count-hook-after-timeout completed 13 1.0m ok 0
✅ count-hook-before-timeout completed 15 1.0m ok 0
✅ stale-read-step-count-fork completed 15 1.0m ok 0
✅ stale-read-equal-step-counts completed 11 1.0m ok 0
✅ step-vs-step-fork completed 9 0ms ok 0
✅ step-vs-step-fork-fenced completed 9 1.0s ok 0
✅ fence-catches-benign-direction completed 9 1.0s ok 0
✅ run-ahead-fenced-overlap completed 10 1.0s ok 0
✅ in-flight-before-decision completed 15 1.0m ok 0
✅ in-flight-before-decision-counted completed 13 1.0m ok 0
✅ in-flight-after-decision completed 15 2.0m ok 0
✅ stale-read-step-count-fork-fenced completed 15 1.0m ok 0
✅ fork-hook-wins completed 10 1.0m ok 0
✅ fork-timeout-wins completed 10 1.0m ok 0
✅ unclaimed-payload-under-fork completed 14 1.0m ok 0
✅ claimed-payload-under-fork completed 14 1.0m ok 0
✅ writers-independent-step-bodies completed 9 0ms ok 0
✅ writers-scripted-tempo completed 9 0ms ok 0
✅ cancel-mid-step cancelled 6 0ms skipped 0

Full trace: world-sim.txt

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor
Framework Flow route Step reg. Framework output
hono 306.6 KiB (±0) 126.1 KiB (±0) 2.43 MiB (+4.1 KiB)
nextjs-turbopack 398.2 KiB (±0) 491 B (±0) 1.14 MiB (+324 B)
About these numbers

Sizes are gzip; parentheses show the change against main.
Flow route and Step reg. gate this job, on raw bytes rather than the gzip shown, at max(2%, 50.0 KiB). Framework output is informational.

be53c54 · run

@VaguelySerious VaguelySerious left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(AI) Preliminary review: AI review: blocking issues found

* not for long. A delivery that finds none, or a stale one, only pays today's
* full load and replay.
*/
const MAX_ENTRIES = 64;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI Review: Blocking

Parked entries are bounded by count (64) and age (10 min) only, and nothing bounds their size. Each one keeps three copies of the run's whole history in memory between deliveries, in a process that keeps serving other invocations:

  • the VM and its hydrated state;
  • the decoded log, with its raw payload bytes;
  • the payload cache, with the decrypted plaintext of every event since the run started.

ReplayPayloadCache's doc comment still says its memory "never crosses workflow runs or queue deliveries", which this PR makes untrue.

I measured it with a throwaway test, not pushed. The workflow is a chat-style hook loop that keeps its history in workflow state, with AppendOnlyWorld serving cloned events as a World over the network does. Under forced GC, I compared heap plus array buffers with the entry parked and after takeParkedSession dropped it:

turns × reply size payload bytes in the parked log retained by the parked entry
1 × 16 B 0 0.24 MB
30 × 4 KB 0.09 MB 0.64 MB
30 × 32 KB 0.71 MB 2.28 MB
30 × 128 KB 2.83 MB 7.91 MB
100 × 32 KB 2.37 MB 7.11 MB
64 runs × 30 × 32 KB 142.8 MB

One entry holds about 0.25 MB plus about 2.8× its log's payload bytes, and that grows without limit with the run's history. One long agent session can park tens of MB by itself. At the 64-entry cap, modest sessions already hold over 140 MB on top of what the live invocations need, and workflows already risk going OOM. Parked data should stay in the single-digit MB range per process.

Suggestion:

  • At the park site (outcome.parkSession === true in delivery.ts), estimate the entry's size. Summing the byteLength of the input/result/payload/error byte arrays in log.events is cheap and closely tracks what I measured. Don't park when one entry is over a small cap, for example about 1–2 MB of payload bytes (about 3–6 MB retained).
  • Keep a per-World byte budget, for example 8 MB estimated, next to or instead of MAX_ENTRIES, and evict oldest-first when a park would go over it.
  • Make the budget configurable, with 0 turning parking off without also turning off in-delivery VM retention (WORKFLOW_RETAINED_VM).
  • Add tests: a run over the cap doesn't park, and eviction keeps the total under the budget.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added in dbcc7e2. WORKFLOW_PARKED_VM_MAX_BYTES (8 MiB by default) is the per-World budget for the estimated memory of parked entries. An entry estimated over half of it isn't parked, and parking evicts oldest-first to stay within it and the 64-entry cap. 0 turns parking off and leaves WORKFLOW_RETAINED_VM alone.

The estimate is 256 KiB plus 3× the byteLength of the byte arrays in the log's eventData, from your measurements (0.25 MB + 2.8×). On #5573, a hook_received handed to a parked entry adds its payload to the entry's count. I also fixed the ReplayPayloadCache comment.

Tests: unit tests for the estimate, the per-entry cap, eviction under the budget, 0, and an invalid value. A runtime test checks that a session over half the budget cold-replays, and another that 0 cold-replays every follow-up.

🤖

);
if (outcome.type === 'return') {
if (
outcome.parkSession === true &&

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI Review: Note

Only a successful resume is visible (workflow.parked_session_resumed). Nothing records any of these:

  • a park;
  • a park skipped (not retained, superseded, or over the budget once there is one);
  • an eviction (by the cap or by age);
  • a stale or reloaded read, which logs at debug only.

Hit rate and parked memory can't be seen in production without them. Consider span attributes on the parking delivery (parked or not, the reason, the estimated bytes) and on the taking delivery (resume/stale/reloaded, and the entry's age).

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added in dbcc7e2, all under workflow.parked_session.*:

  • Parking delivery: parked, bytes, and either evicted (count) or skip_reason: not_retained, superseded, disabled, too_large or newer_parked.
  • Taking delivery: age_ms and read: resume, stale, reloaded, or dropped for a start, timer or wakeUp() delivery. [2/3] [core] Resume a stale parked VM over another orchestrator's events #5470 replaces stale with crossed (resumed over another orchestrator's writes) and unread.

The resumed attribute moved to workflow.parked_session.resumed to match. Age-based expiry isn't counted separately. It shows as evicted only when a park needed the room, and as a missing entry otherwise.

🤖


The resuming delivery hands the VM its own end-of-run drain (`WorkflowSession.resume`). The VM was created with the parking delivery's, whose writer's in-band count stops being the run's once anything else is written, so a run that finished on a resumed VM with work still queued (an unawaited step) would have its drain refused as superseded and the items lost.

Everything after the setup is unchanged: the deployment guard, the run's status, the re-send pass, timers and the fence all apply as on any delivery. The check is not a lease: an overlapping orchestrator that writes after the read is caught by the fence on this delivery's next in-band write, as it is today. Entries are kept per World instance, at most 64 of them, for up to 10 minutes, and a timer drops expired ones. Two deliveries of a run in one process can park in either order (a no-op wake puts back the entry it took after a later delivery parked a newer one), so an entry is never replaced by one whose log ends at a lower slot. QuickJS keeps its own VM lifetime and does not park.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI Review: Note

How long VM state stays in memory is a user-facing change, but only this in-source README covers it. The WORKFLOW_RETAINED_VM entry in docs/content/docs/v5/configuration/runtime-tuning.mdx still says the VM is kept "within one invocation". That variable is now the only way to turn parking off, because it gates getRetentionDecision. Please update the entry to cover parking across deliveries, its bounds, and how to turn it off, plus any dedicated knob that comes out of the size-budget comment.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated runtime-tuning.mdx in dbcc7e2. The WORKFLOW_RETAINED_VM entry now covers parking between deliveries and says 0 turns that off too. A new WORKFLOW_PARKED_VM_MAX_BYTES entry covers the budget, the estimate, the half-budget per-entry cap, eviction, and the 64-entry and 10-minute limits. The spec 9 README and the changeset mention the budget as well.

🤖

"@workflow/core": patch
---

A spec 9 run on the node engine resumes the VM its previous delivery in the same process left suspended, on an incremental read of what landed since, instead of loading its whole log and replaying it from cold. A hook-driven run, such as a chat session, no longer pays a load and replay that grow with its history for each message that arrives after its orchestrator stopped lingering.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI Review: Note

The Vercel prod E2E jobs and the Vercel benchmark failed at "Resolve Vercel deployment" because no preview deployment of this commit was created. So CI never ran world-vercel, the path this change is for. The only coverage there is the spike in the description. Get a green prod E2E and benchmark run before merging.

vi.unstubAllEnvs();
});

describe('a session parked across deliveries', () => {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI Review: Note

Tests that would cover the paths that drop or bypass an entry:

  • A timer wake (waitContinuation) or a wakeUp() delivery drops the entry, and the next plain wake does a full load and cold replay.
  • A run cancelled while parked: run_cancelled lands in the incremental read, and the resumed delivery ends the run without resuming the VM into user code.
  • A parked VM resumed after the run has a later hook_received for a disposed hook, or something else the VM declines: it falls back to a cold replay, and the run finishes correctly.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added in dbcc7e2:

  • a wakeUp() delivery takes and drops the parked VM, then does a full load and a cold replay;
  • a run cancelled while parked ends without running the pending payload's step;
  • a hook_received for a disposed hook reaches the resumed VM, which ignores it as a cold replay would, and the run finishes correctly with no fallback.

The VM didn't decline that event, so there is no cold replay to assert.

🤖

wakeUpWaits.size === 0
) {
// The run is read beside the log, as on the full-load path.
const parkedRun = world.runs.get(runId, { resolveData: 'none' });

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI Review: Nit

On a parked no-op wake, the runs.get started here is discarded. That is one more world-vercel request per no-op wake than the existing no-op path, which does a single tail list. It's reasonable for the resume path's latency, so this is optional. If no-op wakes turn out to be much more common than resumes, start it after the no-op decision instead.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Keeping it for now. In the chat-style runs this is for, almost every taken entry resumes, and starting runs.get after the read would put ~30ms back on the resume path. The new workflow.parked_session.read attribute will show the ratio. If parked no-op wakes turn out to dominate, I'll move it after the no-op decision.

🤖

@benpankow
benpankow force-pushed the ben/park-retained-session branch from 43fe7ba to b7129b6 Compare November 18, 2026 17:30
@benpankow
benpankow changed the base branch from main to ben/wake-names-its-event November 18, 2026 17:30
@benpankow
benpankow added this pull request to stack #5584 November 18, 2026 18:57
Base automatically changed from ben/wake-names-its-event to main November 19, 2026 14:41
Signed-off-by: benpankow <ben.pankow@vercel.com>
A parked VM kept the end-of-run drain of the delivery that created it, so a run
that finished on a resumed VM with an unawaited step had its drain refused as
superseded and lost the item. resume() now takes the caller's drain.

Also: a park never replaces an entry further along the log, expired entries
are dropped on a timer and whenever the map is touched, and a read whose cursor
the World refused is used as the full load instead of reading the log again.

Signed-off-by: benpankow <ben.pankow@vercel.com>
A resume the VM declines falls back to a cold replay of the assembled log
without a full load, so the full-load count alone did not prove a resume.

Signed-off-by: benpankow <ben.pankow@vercel.com>
Signed-off-by: benpankow <ben.pankow@vercel.com>
…d VM

Signed-off-by: benpankow <ben.pankow@vercel.com>
…ark and take

An entry holds the run's history about three times over and grew without limit. WORKFLOW_PARKED_VM_MAX_BYTES (8 MiB by default, 0 turns parking off) caps the estimated total per World, an entry over half of it is not parked, and parking evicts the oldest entries. Spans now record parks, skips with their reason, evictions, and what a taken entry's read found.

Signed-off-by: benpankow <ben.pankow@vercel.com>
A repair that shifts adjacent speculative events reports a -> b and b -> c, and the event that displaced a speculative one holds its old id. Id-keyed aliases followed through each other, so every later resume of a rebased session declined and replayed from cold.

Signed-off-by: benpankow <ben.pankow@vercel.com>
…tributes

Signed-off-by: benpankow <ben.pankow@vercel.com>
…nd async context

Signed-off-by: benpankow <ben.pankow@vercel.com>
…VM_MAX_AGE_MS

Signed-off-by: benpankow <ben.pankow@vercel.com>
Signed-off-by: benpankow <ben.pankow@vercel.com>
… that is safe

Signed-off-by: benpankow <ben.pankow@vercel.com>
@benpankow
benpankow force-pushed the ben/park-retained-session branch from dbcc7e2 to 035ddac Compare November 21, 2026 01:08
@benpankow benpankow changed the title [core] Resume a spec 9 run's parked VM on its next delivery [1/3] [core] Resume a spec 9 run's parked VM on its next delivery Nov 21, 2026
Signed-off-by: benpankow <ben.pankow@vercel.com>
@benpankow
benpankow force-pushed the ben/park-retained-session branch from 035ddac to be53c54 Compare November 21, 2026 21:47
@vercel-internal-security-reviewer

Copy link
Copy Markdown

Security review details

This branch was successfully deployed

1 active deployment
Preview – workflow-docs — be53c54a Deployed Nov 21, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants