Skip to content

perf(pi): reuse the model runtime across wakes of an actor - #408

Open
dcaminos wants to merge 3 commits into
rivet-dev:mainfrom
dcaminos:perf/pi-model-runtime-cache
Open

dcaminos wants to merge 3 commits into
rivet-dev:mainfrom
dcaminos:perf/pi-model-runtime-cache

Conversation

@dcaminos

Copy link
Copy Markdown

Problem

createActorModelRuntime (packages/pi/src/models.ts) builds a new ModelRuntime on every call: it creates the runtime, registers the providers and refreshes the available models. createModels in runtime.ts calls it each time an actor's harness is set up, so an actor that sleeps and wakes often pays this on every wake, although its options have not changed.

Measured with only this package (no application code), 50 calls after 5 warm-up calls, one custom provider, three runs each:

per call main (ce882b8) this branch
CPU 6.2 – 7.0 ms 0.89 – 0.94 ms
wall time 4.7 – 5.3 ms 0.65 – 0.68 ms
V8 heap allocated ~6.8 MB ~1.0 MB
GCs in the window 7 – 9 1

In our hosted setup (many short actor wakes) this was about 12 % of the runner's CPU.

Change

  • createActorModelRuntime(options, credentials, actorId?) keeps one runtime per actor id in a module-level map (at most 256 actors, oldest evicted first).
  • A cached runtime is reused only when a fingerprint of the options that shape it is unchanged: providers, apiKeys, scopedModels, model and PI_OFFLINE (key order does not matter).
  • Credentials are never cached: the runtime reads them through a small forwarding store that is pointed at the current wake's credential store on every reuse.
  • Without an actorId nothing is cached and behaviour is exactly as before. createModels now passes c.actorId.

Tests

New packages/pi/tests/model-runtime-cache.test.ts:

  • the runtime is built once across five wakes of one actor (fails on main: built 5 times);
  • changed apiKeys or model build a new runtime, reordered keys do not (fails on main);
  • a reused runtime reads the credentials of the current wake, not the previous one;
  • two actors never share a runtime or its credentials;
  • the oldest actor is evicted past the limit and built again on its next wake;
  • without an actor id nothing is cached.

pnpm build, pnpm check-types, pnpm check-boundaries pass; pnpm --filter @rivet-dev/pi test: 20 files, 133 tests pass; pnpm lint reports 5 existing warnings in files this PR does not touch.

Notes for review

  • The cache is per process and keyed by actor id; is there a case where two registries in one process could see the same actor id with different options that the fingerprint does not cover?
  • The limit of 256 actors is arbitrary; happy to change it or make it an option.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant