Repository navigation
feat: run each benchmark in its own process - #101
not-matthias wants to merge 6 commits into
Conversation
Benchmarks of one process share allocator, JIT and thread state, so what one benchmark leaves behind (raised mmap threshold, fragmented heap, optimized code, retained caches) changes the measurement of every benchmark after it. Adding or changing one benchmark can then shift unrelated results. Add `Isolation` for the integrations. It is on by default and turned off with `CODSPEED_PROCESS_ISOLATION=false`. `Isolation.run` re-executes the current command with `CODSPEED_ISOLATED_BENCHMARK` set to one benchmark URI, so each benchmark gets a fresh process. The child only runs that benchmark, which overrides user filters. - `CODSPEED_PROCESS_ISOLATION`: unset, `1` or `true` is on, `0` or `false` is off, anything else is an error. - `CODSPEED_ISOLATED_BENCHMARK`: set by the parent on each child. - `CODSPEED_ISOLATION_COMMAND`: JSON command to re-run, for runners that measure in worker processes whose argv is not the benchmark command. The child is started with `process.execPath` and `process.execArgv`, so the V8 flags set by the runner survive. A failing child ends the parent with the same exit code (128 + N when signal N killed it). Children run sequentially with `spawnSync`, which keeps the output in order. Re-exec is used instead of fork because fork only copies the calling thread.
Route every instrumented benchmark through `Isolation.run`. A child process only runs its own benchmark and skips the banner, the summary line and the completion listeners that the parent already handles. A run call that owns no benchmark of the child does not touch the instruments.
The analysis runner measures every task through `Isolation.run` (async and sync). Walltime runs and runs without active instrument hooks are not isolated. A child only measures its own task and stays quiet about the banner and the summary line.
Both the Vitest 5 provider and the legacy analysis runner measure through `Isolation.run`. An isolated child still reports every benchmark to Vitest, but only measures its own. Vitest measures in forked workers, and the argv of a worker is not the command that starts the benchmarks. The plugin records the command of the main Vitest process in `CODSPEED_ISOLATION_COMMAND` from its `config()` hook, before the workers are forked, and each worker re-runs that command once per benchmark. This means one full Vitest run per benchmark.
Run fixtures in simulation mode with the instrument hooks stubbed (they only exist under the CodSpeed runner) and check, for core, benchmark.js, tinybench and Vitest 5, that: - every benchmark runs in a fresh process by default; - `CODSPEED_PROCESS_ISOLATION=false` runs them all in one process; - a failing benchmark fails the run; - core forwards the child exit code (128 + N for signals) and the execArgv. Tests that drive a plugin in-process against a mocked core now set `CODSPEED_PROCESS_ISOLATION=false`, because isolation would start the test runner again for every benchmark.
Document the environment contract, the parent and child roles, and the Vitest and test caveats.
a33e439 to
4bd0611
Compare
Merging this PR will degrade performance by 79.42%3 warnings, 1 note
Warning Please fix the performance issues or acknowledge them on CodSpeed. Performance Changes
Tip Investigate this regression by commenting Comparing 
Footnotes
|
1 participant
TLDR: Benchmarks in one process share allocator and JIT state, so one benchmark shifts the next; each instrumented benchmark now runs in a fresh process (one full Vitest run per benchmark for the Vitest plugin).
CODSPEED_PROCESS_ISOLATION(unset/1/trueon by default,0/falseoff, other values error),CODSPEED_ISOLATED_BENCHMARK=<uri>set on each childprocess.execPath+process.execArgv+process.argv.slice(1), so the V8 flags from the runner survive128 + Nfor signalN)Review notes
CODSPEED_ISOLATION_COMMANDfrom the main process; each worker re-runs the whole Vitest command per benchmark, and the child's Vitest reporter output is not silencedCODSPEED_PROCESS_ISOLATION=falsesetupCore, so it writes a perf map and environment file for a PID that measures nothingMirrors the codspeed-rust implementation (CodSpeedHQ/codspeed-rust, process isolation)
Closes COD-3960