Repository navigation
feat: run each benchmark in its own process - #57
Draft
not-matthias wants to merge 3 commits into
Draft
not-matthias wants to merge 3 commits into
not-matthias wants to merge 3 commits into
Conversation
Benchmarks of one binary share a process, so the allocator state and threads one benchmark leaves behind (raised mmap threshold, fragmented bins, retained caches) change the measurement of every benchmark after it. Adding or changing one benchmark can then shift unrelated results. Add `codspeed::Isolation`. It is on by default and turned off with `CODSPEED_PROCESS_ISOLATION=false` (or `0`); any other value than true/1/false/0 is an error. `Isolation::run` starts the benchmark executable again with the original arguments and `CODSPEED_ISOLATED_BENCHMARK` set to one benchmark URI, so each benchmark gets a fresh process. A failing child's exit code is propagated (128 + N for signal N). Re-exec instead of fork(): fork only copies the calling thread, so threads the binary started before the harness would be missing in the child and work sent to them would hang. Linux uses /proc/self/exe, macOS _NSGetExecutablePath; Windows is not supported.
In analysis builds, store the original argv in `Initialize` and route every benchmark through `Isolation::run`, one child per benchmark including its repetitions. A child only matches its own benchmark name, which overrides `--benchmark_filter`, and does not print the mode line and context again. Walltime builds are unchanged.
Build a fixture in simulation mode and check that: - each benchmark runs in its own process by default, including one that uses a thread started in main; - CODSPEED_PROCESS_ISOLATION=false runs them all in one process; - a failing benchmark fails the run with its exit code; - the child ignores the command line filter; - an invalid setting is rejected. Register it with ctest and as a Bazel sh_test, and run it in CI.
Hooray! CodSpeed harness just leveled up!The base and head of this comparison were measured with different runner settings, so their benchmark values are not directly comparable. What changed between base and head:
Re-run the base with the same settings to get a valid performance comparison. Comparing 
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TLDR: Benchmarks in one process share allocator and thread state, so results depend on which other benchmarks ran before them; now each benchmark runs in a fresh child process (Linux/macOS only; Windows errors out).
CODSPEED_PROCESS_ISOLATION: unset/1/true= one process per benchmark (default),0/false= shared process, any other value = hard error.CODSPEED_ISOLATED_BENCHMARK=<uri>, then waits for the child. Benchmarks run one after another.CODSPEED_ANALYSISbuilds (simulation/memory) are affected; walltime and off builds are untouched.--benchmark_filter.mainexist in the child.sh_test, new CI job): 3 PIDs with isolation on, 1 PID with it off, failing benchmark exits with code 3, filter override, invalid value rejected.Review notes
--benchmark_outand the JSON reporter are written per child; there is no aggregation in the parent.sh_testwould needrules_shell.Mirrors the codspeed-rust implementation (CodSpeedHQ/codspeed-rust, process isolation)
Closes COD-3959