[rjones30]

Branch mcsmear_random_merge_rtj (2 commits on master be7b14e). Independent of #422: either can be merged first, without conflicts, and the combined result is the same.

1. Multithreaded jobs lost the requested random-trigger offset (bug fix)

MyProcessor::BeginRun runs once in every processing thread, because the hits-merger configuration is thread_local. But the random-trigger input streams (files2merge) are shared by all threads, and each BeginRun call repositioned them to the start of the file plus the requested skip. The skip is zeroed after the first call, so with N threads the shared streams were rewound to the start of the file up to N>1 times while the threads started. Consequences for any multithreaded mcsmear job that merges random triggers:

• the random triggers come from the first few thousand events of the file, whatever file:N+skip asked for, so jobs that use different skips on the same file all merge (nearly) the same random events;
• the first few hundred events of the file are merged several times over within one job.

Where the start of a random-trigger file is not typical of the run (beam off, or the current ramping up after a trip), the merged background is biased, run by run. Single-threaded jobs are not affected.

Fix: position the shared streams once per run, by the first thread to begin that run. Single-threaded output is byte-identical to before.

2. New option MERGE:MIN_TAGGER_HITS (default 5 )

Random-trigger events with fewer tagger hits (hodoscope + microscope, all channels, any time) than MERGE:MIN_TAGGER_HITS were recorded with the beam off. With the option set, they are skipped at read time and the next event in the file is used, so beam-off events are never merged, whatever the random-trigger file contains. The numbers of random events read and skipped are printed at the end of the job.

1. Random-trigger files in use contain sizable beam-off fractions: e.g. recon-2019_11-ver01.2 6?56% per run, recon-2023-01-ver04 11?42% per run (events with 0 tagger hits and an empty CDC/FDC). In those files beam-off events have 0 tagger hits (a few 2023-01 events have 1?4 noise hits);
2. beam-on events typically have 40?100. We use -PMERGE:MIN_TAGGER_HITS=5.

Tests (run 71350, recon-2019_11-ver01.2 random triggers, which start with 591 beam-off events)

test	result
MIN_TAGGER_HITS=0, 1 thread, 300 events vs the previous binary	byte-identical output
MIN_TAGGER_HITS=5, 1 thread, 300 events, skip 0	925 read, 625 skipped: exactly the events with < 5 tagger hits in that range (offline scan)
same, vs MIN_TAGGER_HITS=0 on a copy of the file holding only the events with ? 5 tagger hits	byte-identical output
before the fix: 16 threads, skip 1000, 3000 events	16 BeginRun calls; 12323 read, 9323 skipped (the file start re-read), vs 3000 read, 0 skipped with 1 thread
after the fix: 16 threads, skip 1000, 3000 events	3000 read, 0 skipped (as 1 thread)
after the fix: 16 threads, skip 0, MIN_TAGGER_HITS=5 and =1	3625 read / 625 skipped and 3590 / 590: identical to 1 thread and to the offline scan
after the fix: 16 threads vs 1 thread, MIN_TAGGER_HITS=5, merged hits per event	CDC 41.9 vs 41.7, FDC anode 90.0 vs 89.8 (before the fix, 16 threads: 27.6 and 65.3)
after the fix: 1 thread vs before the fix	byte-identical output
this branch alone on master (default FDC options)	1 thread: 925 read / 625 skipped; 16 threads, skip 1000: 3000 / 0

Found in the GlueX rho0 photoproduction data/MC study of 2026-11.

[gluex]

Build status for this pull request: FAILURE

Build log: make_mcsmear_random_merge_rtj.log
Build report: report_mcsmear_random_merge_rtj.txt

[gluex]

Build status for this pull request: FAILURE

Build log: make_mcsmear_random_merge_rtj.log
Build report: report_mcsmear_random_merge_rtj.txt

[sdobbs]

Thanks for looking into these issues, Richard. Regarding the MERGE:MIN_TAGGER_HITS cut, I looked more closely at the random trigger files generated for one of the 2023-01 runs, and indeed there were events in these files that should have been excluded due to the electron beam fiducial cuts (but not consistently included!). This means we have to look into the process of how the random trigger files are created, but in the meantime I think this cut is a good idea - it should efficiently reject these events that should have been excluded.

[gluex]

Test status for this pull request: SUCCESS

Summary: summary.txt
Logs: results/log

Build log: make_mcsmear_random_merge_rtj.log
Build report: report_mcsmear_random_merge_rtj.txt