Skip to content

Time train phases with CUDA events instead of %globaltimer - #701

Open
sylvesterroos wants to merge 1 commit into
PufferAI:5.0from
sylvesterroos:train-timing-events
Open

sylvesterroos wants to merge 1 commit into
PufferAI:5.0from
sylvesterroos:train-timing-events

Conversation

@sylvesterroos

Copy link
Copy Markdown
  • The inline PTX %globaltimer read in src/pufferl.cu is the one line hipify cannot translate, so it blocks building the trainer for AMD GPUs.
  • An event recorded with cudaEventRecordExternal during stream capture becomes an event record node. It timestamps on every graph launch, so cudaEventElapsedTime works inside the captured train graph. The removed comment holds for plain cudaEventRecord.
  • Outside capture (cudagraphs = -1), the same events record with default flags.
// External events timestamp on every launch of a captured graph
cudaStreamCaptureStatus capture;
cudaStreamIsCapturing(stream, &capture);
int ev_flags = capture == cudaStreamCaptureStatusActive
    ? cudaEventRecordExternal : cudaEventRecordDefault;
cudaEvent_t* ev = pufferl->profile.train_ev + slot * NUM_TE;
cudaEventRecordWithFlags(ev[TE_S], stream, ev_flags);
  • Profile.stamps (device buffer + D2H copy) becomes cudaEvent_t train_ev[2 * NUM_TE], same per-slot layout. train_misc / train_model read cudaEventElapsedTime in ms directly.
  • Complements Add a hipify build script for PufferLib 5.0 #688: with this change its three %globaltimer sed rules match nothing and can be dropped.
  • Not yet run on NVIDIA hardware. On an AMD iGPU (hipify + ROCm) the train timings match the old %globaltimer stamps with cudagraphs on and off, and losses are identical.
  • AI-assisted: Opus 5.5 researched the alternatives to the PTX timer and wrote this change. I ran it on my AMD iGPU to verify it works, but I don't have CUDA knowledge, so please review the CUDA side with that in mind.

External event records time phases inside the captured train graph,
replacing the inline-PTX stamp kernel that blocks HIP builds.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant