Skip to content

Add fjmethodid=lazy: profile HotSpot without jmethodIDs - #868

Draft
rkennke wants to merge 24 commits into
mainfrom
investigate/lazy-jmethodid-vm-resolution
Draft

rkennke wants to merge 24 commits into
mainfrom
investigate/lazy-jmethodid-vm-resolution

Conversation

@rkennke

@rkennke rkennke commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

Effect (opt-in via fjmethodid=lazy; HotSpot with cstack=vm; J9, Zing and the default mode are unchanged):

  • No jmethodIDs for profiling. None are preloaded at ClassPrepare, and none are allocated when sampled classes are dumped. Raw Method* frames are named, with line numbers, straight from VM metadata. The only classes that still get jmethodIDs are those redefined during the session.
  • Memory: preloading costs ~16 MiB of native memory per million loaded methods. In lazy mode, the jmethodID-related NMT categories stay at the no-profiler level, with or without sampling. On Renaissance the profiler's NMT cost drops from 2.2–5.1 MiB to 0.1–0.8 MiB.
  • Class loading: nothing happens at ClassPrepare: application threads no longer call GetClassMethods or take the JVM's jmethodID-creation lock. The profiler keeps no per-class state between dumps.
  • Class churn: after ~1.1M short-lived 1-method classes, preloading keeps ~110 MiB on JDK 17–25 (it never frees a jmethodID slot); lazy mode keeps under 2 MiB.
  • Dumps: as fast as with preloading, or faster: about 15% faster on finagle-http.
  • Safe against class unloading, metadata reuse and redefinition: a raw frame whose class has since been unloaded, replaced or redefined resolves as unknown, never as another method. Metadata is only read under crash protection, while its class is pinned. The one explicit limit is a tag collision, which needs a JVM that has created more than 2^25 classes (see "Guarantees and limits").

What does this PR do?:

It adds fjmethodid=lazy, which profiles without jmethodIDs. Today fjmethodid=false already stores raw Method* frames, but only for classes of the boot, platform and application loaders. Custom-loader, hidden and lambda classes, the ones with high churn, are still preloaded. Lazy mode preloads nothing and resolves every raw frame at dump time:

  • each dump pins the loaded classes and resolves only frames of those classes, so a frame's class is known to be loaded, and stays loaded while its metadata is read;
  • every raw frame carries a tag from the holder's JFR trace ID, so frames of different loads of a class at the same address are never confused;
  • method names, signature and line numbers are read from VM metadata under crash protection, without JNI or JVMTI.

This combines the class-liveness design of #864 with the VM-metadata method resolution of #867 (Zhengyu Gu, co-author), and replaces both.

Motivation:
Preloading jmethodIDs costs memory, about 16 bytes per method for every method of every loaded class, and class-loading time. Most of it goes to classes that are never sampled.

How it works:

Flow

  1. Sample time (signal handler). If the sampled method has no jmethodID, HotspotSupport::fillJavaFrame() stores the raw Method* with a tag (below). The method is on the stack, so it is valid at that moment.
  2. ClassPrepare and start. Nothing to do: lazy mode allocates no jmethodIDs and registers nothing.
  3. Dump time (dump thread). Recording::finishChunk() calls GetLoadedClasses at the start of every dump and holds the returned local references until the dump ends, so no loaded class can be unloaded meanwhile. It hands the list to HotspotSupport::beginDump(). The dump's first lazy resolution collects these classes' Klass*s into a temporary hash set, freed by endDump(); a dump without lazy frames builds none. Then, for each raw frame, in HotspotSupport::resolve():
    • Under crash protection, read only the holder Klass* and the idnum from the possibly stale Method* (readMethodHolder()). Both serve as lookup keys and are never trusted. If they can't be read, the frame is unknown (method_resolve_lazy_unreadable).
    • The holder must be one of the pinned classes. Otherwise its class was unloaded after the sample, or the holder read from a freed Method* is garbage, and the frame is unknown (method_resolve_lazy_unloaded). A pinned class keeps its class loader, and so the class's metadata, alive.
    • A class marked as redefined is refused (below).
    • The frame's tag must match the pinned class's trace ID. Otherwise the memory now belongs to a different load of a class (method_resolve_lazy_stale).
    • VMKlass::containsMethod(M, idnum) compares pointers against the class's current methods array, through SafeAccess. A miss means the Method* was freed or replaced (method_resolve_lazy_not_current).
    • Only then is M read: readMethod() copies the method name, signature and holder name, detects Thread.run overrides via the Klass::_super chain, and decodes the compressed line number table. All of this runs under crash protection, with the class pinned (method_resolve_lazy_resolved).
    • Afterwards, the class's redefinition mark is checked again (below). If a redefinition started in the meantime, the result is dropped.
  4. Writing the chunk. Lookup::fillRawMethod() resolves each distinct (Method*, tag) at most once per dump. Refused frames all go to the shared unknown row and never touch a method row. Resolved frames get a row keyed by the untagged Method* plus tag bits, kept apart from jmethodID keys. A row is refilled at its first use in every chunk, so a row from an earlier chunk is never trusted without resolving again.

Lazy mode requires the stack walker to be vm, as resolved by the profiler (cstack=default becomes dwarf where the VM walker is unavailable); otherwise everything is preloaded as in the default mode. It also needs VMStructs::hasKlassMethods() and a calibrated trace-ID offset; without them it uses partial mode. HotSpot only.

Reading names and line numbers from VM metadata

  • Path: Method → ConstMethod → ConstantPool → {name, signature} Symbols, and ConstantPool → holder Klass → name. Every pointer step goes through SafeAccess or runs under a sigsetjmp landing pad. Names longer than the inline buffers (512 B for method names, 1 KiB for signatures and class names) fall back to malloc, up to 64 KiB.
  • Line numbers: HotSpot stores the compressed line number table right after the bytecodes (ConstMethod::code_end()), when ConstMethod::_flags has the "has line number table" bit. That bit is ConstMethod::_has_linenumber_table on JDK 8–20 and ConstMethodFlags::_misc_has_linenumber_table on 21+. VMLineNumberStream mirrors CompressedLineNumberReadStream, including the UNSIGNED5 change in JDK 20. A table whose bcis go past code_size (the bound ConstMethod::verify_on() checks) is rejected as stale metadata.
  • Hidden classes: HotSpot names a hidden class <name>+0x<hex>, and JVMTI's GetClassSignature() turns that last + into .. The raw path does the same, so lambda frames are named alike in every mode.
  • vmStructs fields used, checked against the OpenJDK 11–26 GA sources and exercised on 8u462: ConstMethod::_flags / _code_size, Klass::_super, and the version-dependent flag constant above.

Raw frame tags (Klass* / Method* reuse)

Freed class metadata is reused: a new class can get the same Klass* and the same Method* addresses as an unloaded one. So every lazy-mode raw frame carries a 26-bit tag, stored in 10 spare bci bits plus the top 16 bits of the Method*. The tag is a marker bit plus the low 25 bits of the holder's JFR class trace ID (Klass::_trace_id). JFR assigns that ID at class creation from one global counter, whether or not JFR is recording, so it is unique per load of a class. Frames from different loads of a class therefore never merge into one trace, and stale frames are refused at dump time.

Partial-mode (fjmethodid=false) frames stay untagged, as before. resolve() goes by the marker bit of each frame, not by the current mode. A lazy-mode Method* that uses the tag's address bits, or whose holder's trace ID can't be read, is stored as not walkable (walkvm_raw_frame_untaggable), never untagged.

Finding Klass::_trace_id

The field isn't in gHotSpotVMStructs, and its position moves between releases; on mainline it even depends on a build option. So KlassTraceId::calibrate() finds it by probing:

  • It makes the JVM create java.lang.Void array klasses of six consecutive dimensions, back to back.
  • It accepts the one 8-byte offset within sizeof(Klass) at which all six hold distinct trace IDs, no further apart than consecutive creation allows. Pointers, small integers and mark words fail this test, and more than one match counts as a failure.
  • An attempt can fail if some of those dimensions already existed, or if other threads created many classes in the meantime. So it makes up to four attempts, each on six dimensions above the last.
  • Only a found offset is kept, and it is published as the klass_trace_id_offset gauge.
What calibration relies on, checked against 107 OpenJDK GA tags

_trace_id is declared in class Klass (so array klasses carry it); arrayKlass.cpp assigns it (INIT_ID(this)); the ID is counter << 16. All three hold on every JFR-enabled tag:

Line GA tags Result Field declared after
8u 8u201 – 8u252 (7) no _trace_id (pre-JFR): calibration fails, partial mode —
8u 8u262 – 8u504 (29) holds _biased_lock_revocation_count
11u 11.0.2 – 11.0.32.1 (36) holds _access_flags
17u 17 GA – 17.0.17 (20) holds _access_flags
21u 21 GA – 21.0.9 (10) holds _access_flags
22–24 3 holds _access_flags
25+ 25 GA – 25.0.4.1, 26, 28+18 (8) holds _archived_mirror_index

Calibrated offsets at runtime: 8u462 → 184, 11.0.28 → 168, 17.0.16 → 168, 21.0.8 → 184, 25 → 192, 26 → 192, 27 → 192.

Pinning per dump instead of a class registry

  • The pinning comes from a call every dump already makes in every mode: GetLoadedClasses returns a JNI local reference for every loaded class, including hidden classes and arrays, after marking each class loader alive. Lazy mode adds one Klass* read per loaded class (GetLongField on java.lang.Class.klass), only in dumps that have lazy frames.
  • The set is an open-addressing table of Klass* with at least twice as many slots as classes, allocated for the dump and freed after it. Nothing is kept between dumps.
  • A class loaded after the dump's GetLoadedClasses call isn't pinned, so a frame of it in that dump resolves as unknown.
  • lazy_dump_classes and lazy_dump_pin_ns report how many classes a dump pinned and how long collecting their Klass*s took.

Redefinition

A redefined class stays alive, but its old methods array and old Method*s are freed once they're off every stack, and a later redefinition can reuse those addresses within the same class. So:

  • The RedefineClasses/RetransformClasses hooks register the class in KlassRegistry and mark it before calling the original function. The registry holds only redefined classes: Klass* → weak global ref to the mirror plus the mark, 24 B per entry. The mark lasts while the call runs, and for good if it succeeds. After a successful call in lazy mode, the class's jmethodIDs are preloaded, so its later frames aren't raw. A call that JVMTI rejects changes nothing.
  • Raw frames of a marked class are refused (method_resolve_lazy_redefined).
  • resolve() checks the mark again after reading the metadata, under the registry mutex. If the mark is still absent then, every read happened before the methods could be swapped.
  • A mark is only reported while the marked class is alive (its weak ref isn't cleared), so a new class at the same Klass* address starts unmarked.
  • Redefinitions are tracked in every mode once lazy mode has started, because raw frames of a lazy session can outlive a restart in another mode.
  • The hooks don't inspect their arguments before JVMTI has validated them: a null array, an array class or a non-class goes to JVMTI unchanged and fails there.

To check that a method is one of its class's current methods, the profiler looks it up in the methods array. Normally a method sits at the index given by its idnum, so that's a single read. Methods added after parsing can shift entries: bridge methods the JVM generates for default-method conflicts at link time, or a redefinition. Then the array is scanned, but only in classes of at most 256 methods. In a larger class such a frame is reported as unknown.

Partial mode

fjmethodid=false now reads raw frames' names from VM metadata too, instead of a JNI FindClass/GetMethodID lookup. This also covers <clinit>, which JNI can't look up, without a special case. As before, partial mode keeps no redefinition marks. A Method* freed by a redefinition and then reused resolves to whichever method lives at that address now.

Overhead compared with preloading:

Where Preloading jmethodIDs fjmethodid=lazy
Memory a JNI method-ID slot per method plus a per-class ID cache, for every loaded class; before JDK 26 the slots are never freed nothing between dumps; during a dump with lazy frames, a set of ≥ 2 × 8 B per loaded class; 24 B plus a weak ref per redefined class
Class loading (app threads) GetClassMethods plus the jmethodID-creation lock at every ClassPrepare nothing
Signal handler reads the jmethodID cache same read; on a miss it stores the raw Method* plus its tag: four SafeAccess loads (holder chain and trace ID)
Dump thread JVMTI lookups for every method row once per dump with lazy frames: one Klass* read per loaded class (typically 10–70 ns each). Once per distinct raw method per dump: crash-protection setup, pinned-set lookup, redefinition mark (registry mutex), tag and methods checks, metadata copy and line-table decode

Measurements:

All on x64 Linux (AWS VM, 16 vCPU, not bare metal), release build, cpu=10ms,cstack=vm, -Xms4g -Xmx4g. Each run is 40 s with a dump every 10 s; 5 fresh JVMs per mode, with modes interleaved. Every lazy run confirmed lazy mode was active.

Memory, NMT committed Class + Internal Δ over the no-profiler run, in MiB. These Renaissance runs used the earlier version of this PR, which kept a registry entry and a weak ref for every loaded class; the lazy numbers include those weak refs, and the registry table (not visible to NMT) came on top. Both are gone now, so lazy's cost here can only be lower.

Workload JDK 11: true / lazy JDK 21: true / lazy JDK 25: true / lazy
dotty +2.3 / +0.1 +2.2 / +0.2 +2.2 / +0.2
finagle-http +2.6 / +0.8 +2.3 / +0.3 +2.3 / +0.2
als (Spark) +5.1 / +0.4 +4.3 / +0.3 +4.4 / +0.3

About 16,000 raw frames per run are resolved in these lazy runs. Preloading puts the per-class jmethodID cache under Class (NEW_C_HEAP_ARRAY(jmethodID, …, mtClass)) and the method-ID slots under Internal (JNIMethodBlockNode), 8 B per method each.

Synthetic: 1M methods (macOS arm64, JDK 25)

NMT committed Δ in KB, 20,000 classes × 50 methods in custom class loaders, median of 3. "sampled" also calls every class's methods for 8 s at cpu=1ms and dumps before measuring; ~9,300 raw frames are resolved in lazy mode.

Mode Class (load) Internal (load) Class (sampled) Internal (sampled)
no profiler +14,233 +10 +14,236 +22
fjmethodid=true +22,981 +8,136 +22,982 +8,409
fjmethodid=false +22,981 +8,135 +22,981 +8,334
fjmethodid=lazy +14,233 +191 +14,234 +471

The extra ~280 KB of Internal in the sampled runs appears with true and false too, so it comes from sampling and dumping, not from jmethodIDs.

Lambda churn: ~1.1M short-lived 1-method classes (macOS arm64, release)

4 threads keep defining hidden classes (JDK 15+) with one method each, run it briefly so it gets sampled, and drop it, at ~18,000 classes/s for 60 s; then System.gc() and 20 s without new classes, so every churned class is unloaded. Dump every 10 s. Profiler cost over the no-profiler run once everything is unloaded: NMT Class + Internal, plus the table NMT can't see where there is one.

JDK always-preload (true) lazy, earlier version (registry of all classes) lazy, this PR
17 +110.3 MiB +2.0 MiB +0.4 MiB
21 +111.8 MiB +9.0 MiB +1.7 MiB
25 +109.7 MiB +3.9 MiB +0.7 MiB
26 +1.9 MiB +16.9 MiB +4.2 MiB
  • Before JDK 26 preloading never frees a jmethodID slot, so it grows by ~100 B per class ever loaded. JDK 26 frees them (JDK-8268406).
  • The registry version kept dead entries and their weak refs until the table had to grow, which after a burst might be never; that made it worse than preloading on JDK 26. Pinning per dump keeps nothing.
  • On JDK 26 lazy still keeps 2.3 MiB more than preloading; where that comes from isn't known yet.
  • Building the dump's set: 0.4–1.5 ms with 30–90k loaded classes, 2–20 ms with 120–610k. One dump took 164 ms for it, with 476k classes loaded; it didn't recur in three more runs with safepoint logging (no safepoint overlapped the slower passes), and its cause isn't known.

Dump latency, median ms:

Workload JDK First true First lazy Later true Later lazy
finagle-http 11 / 21 / 25 131 / 137 / 137 112 / 116 / 117 105 / 107 / 97 92 / 91 / 87
dotty 11 / 21 / 25 75 / 54 / 57 70 / 67 / 48 41 / 38 / 38 38 / 36 / 37
als 11 / 21 / 25 26 / 29 / 29 24 / 28 / 25 75 / 70 / 71 60 / 64 / 58
  • On finagle-http, lazy is about 15% faster for both the first and later dumps, with min–max ranges that don't overlap. Elsewhere the modes are within noise; JDK 21's first dotty dump overlaps (lazy 46–70, true 49–77).
  • Raw frames are resolved again in every dump, since they never gain jmethodIDs. That doesn't make later dumps slower.
  • Renaissance iteration counts were the same in all modes.

Names of classes unloaded before the dump are lost: such frames serialize as unknown. With jmethodIDs they are lost too, as <unloaded>. In a survey of all 25 Renaissance benchmarks (lazy, JDK 21, 60 s each), 6,191 classes were unloaded and 269,803 raw frames resolved, and only 6 frames were lost (0.002%). Heavy synthetic churn (~800 classes dying per second) with 60 s dumps loses a large share. Keeping those names is a possible follow-up, but it needs work during GC pauses, so it's left out here.

Additional Notes:

  • Mode selection uses the stack walker mode as the profiler resolved it, not the raw cstack argument. Before, fjmethodid=false with cstack=default on a JVM where that resolves to dwarf skipped preloading, although dwarf gets its Java frames from AsyncGetCallTrace, which needs jmethodIDs.
  • JDK 26+: lazy mode runs unchanged. It doesn't depend on dereferenceable jmethodIDs, which JDK-8268406 removed. Tested locally on Corretto 26.0.2.1 and 27; no CI image has JDK 26 yet.
  • Counter method_resolve_lazy_class_loaded from earlier iterations is gone; method_resolve_lazy_resolved counts raw frames resolved per dump.
Guarantees and limits
  • No crash path: every read of a Method* or its metadata goes through SafeAccess or runs under a landing pad, and reads past the holder only happen while the class is pinned. The holder itself is only a lookup key.
  • Unloading, replacement, metadata reuse: a frame is only read if its holder is one of the classes the dump pins, and its tag matches that class's trace ID; whatever class now owns the memory a stale Method* points into is a different load with a different ID. Classes loaded after the dump's GetLoadedClasses call, and classes whose Klass* can't be read, aren't pinned; their frames resolve as unknown.
  • Redefinition: refused through the mark described above. This relies on every redefinition going through the hooked JVMTI functions. If a mark can't be recorded (no memory for the registry entry), every lazy frame is refused from then on (klass_registry_add_failed).
  • Tag collisions, the explicit limit:
    • Trace-ID tags can only collide once two classes' IDs are ≥ 2^25 apart, so a JVM that has created fewer than 33,554,432 classes cannot misattribute.
    • Beyond that, a misattribution needs all of: the sampled class is unloaded before its frame is resolved; a newer class takes over the exact Method* address; and the IDs differ by an exact multiple of 2^25.
    • The result would be one frame attributed to a live method of the wrong class, never a crash.
    • klass_trace_id_beyond_tag_range counts registrations past the tag range.
  • Naming differences from the jmethodID path:
    • On JDK 8–14, JVMTI appends /<identity hash> to VM-anonymous class names (lambdas there); lazy mode reports them without it.
    • A method can get two method rows in one chunk: one from raw frames and one from frames sampled after its class gained jmethodIDs (after a redefinition, or created by another agent). The names are identical.
  • Line numbers need the "has line number table" constant; JDK 21 early-access builds before 21+26 lack it and get no line numbers. Every GA release has it.
  • Open: with the earlier registry, dotty on JDK 11 counted 14 unloaded frames over 5 runs although the JVM unloaded no classes. Those frames resolved as unknown, not as a wrong method, and the cause wasn't found. JDK 21 and 25 showed 0. Not re-measured with per-dump pinning yet.
  • Possible follow-ups to narrow the pinning: frames of the boot, platform and application loaders' classes don't need it, since those never unload; comparing the holder's ClassLoaderData with those loaders plus the trace-ID check would do, and only other classes would go into the set. Collecting the dump's distinct holders first would bound the set by the sampled classes. Neither avoids the full GetLoadedClasses call, which the dump also makes for the jmethodID path.
Alternatives considered
  • A registry of every loaded class (Klass* → weak ref, filled at ClassPrepare), pinned per frame through NewLocalRef: an earlier version of this PR. It cost a hash insert and a weak ref at every class load, and after heavy class churn its dead entries could stay until the table had to grow, which made it worse than preloading on JDK 26 (+16.9 MiB vs +1.9 MiB in the churn benchmark). The dump already pins every loaded class, so that is used instead.
  • Reading metadata without pinning, relying on crash protection and validation (trace ID, method fingerprint): crash protection only catches unmapped memory, while freed metadata usually stays mapped and readable; a dead class's name Symbols can be freed and reused independently, giving wrong names that no check after the read can catch. The fingerprint only encodes the signature's shape, so many methods share it.
  • Allocating jmethodIDs at dump time, only for sampled classes: removes preloading, but sampled classes still pay the full cost. On Renaissance that still paid 45–65% of the preload memory, and it made the first dump 2–4× slower than with preloading.
  • Keeping names of unloading classes by writing a JFR chunk at every GC start: adds GC-pause time (0.84 ms vs 0.35 ms per young GC in one benchmark) and a chunk per GC.
  • Runtime cross-check of the offset against JFR's JVM.getClassId(): crashed the JVM, because it needs JFR's recorder to exist. The offset is validated against JFR in CI instead (LazyTraceIdJfrValidationTest).
  • Registry-epoch tags without JFR trace IDs: a 25-bit epoch wraps, and bumping it on every registration splits identical stacks into separate traces. JVMs without JFR trace IDs use partial mode instead.
  • A per-version offset table: offsets move between releases and with build options.
  • JFR's own sampler (JDK 25+): doesn't cover JDK 8–21, J9 or Zing.
New counters

method_resolve_lazy_{resolved,unloaded,unreadable,not_current,stale,redefined}, lazy_dump_classes and lazy_dump_pin_ns (gauges), klass_registry_redefinitions_marked, klass_registry_add_failed, klass_registry_{capacity,entries} (gauges) and klass_registry_purged, walkvm_raw_frame_untaggable, klass_trace_id_offset and klass_trace_id_beyond_tag_range (gauges), raw_frame_resolve_cache_hits.

How to test the change?:

  • gtests:
    • klassRegistry_ut: redefinition marks (in progress, failed, overlapping, not carried over to a new class at the same address), table growth and its counters, containsMethod including the scan cap.
    • klassTraceId_ut: offset calibration against decoy layouts.
    • hotspotLazyResolve_ut: end to end on a class without jmethodIDs, checking none are allocated; frames resolve only against the classes a dump pins, the set is only built for lazy frames, a dump without a class list pins nothing, and lookups work among thousands of pinned classes; every reuse and redefinition case; partial-mode frames; frames resolved after a mode switch; method rows and the per-dump cache; hidden-class names; the tag-range gauge.
    • hotspotMethodId_ut: crash protection of resolve(), the malloc fallback for long names, line-table decoding for both UNSIGNED5 variants, the bci bound, a fault in the middle of a table, Thread.run detection.
    • redefineHooks_ut: invalid arguments to the hooks.
    • Additions to frame_ut, arguments_ut and jvmSupport_ut.
  • Java:
    • LazyJmethodIdTest: a custom-loader class's frames resolve with the right line numbers; frames of an unloaded class are dropped; calibration finds the offset.
    • LazyLineNumberConsistencyTest: in a fresh JVM, the same workload runs in lazy mode and with fjmethodid=true. Every (method, bci) in both recordings must have the same line, and no class may keep HotSpot's internal hidden-class name.
    • LazyTraceIdJfrValidationTest: the calibrated trace ID equals JFR's class ID from jdk.ClassDefine (debug builds).
    • LazyTraceIdCalibrationTest: calibration succeeds when some probe dimensions already exist.
    • LazyRedefinitionStressTest: ~1000 live redefinitions racing ~47 dumps.
    • ClinitResolutionTest: <clinit> frames resolve in partial mode.
  • Where it passed:
*Lazy* tests gtests klass_trace_id_offset profiler ID = JFR ID VM vs JVMTI line tables
x64 glibc debug, JDK 8u462 pass (calibration test skips on 8) pass 184 yes 0 mismatches
x64 glibc debug, JDK 11.0.28 / 17.0.16 / 21.0.8 / 25.0.2 pass pass 168 / 168 / 184 / 192 yes 0
x64 musl debug, JDK 21 pass pass 184 yes 0
x64 glibc ASan, JDK 21 pass all 79 binaries, no ASan reports 184 — 0
aarch64 glibc debug, JDK 17 / 21 / 25, and JDK 21 ASan pass pass — yes (debug) 0
aarch64 musl debug, JDK 8 / 11 / 21 pass pass — yes 0
local x64, Corretto 26.0.2.1 and 27 pass — 192 yes 0
macOS arm64, JDK 25 pass debug and release pass 192 yes 0

Line-table comparisons covered 249–389 frames per run. The x64 matrix ran before the calibration retry, the hidden-class naming and per-dump pinning were added. The aarch64 matrix was rerun with per-dump pinning (d0f4041fe): glibc JDK 17 / 21 / 25 debug, JDK 21 ASan, and musl JDK 8 / 11 / 21 debug all pass every gtest and *Lazy* test, with 0 line-table mismatches and matching JFR trace IDs.

For Datadog employees:

  • If this PR touches code that signs or publishes builds or packages, or handles
    credentials of any kind, I've requested a security review (run the dd:platform-security-review
    skill, or file a request via the PSEC review form).
    bewaire also runs automatically on every PR.
  • This PR doesn't touch any of that.
  • JIRA: [JIRA-XXXX]

🤖 Generated with Claude Code

rkennke and others added 15 commits October 15, 2026 11:00
…registry

With fjmethodid=lazy (HotSpot, cstack=vm), no jmethodIDs are preloaded at
ClassPrepare. Instead every class is recorded in KlassRegistry, a Klass* to
weak-mirror table. walkVM keeps storing raw Method* frames for classes
without jmethodIDs; at dump time resolve() reads only the holder Klass* and
idnum from the Method*, pins the registered mirror (null if the class was
unloaded), checks that the Method* is among the pinned class's current
methods, and only then allocates jmethodIDs for that one class.

Falls back to the partial mode when the Klass*/methods vmStructs are
missing. Adds method_resolve_lazy_{unloaded,not_current,class_loaded}
counters, gtests for the registry, containsMethod, argument parsing and the
load-state selection, and a Java test covering custom-loader resolution and
frames of an unloaded class.

Klass* address reuse after class unloading is not handled yet; see
doc/LazyJmethodIdResolution.md (investigation notes, to be removed before
landing).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ass* addresses

A class unloaded and replaced by a new class at the same Klass* (and
Method*) addresses made frames sampled from the old class resolve as
methods of the new one, and made both produce identical raw frames.

KlassRegistry now keeps an epoch, bumped when a known Klass* gets a
different mirror and when a rehash purges dead entries; each entry records
the epoch it was registered at. walkVM stores the current epoch in every
raw frame as a 26-bit tag (10 spare bci bits plus the top 16 bits of the
Method*), and pin() refuses frames older than the entry. A Method* that
uses the tag's address bits is stored as not walkable. If registering a
class fails, its jmethodIDs are preloaded so it never yields raw frames.

resolve() now takes the frame's bci to recover the tag. Adds
method_resolve_lazy_stale, klass_registry_add_failed and
walkvm_raw_frame_untaggable counters, an end-to-end gtest for the
Klass*-reuse cases, epoch tests for the registry and tag round-trip tests
for FrameType.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
KlassTraceId::calibrate() makes the JVM create six array klasses of
java.lang.Void back to back and accepts the single 8-byte-aligned offset
within sizeof(Klass) at which all of them hold distinct JFR trace IDs
(raw >> 16) no further apart than consecutive creation allows; anything
else, including an ambiguous match, leaves the offset unknown. Runs when
fjmethodid=lazy is selected and publishes the result in the
klass_trace_id_offset gauge. Not used by the sampler yet.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
When KlassTraceId found Klass::_trace_id, walkVM now stores the low 26
bits of the holder's trace ID (read with SafeAccess via the new
VMMethod::holderOrNull()) as the raw frame's tag instead of the registry
epoch, and resolveViaRegistry() compares it with the pinned class's trace
ID. The ID is unique per load of a class, so frames of an earlier class at
a reused Klass* are refused without the epoch, and unrelated registry
bumps no longer change a stack's frames. Frames whose holder trace ID is
unreadable are stored as not walkable.

The tag source is chosen once per process: calibration now runs only on
the first lazy start, and the klass_trace_id_offset gauge is republished
on every start since counters are reset then. The end-to-end lazy-resolve
gtest runs every case under both tag sources.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…st JFR in CI

The top bit of a raw frame's tag now says whether the rest is a trace ID
or a registry epoch (25 bits each), and resolve() goes by the frame's own
kind rather than the current tag source.

In debug builds, resolveViaRegistry() logs the trace ID it read for each
class it allocates jmethodIDs for. LazyTraceIdJfrValidationTest runs a
child JVM (new ExternalLauncher mode lazy-trace-id) with the profiler in
lazy mode and JFR's own sampler, and requires JFR's RecordedClass.getId()
for the probe class to equal that ID. The spin class generator moves to a
shared SpinClasses test helper.

The profiler does not call into JFR at runtime: JVM.getClassId() reaches
JFR state that only exists once its recorder has been created, and the
profiler itself initializes jdk.jfr.internal.JVM (tsc.cpp), so that class
being initialized is no sign the recorder exists.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review fixes for fjmethodid=lazy:

- VMKlass::containsMethod() reads the methods array only through
  SafeAccess. Pinning the class does not keep the array alive: a
  redefinition swaps in a new one at a safepoint and frees the old one at
  a later safepoint, neither of which waits for the native dump thread.

- Redefinition: the RedefineClasses/RetransformClasses hooks mark the
  class in KlassRegistry before calling the original function and preload
  its jmethodIDs afterwards. pin() refuses raw frames of a marked class,
  and resolveViaRegistry() checks the mark again after its metadata reads,
  since a redefinition can free a Method* and reuse its address within
  the same class without changing its mirror, trace ID or epoch. New
  counter method_resolve_lazy_redefined; pin() now reports a PinStatus.

- Every registration bumps the registry epoch. Previously only
  replacements and purges did, so a dead class's Method* reused by a class
  loaded at a different Klass* could match an epoch-tagged frame.

- klass_trace_id_beyond_tag_range counts registrations of classes whose
  trace ID no longer fits in the 25-bit tag, the first point at which two
  classes' trace-ID tags can collide.

- klassRegistry_ut includes vmStructs.inline.h so it links in release.

The doc gets a "Guarantees and limits" section stating the exact
condition under which a tag collision can still misattribute a frame.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tion samples

LazyTraceIdJfrValidationTest now records only jdk.ClassDefine in the
child JVM (settings written by the test) and compares the probe class's
RecordedClass.getId() from that event with the trace ID the profiler
logged. JFR emits the event once per class on every JFR-enabled release,
whereas JFR's pre-25 sampler recorded almost no jdk.ExecutionSample
events for the spinning probe, even without the profiler running.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Lookup now keeps a per-dump map from (untagged Method*, raw-frame tag) to
the resolved jmethodID, or nullptr for a refused frame, so a raw method
that recurs across traces is resolved once per dump instead of once per
occurrence. Each resolution sets up crash protection, takes the
KlassRegistry mutex and pins the class through JNI. New counter
raw_frame_resolve_cache_hits. Tests cover repeated occurrences with
different bci and frame type, a fresh cache per dump, and frames from two
loads of a class at the same addresses staying apart.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
RedefineClassesHook and RetransformClassesHook now forward a call with a
positive count and a null array straight to the original function, which
returns JVMTI_ERROR_NULL_POINTER; before, the hooks' own loops
dereferenced the array first and crashed, in every jmethodID mode.

In fjmethodid=lazy, the per-class work before and after the call only
touches elements that JVMTI's GetClassStatus accepts as instance classes
(HotspotSupport::isInstanceClass, also used by registerClass), so a
non-class object or an array class is never looked up as a Klass* or
preloaded.

redefineHooks_ut calls both hooks with stubbed originals in lazy and
full-preload mode: null arrays, non-class, array-class and null elements.
Without the fix its first case crashes with SIGSEGV.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
LazyRedefinitionStressTest loads a small agent by self-attach to get
Instrumentation, then redefines a generated class with its own bytes
about every 2 ms while another thread spins in it, fjmethodid=lazy
samples it and the profiler dumps every 100 ms. Raw frames of the class
are collected for 300 ms before the first redefinition, so the first
dumps afterwards have stale raw frames of a marked class to refuse. The
test requires the JVM to survive, every redefinition to be marked, a
stale raw frame to be refused, and frames sampled after the
redefinitions to resolve by name.

New counter klass_registry_redefinitions_marked counts the marks the
redefinition hooks set. SpinClasses can hand out a class's bytes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Raw frames of fjmethodid=lazy are now always tagged with the holder's JFR
trace ID. When the trace-ID offset can't be calibrated (8u before 8u262,
builds without JFR), lazy mode falls back to partial mode instead of
tagging frames with the registry epoch. The 25-bit epoch wrapped after
2^24 registrations, after which every frame of a long-lived class was
refused, and bumping it on every registration split identical stacks into
separate traces. KlassRegistry no longer keeps an epoch.

- Only lazy mode tags frames. Partial-mode frames are plain Method*s
  again, and resolve() picks the lazy or partial path from the frame's
  marker bit rather than the current mode, so frames that outlive a
  restart in the other mode still resolve the way they were sampled.
- Redefinition: the hooks call KlassRegistry::beginRedefinition() before
  and endRedefinition() after the original function. A class is refused
  while a call runs and for good once one succeeds; a call JVMTI rejects
  leaves it as before and preloads nothing. Redefinitions are tracked in
  every mode once the registry has entries.
- Mode selection uses the stack walker mode the profiler resolved, not
  the cstack argument: cstack=default resolves to dwarf where the VM
  walker is unavailable, and then all jmethodIDs must be preloaded. This
  also applied to fjmethodid=false.
- VMKlass::containsMethod() scans at most 256 methods after the idnum
  fast path.
- readMethodHolder() uses VMMethod::holderOrNull().
- The rehash() comment now matches what the code does.

The prototype design notes (doc/LazyJmethodIdResolution.md) are removed;
their content moves to the PR description.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The parse runs in the thread that reads the child JVM's output, so an
exception there would stop that thread instead of failing the test. A
value that does not parse now leaves the logged ID unset, which the
existing assertion reports.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Working notes for the branch, to be removed before the PR lands.

Co-Authored-By: Zhengyu Gu <zhengyu.gu@datadoghq.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
HotspotSupport::resolve() now reads a raw Method* frame's class name,
method name, signature, Thread.run status and line number table straight
from HotSpot metadata (readMethod(), under crash protection) and returns
them in a ResolvedMethod. In fjmethodid=lazy mode this runs while the
class is pinned through KlassRegistry, after the trace-ID and
current-method checks, so the dump no longer allocates jmethodIDs for
sampled classes. Partial mode reads the same metadata instead of looking
the method up through JNI FindClass/GetMethodID, which also covers
<clinit> without a special case.

- vmStructs: ConstMethod::_flags and _code_size, Klass::_super, the
  version-dependent has-line-number-table constant, and a decoder for
  the compressed line number table (VMLineNumberStream).
- Lookup: raw frames get MethodMap rows keyed by the untagged Method*
  and tag bits (RAW_METHOD_MARK); refused frames never touch a row, and
  rows are refilled at their first use in every chunk. The JVMTI and
  VM-metadata paths share fillJavaMethodNames(); line tables decoded
  from VM metadata are malloc'd and freed with free().
- Counter method_resolve_lazy_class_loaded becomes
  method_resolve_lazy_resolved.
- Tests: the resolve() crash-protection gtests and the line table,
  Thread.run and bci-bound cases; the lazy-resolution gtests now run
  on a class without jmethodIDs and check none are allocated;
  LazyLineNumberConsistencyTest compares decoded line numbers with
  JVMTI's; SpinClasses' generated method has a line number table.

Co-Authored-By: Zhengyu Gu <zhengyu.gu@datadoghq.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Zhengyu Gu <zhengyu.gu@datadoghq.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@datadog-datadog-prod-us1-2

datadog-datadog-prod-us1-2 Bot commented Oct 16, 2026 •

Copy link
Copy Markdown

Pipelines

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 1d1a3d7 | Docs | View more details | Give us feedback!

@dd-octo-sts

dd-octo-sts Bot commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

CI Test Results

Run: #39307521270 | Commit: e8a9437 | Duration: 18m 49s (longest job)

✅ All 32 test jobs passed

Status Overview

JDK glibc-aarch64/debug glibc-amd64/debug musl-aarch64/debug musl-amd64/debug
8 - ✅ - -
8-ibm - ✅ - -
8-j9 ✅ ✅ - -
8-librca - - ✅ ✅
8-orcl - ✅ - -
11 - ✅ - -
11-j9 ✅ ✅ - -
11-librca - - ✅ ✅
17 ✅ ✅ - -
17-graal ✅ ✅ - -
17-j9 ✅ ✅ - -
17-librca - - ✅ ✅
21 ✅ ✅ - -
21-graal ✅ ✅ - -
21-librca - - ✅ ✅
25 ✅ ✅ - -
25-graal ✅ ✅ - -
25-librca - - ✅ ✅

Legend: ✅ passed | ❌ failed | ⚪ skipped | 🚫 cancelled

Summary: Total: 32 | Passed: 32 | Failed: 0


Updated: 2026-10-19 19:24:01 UTC

@dd-octo-sts

dd-octo-sts Bot commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

✅ All 40 integration tests passed

📊 Dashboard · 👷 Pipeline · 📦 1d1a3d77

rkennke and others added 3 commits October 16, 2026 11:12
The test attaches its agent through com.sun.tools.attach.VirtualMachine,
which on JDK 8 lives in the JDK's lib/tools.jar and is not on the test
classpath, so the test was always skipped there. Load the attach API from
tools.jar through a child class loader when it isn't on the classpath.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A lazy raw frame whose Method* no longer leads to a holder Klass* and
ConstMethod (its metadata freed after class unloading, or the read
faulted) was refused without being counted by any
method_resolve_lazy_* counter. Under class-loader churn this is most of
the lost frames, so the loss was largely invisible. Count it as
method_resolve_lazy_unreadable.

Adds a gtest that clears a sampled method's ConstMethod and checks that
only the new counter moves, and prints the counter in
LazyJmethodIdTest's unload test.

Environment: Datadog workspace

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Calibration probed Void arrays of dimensions 1-6 once per process and
kept a failure for good. If some of those dimensions already existed,
their trace IDs were far apart from the new ones, no offset qualified,
and fjmethodid=lazy stayed off for every later start in that JVM.

calibrate() now makes up to four attempts, each on the six dimensions
above the previous attempt's, so every attempt probes klasses it has
just created. Only a found offset is kept; after a failure the next
start tries again. Debug builds log which early exit was taken and dump
the probes' words when no offset qualifies.

LazyTraceIdCalibrationTest creates Void[] and Void[][] with a few hundred
class definitions in between before starting lazy mode in a child JVM,
and requires calibration to succeed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
LazyLineNumberConsistencyTest and LazyTraceIdCalibrationTest parse a
counter from the child JVM's output in the line consumer. A malformed
line threw NumberFormatException from the stdout reader and left the
value unset, which the tests' assumptions then turned into a skip. Keep
the line and fail on it after the child exits.

The line-number workload no longer parses the regex's digits; it merges
their length, which is the same regex and map work.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
rkennke and others added 2 commits October 19, 2026 14:59
HotSpot names a hidden class "<name>+0x<hex>", and JVMTI
GetClassSignature() reports it with that last '+' turned into '.'.
Raw frames copied Klass::_name as is, so a lambda's frames were named
differently with and without jmethodIDs, and one class could appear
under both names in a chunk. readMethod() now applies the same rename;
the hidden flag is not in vmStructs on JDK 21+, so the "+0x<hex>"
suffix identifies the class.

Also correct which frames reach the partial-mode path in resolve().

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Zhengyu Gu <zhengyu.gu@datadoghq.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@rkennke rkennke changed the title Lazy jmethodIDs: resolve raw frames from VM metadata (combines #864 and #867) Add fjmethodid=lazy: profile HotSpot without jmethodIDs Oct 19, 2026
rkennke and others added 2 commits October 19, 2026 17:49
NMT does not see the registry's table, which the profiler allocates with
calloc. klass_registry_capacity and klass_registry_entries (gauges) give
its slots (24 B each) and occupied entries, live or dead, and
klass_registry_purged counts dead entries dropped when the table is
rebuilt. Lazy mode's start republishes the gauges after counters are
reset.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Lazy mode kept a JNI weak reference and a 24-byte table entry for every
loaded class, registered at ClassPrepare. Dead entries were only purged
when the table had to grow, so after heavy class churn they could stay
for good: on JDK 26, where the JVM frees jmethodIDs, that left lazy mode
using more memory than preloading.

Every dump already pins all loaded classes: Recording::finishChunk()
holds the local references GetLoadedClasses() returns until it ends.
The dump now hands that list to HotspotSupport::beginDump(), and the
first lazy resolution of the dump collects the classes' Klass*s into a
temporary set, freed by endDump(). A raw frame is read only if its
holder is in that set; the trace-ID, current-method and redefinition
checks are unchanged. A dump without lazy frames builds no set.

ClassPrepare no longer does anything in lazy mode. KlassRegistry now
only holds the redefinition marks: the hooks register the class they
mark, and a mark is no longer reported once that class has been
unloaded, so a new class at the same address starts unmarked.

New gauges lazy_dump_classes and lazy_dump_pin_ns report the classes a
dump pinned and the time it took to collect them;
klass_trace_id_beyond_tag_range becomes a gauge computed from them.
klass_registry_add_failed is removed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- finishChunk() always calls beginDump(), with no list if
  GetLoadedClasses() failed, and ends the dump through a scope guard, so
  an exception out of writeCpool() can't leave a later dump reading a
  stale class list whose local references are gone.
- A redefinition whose mark can't be recorded (no memory for the
  registry entry) is counted in klass_registry_add_failed, and every lazy
  frame is refused from then on instead of being trusted unmarked.
- _lazy_used is read by redefining threads, so it is published with
  release/acquire atomics.
- The per-dump Klass* table is mmap'd (OS::safeAlloc): at hundreds of
  thousands of classes it is megabytes, allocated and freed every dump,
  which would fragment the malloc arenas.
- Lazy mode skips the start-up GetLoadedClasses() pass, which only
  registered classes.
- Stale comments describing the class registry are updated.

New gtests cover a dump without a class list, a lookup among 3000
pinned classes, an unmarkable redefinition, and a redefinition before
lazy mode started.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant