Reference-chains design and architecture docs

doc/README.md

@@ -1,3 +1,5 @@
+<!-- Copyright 2026, Datadog, Inc. -->
+
 # Java Profiler Documentation
 
 ## Directory Structure
@@ -17,10 +19,11 @@ All documentation files use **PascalCase** naming (e.g., `BuildSystemGuide.md`).
 
 ### Architecture
 - [CallTraceStorage](architecture/CallTraceStorage.md) - Triple-buffer architecture for call traces
+- [LiveHeapReferenceChainsImplementation](architecture/LiveHeapReferenceChainsImplementation.md) - Current reference-chain implementation
 - [RefCountGuard](architecture/RefCountGuard.md) - Lock-free RAII reference-counting primitive used to drain readers before resource reclamation
+- [ReferenceChainsSignalsExplained](architecture/ReferenceChainsSignalsExplained.md) - Reference-chain scheduling and cost control
 - [StringDictionary](architecture/StringDictionary.md) - Concurrency model: RefCountGuard, clearAll, and rotation protocols
 - [TLSContext](architecture/TLSContext.md) - Thread-local context for distributed tracing
-- [TLSPriming](architecture/TLSPriming.md) - Signal-safe TLS initialization
 
 ### Build System
 - [BuildSystemGuide](build/BuildSystemGuide.md) - Comprehensive build system documentation
@@ -36,5 +39,10 @@ All documentation files use **PascalCase** naming (e.g., `BuildSystemGuide.md`).
 - [SamplerPerfCounters](reference/SamplerPerfCounters.md) - Opt-in per-sampler timing counters and the stop-time report
 - [TestFlakinessAnalysis](reference/TestFlakinessAnalysis.md) - Test flakiness investigation results
 
+### Design
+
+- [ReferenceChainsCollectionSummary](ReferenceChainsCollectionSummary.md) - Short reference-chain overview and limits
+- [ReferenceChainsDesign](ReferenceChainsDesign.md) - Reference-chain design decision and trade-offs
+
 ### Work State (temp/)
 Session logs and analysis reports for ongoing work. Not considered permanent documentation.

doc/ReferenceChainsCollectionSummary.md

@@ -0,0 +1,119 @@
+<!-- Copyright 2026, Datadog, Inc. -->
+
+# Reference-chain collection summary
+
+## Purpose
+
+Reference-chain collection helps explain why a sampled object remains live. It records a bounded chain of referrer classes from a GC root to a sampled object. It does not replace a heap dump or compute an exact retained set.
+
+The profiler disables this feature by default. Enable generation tracking and reference-chain
+collection with `generations=true,referencechains=true`. The bare `referencechains=true` option
+does not activate the allocation engine that supplies candidates.
+
+## Data flow
+
+1. `LivenessTracker` counts sampled live objects by class across GC epochs.
+2. It selects classes whose sampled population has sustained growth.
+3. It assigns leak tags to tracked instances of those classes.
+4. `ReferenceChainTracker` walks from roots toward those tagged instances.
+5. The tracker stores its frontier as JVMTI object tags and resumes it in later passes.
+6. When the walk reaches a tagged instance, the tracker reconstructs its chain.
+7. The dump path writes the chain as a `datadog.ReferenceChain` JFR event.
+
+In this document, *class* means the Java class. Source identifiers use the existing `klass` spelling.
+
+## Candidate selection
+
+`LivenessTracker` keeps a fixed-size population table. Each entry has a ring of 30 GC-epoch samples. The tracker requires at least 10 samples before it evaluates a trend.
+
+A class must show sustained growth. The base hysteresis is five qualifying epochs. It decreases to three epochs when the post-GC heap floor also rises. The tracker returns at most five candidate classes.
+
+The population table has at most 256 entries. A linear scan is therefore bounded and occurs outside the allocation hot path. The existing table lock protects the population data.
+
+Candidate selection uses sampled objects. It can miss a leak when sampling does not retain enough representative objects or when the population does not satisfy the trend gates.
+
+## Resumable walk
+
+`ReferenceChainTracker` owns a background thread and a persistent frontier. The frontier contains positive JVMTI tags. JVMTI preserves tag identity when the GC moves an object. A tag does not keep an object live.
+
+Each pass can perform these operations:
+
+1. Enumerate roots and stack references. The first pass always does this. Later passes retry a truncated enumeration or wait at least two seconds.
+2. Walk candidate thread objects when the candidate data identifies qualifying threads.
+3. Scan static fields in chunks when the loaded-class count changes.
+4. Expand a batch of frontier entries.
+5. Revisit selected entries whose references can change after their first expansion.
+
+The walk records parent tags, class identifiers, depth, root kind, and leak-tag data in the frontier table. These records let the tracker reconstruct a chain after the walk reaches a candidate instance.
+
+The walk is best effort. The heap can change between passes. A recorded chain is therefore historical evidence from several observations, not a single consistent heap snapshot.
+
+## Candidate correlation
+
+`LivenessTracker::tagLeakInstances()` assigns positive leak tags from a fixed pool. The heap walk correlates a leak tag with a frontier entry when it reaches that object. The tracker can keep up to eight discovered instances for each watched class.
+
+The implementation does not pre-tag candidate representatives with negative marker tags. Older design material that describes marker tags is obsolete.
+
+While a candidate search is active, `LivenessTracker` admits allocations from qualifying threads at 100 percent. Other allocations still use the configured liveness-sampling ratio. This admission increase improves correlation for small per-thread populations.
+
+## Mutable-container coverage
+
+A one-time breadth-first walk cannot observe references that appear after the first expansion of an object. This matters for a long-lived collection that receives new elements.
+
+The tracker addresses this case with bounded rotation:
+
+- Leak-accumulation rotation selects parent objects that receive instances of watched classes.
+- Stale-entry rotation revisits older expanded entries.
+- Root-kind rotation revisits entries with transient root attribution.
+- Static-anchor rotation descends from selected objects held by static fields.
+
+Rotation improves coverage. It does not guarantee that the tracker observes every mutation.
+
+## Cost controls
+
+The implementation uses several independent controls:
+
+- An edge-admission budget limits work admitted by callbacks.
+- A pause target guides a PID controller that adjusts the edge budget and fallback cadence.
+- A safepoint-cost budget gates the start of a later search.
+- A CPU-cost budget gates passes in a running search.
+- A frontier cap limits metadata growth.
+- A search TTL and no-progress checks stop searches that do not converge.
+
+Most traversal work is resumable. A callback asks JVMTI to stop when it observes the
+pause deadline, and a later pass continues from the saved frontier. The operation can
+still exceed the target because the JVM can work between callbacks.
+
+Root enumeration is the main exception. JVMTI does not expose a cursor for resuming
+`IterateOverReachableObjects()`, and the edge budget counts admitted roots rather than
+all examined roots. For this release, a root-enumeration pause is acceptable if it is
+shorter than three seconds. This is an acceptance threshold, not the desired latency or
+a hard real-time guarantee. [PROF-16284](https://datadoghq.atlassian.net/browse/PROF-16284)
+tracks measurement and improvement of this phase.
+
+`stopThread()` requests cooperative cancellation. The heap callback checks the request on each invocation. Cancellation therefore depends on the JVM reaching another callback; it is not instantaneous.
+
+## Completion and abandonment
+
+A search completes when all current candidates have chains, or when no work remains and no class remains under watch.
+
+A search can stop with one of these abandonment reasons:
+
+- `FRONTIER_CAP`: the frontier table cannot admit more entries.
+- `TTL`: the configured lifetime expired, or the ordinary frontier made no progress for 30 passes.
+- `CANARY_STUCK`: candidate discovery and frontier growth both remained stalled for the configured checks.
+
+The tracker reports abandonment through `datadog.ReferenceChainAbandoned`. It releases search-owned tags before a restart. A restart also requires a current leak signal and sufficient safepoint-cost budget.
+
+## Output and limits
+
+The resolved-chain cache holds at most 128 entries. The dump path copies valid cached chains without clearing the cache, so a live sample can appear in later JFR chunks. The cache removes a chain when the leak-tag ownership no longer matches the recorded instance.
+
+The output has these limits:
+
+- It contains referrer classes and selected edge metadata, not a complete object graph.
+- It can omit paths because of sampling, hop limits, budgets, mutation, or abandonment.
+- It can describe observations from different points in time.
+- It does not prove that a reported path is the shortest path or the only retaining path.
+
+For implementation details, see [LiveHeapReferenceChainsImplementation.md](architecture/LiveHeapReferenceChainsImplementation.md). For scheduling details, see [ReferenceChainsSignalsExplained.md](architecture/ReferenceChainsSignalsExplained.md).

doc/ReferenceChainsDesign.md

@@ -0,0 +1,121 @@
+<!-- Copyright 2026, Datadog, Inc. -->
+
+# Reference chains for surviving live-heap samples
+
+**Status:** Implemented
+
+## Decision
+
+Use an incremental breadth-first walk from GC roots. Persist the frontier with JVMTI object tags. Run the walk in multiple bounded-work passes.
+
+This document records the design decision. It does not describe every current implementation detail. See [LiveHeapReferenceChainsImplementation.md](architecture/LiveHeapReferenceChainsImplementation.md) for the as-built behavior.
+
+## Goal
+
+For selected live-heap samples, record a chain of referrer classes from a GC root to the sampled object.
+
+The chain is diagnostic data. It is not an exact retainer analysis.
+
+## Constraints
+
+- The profiler must work on supported stock JDKs.
+- The profiler must not require a patched JVM.
+- Graph expansion must not require one unbounded stop-the-world operation.
+- The feature must limit memory use and background work.
+- The result can be incomplete.
+
+## Non-goals
+
+- Find every path to every GC root.
+- Compute retained sizes or dominators.
+- Record a complete heap graph.
+- Provide a consistent heap snapshot across all passes.
+- Guarantee a fixed maximum pause time.
+
+## Options
+
+### Full reverse-reachability walk
+
+A complete reverse graph gives strong results, but its work grows with the live heap. Starting it during an existing GC pause does not remove that work. Many collectors also do not provide a suitable full-heap pause on demand.
+
+We rejected this option because it does not meet the pause-distribution goal.
+
+### Incremental breadth-first walk
+
+HotSpot JFR uses a root-based graph search for old-object paths. The profiler uses the same general direction, but it persists work between passes and records a smaller result.
+
+We chose this option because JVMTI supplies the required portable operations:
+
+- Root and stack-reference enumeration.
+- Reference traversal.
+- Object tags that survive object relocation.
+- Object lookup by tag.
+
+This option remains best effort. A moving and mutating heap can change between passes.
+
+### Collector-internal hooks
+
+G1, Shenandoah, and ZGC have internal traversal mechanisms. An external agent does not have a stable API for these implementation details. Calling private symbols would bind the profiler to JVM build details and collector internals.
+
+We rejected this option because the profiler supports multiple JDK vendors and versions.
+
+## Why tags make resumption possible
+
+The GC can move an object between passes. A raw address would then become invalid. A JVMTI tag identifies the object after relocation.
+
+A tag does not keep the object live. If the object dies before the next pass, lookup by tag does not return it. The tracker can then remove that branch from pending work.
+
+The frontier table stores the information needed to reconstruct a chain:
+
+- The object tag.
+- The parent tag.
+- The referrer class.
+- The reference depth.
+- The root kind.
+- The candidate leak tag, when available.
+
+The implementation places fixed caps on the frontier and on chain depth.
+
+## Pass model
+
+The tracker separates root discovery from frontier expansion.
+
+1. Root enumeration admits root-attached objects.
+2. Static-field scanning admits objects that root callbacks do not report as static-field edges.
+3. Frontier expansion resolves a batch of tags and follows one bounded step from those objects.
+4. Rotation revisits selected expanded objects whose outgoing references can change.
+
+The edge budget and callback deadline stop admission after the current limit. A later
+pass continues frontier expansion from saved tags. JVMTI can still perform work before
+or between callbacks, so the deadline is not a hard real-time bound.
+
+Root enumeration is not resumable through the JVMTI API. Aborting it and retrying starts
+again from the first root, which can prevent later roots from being examined. The current
+release therefore permits this phase to finish. For this release, a root-enumeration
+pause is acceptable if it is shorter than three seconds. Improvement is
+tracked in [PROF-16284](https://datadoghq.atlassian.net/browse/PROF-16284).
+
+## Scheduling
+
+The JVMTI specification restricts functions that an agent can call from heap and GC callbacks. The profiler therefore does not run the heap walk from a GC callback. A GC callback only updates an atomic epoch.
+
+The background thread calls the JVMTI heap functions. The JVM performs the required safepoint operation for those calls.
+
+The scheduler uses the GC epoch, a fallback cadence, leak signals, adaptive cost budgets, and candidate-search backoff. See [ReferenceChainsSignalsExplained.md](architecture/ReferenceChainsSignalsExplained.md).
+
+## Correctness model
+
+The tracker constructs a chain from observations that can occur in different passes. It guarantees neither a single-snapshot view nor the shortest retaining path.
+
+The chain remains useful when it identifies a plausible root class and a sequence of referrer classes. Consumers must treat missing chains as an expected outcome. Sampling, mutation, depth limits, time limits, and frontier limits can all prevent a result.
+
+## Trade-offs
+
+The design improves pause distribution. It does not guarantee lower total stop-the-world time. Repeated passes can use more total work than one complete walk because they resolve tags and revisit entries.
+
+The design also uses native metadata for the frontier. The frontier cap bounds this memory use. A cap hit abandons the search and emits a diagnostic event.
+
+## Sources
+
+- [JVM TI specification: Heap functions](https://docs.oracle.com/en/java/javase/25/docs/specs/jvmti.html#Heap)
+- [OpenJDK JFR leak-profiler chain implementation](https://github.com/openjdk/jdk/tree/master/src/hotspot/share/jfr/leakprofiler/chains)