diff --git a/doc/README.md b/doc/README.md index de25fe24fe..2ca411ea74 100644 --- a/doc/README.md +++ b/doc/README.md @@ -1,3 +1,5 @@ + + # Java Profiler Documentation ## Directory Structure @@ -17,10 +19,11 @@ All documentation files use **PascalCase** naming (e.g., `BuildSystemGuide.md`). ### Architecture - [CallTraceStorage](architecture/CallTraceStorage.md) - Triple-buffer architecture for call traces +- [LiveHeapReferenceChainsImplementation](architecture/LiveHeapReferenceChainsImplementation.md) - Current reference-chain implementation - [RefCountGuard](architecture/RefCountGuard.md) - Lock-free RAII reference-counting primitive used to drain readers before resource reclamation +- [ReferenceChainsSignalsExplained](architecture/ReferenceChainsSignalsExplained.md) - Reference-chain scheduling and cost control - [StringDictionary](architecture/StringDictionary.md) - Concurrency model: RefCountGuard, clearAll, and rotation protocols - [TLSContext](architecture/TLSContext.md) - Thread-local context for distributed tracing -- [TLSPriming](architecture/TLSPriming.md) - Signal-safe TLS initialization ### Build System - [BuildSystemGuide](build/BuildSystemGuide.md) - Comprehensive build system documentation @@ -36,5 +39,10 @@ All documentation files use **PascalCase** naming (e.g., `BuildSystemGuide.md`). - [SamplerPerfCounters](reference/SamplerPerfCounters.md) - Opt-in per-sampler timing counters and the stop-time report - [TestFlakinessAnalysis](reference/TestFlakinessAnalysis.md) - Test flakiness investigation results +### Design + +- [ReferenceChainsCollectionSummary](ReferenceChainsCollectionSummary.md) - Short reference-chain overview and limits +- [ReferenceChainsDesign](ReferenceChainsDesign.md) - Reference-chain design decision and trade-offs + ### Work State (temp/) Session logs and analysis reports for ongoing work. Not considered permanent documentation. diff --git a/doc/ReferenceChainsCollectionSummary.md b/doc/ReferenceChainsCollectionSummary.md new file mode 100644 index 0000000000..65e8122f51 --- /dev/null +++ b/doc/ReferenceChainsCollectionSummary.md @@ -0,0 +1,119 @@ + + +# Reference-chain collection summary + +## Purpose + +Reference-chain collection helps explain why a sampled object remains live. It records a bounded chain of referrer classes from a GC root to a sampled object. It does not replace a heap dump or compute an exact retained set. + +The profiler disables this feature by default. Enable generation tracking and reference-chain +collection with `generations=true,referencechains=true`. The bare `referencechains=true` option +does not activate the allocation engine that supplies candidates. + +## Data flow + +1. `LivenessTracker` counts sampled live objects by class across GC epochs. +2. It selects classes whose sampled population has sustained growth. +3. It assigns leak tags to tracked instances of those classes. +4. `ReferenceChainTracker` walks from roots toward those tagged instances. +5. The tracker stores its frontier as JVMTI object tags and resumes it in later passes. +6. When the walk reaches a tagged instance, the tracker reconstructs its chain. +7. The dump path writes the chain as a `datadog.ReferenceChain` JFR event. + +In this document, *class* means the Java class. Source identifiers use the existing `klass` spelling. + +## Candidate selection + +`LivenessTracker` keeps a fixed-size population table. Each entry has a ring of 30 GC-epoch samples. The tracker requires at least 10 samples before it evaluates a trend. + +A class must show sustained growth. The base hysteresis is five qualifying epochs. It decreases to three epochs when the post-GC heap floor also rises. The tracker returns at most five candidate classes. + +The population table has at most 256 entries. A linear scan is therefore bounded and occurs outside the allocation hot path. The existing table lock protects the population data. + +Candidate selection uses sampled objects. It can miss a leak when sampling does not retain enough representative objects or when the population does not satisfy the trend gates. + +## Resumable walk + +`ReferenceChainTracker` owns a background thread and a persistent frontier. The frontier contains positive JVMTI tags. JVMTI preserves tag identity when the GC moves an object. A tag does not keep an object live. + +Each pass can perform these operations: + +1. Enumerate roots and stack references. The first pass always does this. Later passes retry a truncated enumeration or wait at least two seconds. +2. Walk candidate thread objects when the candidate data identifies qualifying threads. +3. Scan static fields in chunks when the loaded-class count changes. +4. Expand a batch of frontier entries. +5. Revisit selected entries whose references can change after their first expansion. + +The walk records parent tags, class identifiers, depth, root kind, and leak-tag data in the frontier table. These records let the tracker reconstruct a chain after the walk reaches a candidate instance. + +The walk is best effort. The heap can change between passes. A recorded chain is therefore historical evidence from several observations, not a single consistent heap snapshot. + +## Candidate correlation + +`LivenessTracker::tagLeakInstances()` assigns positive leak tags from a fixed pool. The heap walk correlates a leak tag with a frontier entry when it reaches that object. The tracker can keep up to eight discovered instances for each watched class. + +The implementation does not pre-tag candidate representatives with negative marker tags. Older design material that describes marker tags is obsolete. + +While a candidate search is active, `LivenessTracker` admits allocations from qualifying threads at 100 percent. Other allocations still use the configured liveness-sampling ratio. This admission increase improves correlation for small per-thread populations. + +## Mutable-container coverage + +A one-time breadth-first walk cannot observe references that appear after the first expansion of an object. This matters for a long-lived collection that receives new elements. + +The tracker addresses this case with bounded rotation: + +- Leak-accumulation rotation selects parent objects that receive instances of watched classes. +- Stale-entry rotation revisits older expanded entries. +- Root-kind rotation revisits entries with transient root attribution. +- Static-anchor rotation descends from selected objects held by static fields. + +Rotation improves coverage. It does not guarantee that the tracker observes every mutation. + +## Cost controls + +The implementation uses several independent controls: + +- An edge-admission budget limits work admitted by callbacks. +- A pause target guides a PID controller that adjusts the edge budget and fallback cadence. +- A safepoint-cost budget gates the start of a later search. +- A CPU-cost budget gates passes in a running search. +- A frontier cap limits metadata growth. +- A search TTL and no-progress checks stop searches that do not converge. + +Most traversal work is resumable. A callback asks JVMTI to stop when it observes the +pause deadline, and a later pass continues from the saved frontier. The operation can +still exceed the target because the JVM can work between callbacks. + +Root enumeration is the main exception. JVMTI does not expose a cursor for resuming +`IterateOverReachableObjects()`, and the edge budget counts admitted roots rather than +all examined roots. For this release, a root-enumeration pause is acceptable if it is +shorter than three seconds. This is an acceptance threshold, not the desired latency or +a hard real-time guarantee. [PROF-16284](https://datadoghq.atlassian.net/browse/PROF-16284) +tracks measurement and improvement of this phase. + +`stopThread()` requests cooperative cancellation. The heap callback checks the request on each invocation. Cancellation therefore depends on the JVM reaching another callback; it is not instantaneous. + +## Completion and abandonment + +A search completes when all current candidates have chains, or when no work remains and no class remains under watch. + +A search can stop with one of these abandonment reasons: + +- `FRONTIER_CAP`: the frontier table cannot admit more entries. +- `TTL`: the configured lifetime expired, or the ordinary frontier made no progress for 30 passes. +- `CANARY_STUCK`: candidate discovery and frontier growth both remained stalled for the configured checks. + +The tracker reports abandonment through `datadog.ReferenceChainAbandoned`. It releases search-owned tags before a restart. A restart also requires a current leak signal and sufficient safepoint-cost budget. + +## Output and limits + +The resolved-chain cache holds at most 128 entries. The dump path copies valid cached chains without clearing the cache, so a live sample can appear in later JFR chunks. The cache removes a chain when the leak-tag ownership no longer matches the recorded instance. + +The output has these limits: + +- It contains referrer classes and selected edge metadata, not a complete object graph. +- It can omit paths because of sampling, hop limits, budgets, mutation, or abandonment. +- It can describe observations from different points in time. +- It does not prove that a reported path is the shortest path or the only retaining path. + +For implementation details, see [LiveHeapReferenceChainsImplementation.md](architecture/LiveHeapReferenceChainsImplementation.md). For scheduling details, see [ReferenceChainsSignalsExplained.md](architecture/ReferenceChainsSignalsExplained.md). diff --git a/doc/ReferenceChainsDesign.md b/doc/ReferenceChainsDesign.md new file mode 100644 index 0000000000..b088ff8794 --- /dev/null +++ b/doc/ReferenceChainsDesign.md @@ -0,0 +1,121 @@ + + +# Reference chains for surviving live-heap samples + +**Status:** Implemented + +## Decision + +Use an incremental breadth-first walk from GC roots. Persist the frontier with JVMTI object tags. Run the walk in multiple bounded-work passes. + +This document records the design decision. It does not describe every current implementation detail. See [LiveHeapReferenceChainsImplementation.md](architecture/LiveHeapReferenceChainsImplementation.md) for the as-built behavior. + +## Goal + +For selected live-heap samples, record a chain of referrer classes from a GC root to the sampled object. + +The chain is diagnostic data. It is not an exact retainer analysis. + +## Constraints + +- The profiler must work on supported stock JDKs. +- The profiler must not require a patched JVM. +- Graph expansion must not require one unbounded stop-the-world operation. +- The feature must limit memory use and background work. +- The result can be incomplete. + +## Non-goals + +- Find every path to every GC root. +- Compute retained sizes or dominators. +- Record a complete heap graph. +- Provide a consistent heap snapshot across all passes. +- Guarantee a fixed maximum pause time. + +## Options + +### Full reverse-reachability walk + +A complete reverse graph gives strong results, but its work grows with the live heap. Starting it during an existing GC pause does not remove that work. Many collectors also do not provide a suitable full-heap pause on demand. + +We rejected this option because it does not meet the pause-distribution goal. + +### Incremental breadth-first walk + +HotSpot JFR uses a root-based graph search for old-object paths. The profiler uses the same general direction, but it persists work between passes and records a smaller result. + +We chose this option because JVMTI supplies the required portable operations: + +- Root and stack-reference enumeration. +- Reference traversal. +- Object tags that survive object relocation. +- Object lookup by tag. + +This option remains best effort. A moving and mutating heap can change between passes. + +### Collector-internal hooks + +G1, Shenandoah, and ZGC have internal traversal mechanisms. An external agent does not have a stable API for these implementation details. Calling private symbols would bind the profiler to JVM build details and collector internals. + +We rejected this option because the profiler supports multiple JDK vendors and versions. + +## Why tags make resumption possible + +The GC can move an object between passes. A raw address would then become invalid. A JVMTI tag identifies the object after relocation. + +A tag does not keep the object live. If the object dies before the next pass, lookup by tag does not return it. The tracker can then remove that branch from pending work. + +The frontier table stores the information needed to reconstruct a chain: + +- The object tag. +- The parent tag. +- The referrer class. +- The reference depth. +- The root kind. +- The candidate leak tag, when available. + +The implementation places fixed caps on the frontier and on chain depth. + +## Pass model + +The tracker separates root discovery from frontier expansion. + +1. Root enumeration admits root-attached objects. +2. Static-field scanning admits objects that root callbacks do not report as static-field edges. +3. Frontier expansion resolves a batch of tags and follows one bounded step from those objects. +4. Rotation revisits selected expanded objects whose outgoing references can change. + +The edge budget and callback deadline stop admission after the current limit. A later +pass continues frontier expansion from saved tags. JVMTI can still perform work before +or between callbacks, so the deadline is not a hard real-time bound. + +Root enumeration is not resumable through the JVMTI API. Aborting it and retrying starts +again from the first root, which can prevent later roots from being examined. The current +release therefore permits this phase to finish. For this release, a root-enumeration +pause is acceptable if it is shorter than three seconds. Improvement is +tracked in [PROF-16284](https://datadoghq.atlassian.net/browse/PROF-16284). + +## Scheduling + +The JVMTI specification restricts functions that an agent can call from heap and GC callbacks. The profiler therefore does not run the heap walk from a GC callback. A GC callback only updates an atomic epoch. + +The background thread calls the JVMTI heap functions. The JVM performs the required safepoint operation for those calls. + +The scheduler uses the GC epoch, a fallback cadence, leak signals, adaptive cost budgets, and candidate-search backoff. See [ReferenceChainsSignalsExplained.md](architecture/ReferenceChainsSignalsExplained.md). + +## Correctness model + +The tracker constructs a chain from observations that can occur in different passes. It guarantees neither a single-snapshot view nor the shortest retaining path. + +The chain remains useful when it identifies a plausible root class and a sequence of referrer classes. Consumers must treat missing chains as an expected outcome. Sampling, mutation, depth limits, time limits, and frontier limits can all prevent a result. + +## Trade-offs + +The design improves pause distribution. It does not guarantee lower total stop-the-world time. Repeated passes can use more total work than one complete walk because they resolve tags and revisit entries. + +The design also uses native metadata for the frontier. The frontier cap bounds this memory use. A cap hit abandons the search and emits a diagnostic event. + +## Sources + +- [JVM TI specification: Heap functions](https://docs.oracle.com/en/java/javase/25/docs/specs/jvmti.html#Heap) +- [OpenJDK JFR leak-profiler chain implementation](https://github.com/openjdk/jdk/tree/master/src/hotspot/share/jfr/leakprofiler/chains) diff --git a/doc/architecture/LiveHeapReferenceChainsImplementation.md b/doc/architecture/LiveHeapReferenceChainsImplementation.md new file mode 100644 index 0000000000..e89527e187 --- /dev/null +++ b/doc/architecture/LiveHeapReferenceChainsImplementation.md @@ -0,0 +1,230 @@ + + +# Live-heap reference chains: implementation reference + +**Status:** Implemented and disabled by default + +## Scope + +This document describes the current reference-chain implementation. It focuses on data flow, traversal, lifecycle, limits, and output. + +Related documents: + +- [Reference-chain design](../ReferenceChainsDesign.md) records the design decision and trade-offs. +- [Reference-chain collection summary](../ReferenceChainsCollectionSummary.md) gives a short overview. +- [Reference-chain scheduling and cost control](ReferenceChainsSignalsExplained.md) explains scheduling. + +## Main components + +`LivenessTracker` detects sustained growth in sampled live-object populations. It assigns leak tags to selected instances. + +`ReferenceChainTracker` maintains a root-based frontier and searches for those instances. It runs on one agent-owned background thread. `_engine_lock` serializes engine state across lifecycle and test paths. + +`FrontierTable` stores one record for each admitted object. Important fields include: + +| Field | Purpose | +|---|---| +| `parent_tag` | Identifies the entry that admitted this object. Zero means that the entry is root-attached. | +| `referrer_klass` | Identifies the object's class in the profiler class dictionary. | +| `referrer_class_tag` | Identifies the referrer class with a stable class tag. | +| `depth` | Records the number of edges from the root. | +| `state` | Records `FRONTIER`, `EXPANDED`, `EDGE`, or `ABANDONED`. | +| `root_kind` | Records the JVMTI root or edge kind for a root-attached entry. | +| `class_tag` | Provides class identity across class-dictionary regeneration. | +| `leak_tag` and incarnation | Correlate the entry with the current owner of a sampled leak tag. | + +## Search lifecycle + +A search starts only after the scheduler accepts the leak signal and the cost gate. + +The first pass starts the search timestamp and enumerates roots. Later passes resume the same frontier. A running search does not restart itself. + +A terminal search releases its search-owned tags. A later search can start only after release succeeds, the current leak signal permits it, and the safepoint cost budget permits it. + +`restartSearch()` resets the frontier state and the frontier-tag sequence. Stable class tags and valid resolved-chain entries have separate lifecycles. + +## Pass structure + +`runPass()` resolves loaded classes and calls `runPassManualWalk()`. The pass can contain these phases: + +1. Root and stack-reference enumeration. +2. Candidate thread-object traversal. +3. Static-field scanning. +4. Ordinary frontier expansion. +5. Rotation and static-anchor traversal. + +Candidate thread traversal, ordinary frontier expansion, and rotation each receive a fresh +callback deadline. Static-field scanning does not receive a fresh deadline. It shares the +candidate-thread deadline when that phase runs. Otherwise, it uses the deadline set before root +enumeration. Earlier work can therefore reduce or consume the time available for static-field +scanning. + +The tracker measures total pass time and in-safepoint time separately. It charges the remainder to the CPU cost budget. + +## Root discovery + +### Root and stack references + +`IterateOverReachableObjects()` calls `heapRootCallback()` and `stackRefCallback()`. These callbacks admit root-attached objects. They do not traverse all children of each root. + +The first pass always performs root enumeration. A later pass performs it after a truncated enumeration or an interval of at least two seconds. + +Root enumeration uses `firstpassbudget`. The name is historical: the implementation can use this budget again during later root-enumeration attempts. + +This phase does not use the callback deadline. JVMTI does not expose a cursor for +resuming `IterateOverReachableObjects()`. An aborted attempt starts again from the first +root and can repeatedly omit roots later in the enumeration order. The edge budget does +not bound elapsed time when callbacks examine roots that the frontier already contains. + +For this release, a root-enumeration pause is acceptable if it is shorter than three +seconds. This threshold is not the desired latency and is not a hard real-time +guarantee. [PROF-16284](https://datadoghq.atlassian.net/browse/PROF-16284) +tracks separate measurement and a progress-preserving mitigation. + +### Static fields + +Root callbacks do not provide the static-field edges needed by this implementation. `admitStaticFieldRoots()` therefore scans loaded classes with `FollowReferences()`. + +The scan: + +- Partitions each `GetLoadedClasses()` result so application classes come first. +- Processes at most 512 classes in one chunk. +- Saves a cursor and retries a truncated chunk. +- Admits static-field edges. +- Admits at most 32 other class edges per class. +- Starts a new completed sweep when the loaded-class count changes. + +The loaded-class API does not guarantee stable ordering. The code repartitions each returned array before it uses the saved cursor. + +### Candidate thread objects + +Candidate selection records qualifying native thread identifiers. The tracker maps them to Java `Thread` objects and walks a rotating set of at most four thread objects per pass. + +The descend walk admits at most 16 hops below an anchor and still obeys the global hop cap. This path improves coverage for `ThreadLocal` retention. It does not assume that every candidate is thread-local. + +## Frontier expansion + +`expandFrontier()` takes a batch of pending frontier tags and calls `GetObjectsWithTags()`. Dead objects do not resolve and leave no object to expand. + +The tracker places the resolved objects in one JNI object array. It calls `FollowReferences()` with that array as the initial object. Callback filtering limits normal expansion to the selected boundary objects. + +If JNI setup fails or leaves an exception, the tracker marks the operation as truncated. It does not mark the affected entries as expanded. A later pass can retry them. + +The callback stops admission when it reaches the edge budget, deadline check, hop cap, frontier cap, or abort request. These checks bound admitted work. They do not provide a strict bound on JVMTI operation duration. + +## Candidate tags and correlation + +`pollWatchedTargets()` asks `LivenessTracker` for at most five candidate classes. It then calls `tagLeakInstances()` for the current candidates. + +Leak tags are positive tags from a fixed pool. Each reuse has an incarnation value. The incarnation prevents emission of an old owner's cached chain for a new owner of the same numeric tag. + +Candidate representatives are not pre-tagged with negative marker tags. The walk discovers a candidate when it reaches a leak-tagged instance or correlates a leak tag with an object that the frontier already admitted. + +For each watched class, the tracker keeps at most eight discovered frontier entries. A leak-correlated entry can replace an uncorrelated entry when those slots are full. + +The tracker can complete a candidate search after each occupied candidate slot has produced a valid chain. + +## Mutable references and rotation + +An object can change after its first expansion. For example, a long-lived collection can receive a new backing array or new elements. The tracker uses bounded rotation to revisit selected entries. + +| Rotation source | Per-pass selection limit | Purpose | +|---|---:|---| +| Leak accumulation | 16 | Select parents that accumulate instances of watched classes. | +| Stale expanded entries | 256 | Provide eventual low-priority rechecks across the frontier table. | +| Transient root kinds | 16 | Improve root attribution when later observations provide a more durable root kind. | +| Static anchors | 32 selected, plus 16 FIFO entries | Descend from selected objects held by static fields. | + +Leak accumulation has two levels. The first groups admissions by leaf class and parent class. The second ranks parent object tags within the selected group. + +When a class first becomes watched, the tracker scans existing expanded entries for matching stable class tags. This scan supplies data that admission-time tracking could not collect before the class became watched. + +Rotation improves coverage but remains bounded and best effort. + +## Search termination + +The implementation evaluates terminal conditions after each pass. A search can: + +- Complete when the pass has no pending truncated work and no class remains under watch. +- Complete when all current candidates have valid chains. +- Abandon with `FRONTIER_CAP` when the frontier cannot admit another entry. +- Abandon with `TTL` when the configured search lifetime expires. +- Abandon with `TTL` when the frontier makes no progress for 30 passes and the urgent latch is not active. +- Abandon with `CANARY_STUCK` when both frontier progress and candidate progress remain stalled for their limits. + +The base candidate-stall limit is 30 passes. Consecutive `CANARY_STUCK` restarts can double this limit, up to eight doublings. Other terminal outcomes reset that escalation. + +`datadog.ReferenceChainAbandoned` records an abandonment. The event queue is separate from the resolved-chain cache. + +## Cost control + +The implementation controls different costs separately: + +- `_effective_budget` limits edge admission for ordinary work. +- `_pause_pid` adjusts that budget from measured in-safepoint time. +- `_effective_cadence_ns` changes when the budget controller reaches a limit. +- `_safepoint_pain_budget` gates a later search. +- `_cpu_pain_budget` gates passes in the current search. +- `_configured_frontier_cap` limits native metadata. +- `_ttl_ms` limits search lifetime. + +For resumable traversal phases, the pause target controls when a callback asks JVMTI to +stop. Callback granularity and JVM work can cause overshoot. Root enumeration does not +use this deadline. See [Reference-chain scheduling and cost control](ReferenceChainsSignalsExplained.md). + +## Automatic defaults + +`autoTuneDefaults()` changes only options that the user did not set. + +- The edge budget scales with the square root of the maximum heap size. +- The first-pass budget defaults to 10 times the resulting edge budget, up to its cap. +- The TTL scales with heap size, up to 30 minutes. +- The frontier cap scales with the edge-budget ratio. +- The code caps the pause target at 50 milliseconds. +- The code caps the cost-budget refill percentage at 5 percent. + +These values are tuning inputs. They do not state guaranteed overhead. + +## Configuration + +Reference-chain collection requires the allocation engine and generation tracking. Enable them in +the same command: + +```text +generations=true,referencechains=true:hops=N:budget=N:ttl=N:framecap=N:pausetarget=N:painbudget=N:firstpassbudget=N +``` + +- `generations=true` activates the allocation engine and supplies leak candidates. +- `hops` limits chain depth. +- `budget` supplies the base edge-admission budget. +- `ttl` sets search lifetime in milliseconds. A value at or below zero disables this time check. +- `framecap` limits frontier entries. +- `pausetarget` supplies the controller target in milliseconds. +- `painbudget` supplies the cost-budget refill percentage. +- `firstpassbudget` supplies the root-enumeration edge budget. + +The parser clamps values to implementation limits. Consult `arguments.h` for current defaults and maximum values. + +## JFR output + +`buildChainEvent()` reconstructs a chain by following `parent_tag` values in the frontier table. Static-field-rooted chains can add the declaring class as the root-side type. + +`cacheResolvedChain()` stores resolved chains per source object, not per class. The cache holds at most 128 entries. When it is full, the code drops a new entry and increases the drop counter. + +`drainPendingChainEvents()` is a snapshot operation despite its name. It copies valid cached chains and keeps them for later dumps. Before emission, it checks the leak-tag incarnation and removes entries whose tag now belongs to another object. + +The dump thread writes `datadog.ReferenceChain` events. The traversal thread does not write JFR events. + +## Result semantics + +A chain can combine observations from different passes. It is not a consistent heap snapshot. + +The implementation does not guarantee: + +- The shortest path. +- The only retaining path. +- A result for every candidate. +- Observation of every reference mutation. +- A hard real-time pause limit. + +Consumers must use the chain as diagnostic evidence and must also inspect abandonment events and drop counters. diff --git a/doc/architecture/ReferenceChainsSignalsExplained.md b/doc/architecture/ReferenceChainsSignalsExplained.md new file mode 100644 index 0000000000..5da376657f --- /dev/null +++ b/doc/architecture/ReferenceChainsSignalsExplained.md @@ -0,0 +1,158 @@ + + +# Reference-chain scheduling and cost control + +## Scope + +This document explains when `ReferenceChainTracker` runs a pass, delays a pass, starts a search, or restarts a search. See [LiveHeapReferenceChainsImplementation.md](LiveHeapReferenceChainsImplementation.md) for the heap-walk operations. + +The source uses the name *pain budget* for a leaky-bucket cost limiter. This document uses *cost budget* except where it names a source field. + +## GC callbacks + +`GarbageCollectionStart()` and `GarbageCollectionFinish()` increment atomic epoch counters. They do not call JVMTI heap functions. + +JVMTI callback restrictions require this separation. It also keeps the callback work small. The background thread reads the epoch and decides whether to run a pass. + +The GC callback does not wake the thread. The normal idle loop checks the state at approximately the current fallback cadence. Shutdown uses a signal to wake a sleeping thread. + +## Starting a search + +Before the first search, `shouldRunPass()` calls `canAffordNewSearch()`. The gate requires a leak indication when GC-generation tracking is active. The indication is one of these conditions: + +- `selectLeakCandidates()` finds a class with sustained population growth. +- The time-to-OOM estimate enters the urgent range and the current urgent episode has not started a search. + +The safepoint cost budget must also permit the search. + +If GC-generation tracking is not active, the leak-signal part of this gate does not apply. + +## Scheduling a running search + +A running search uses the following order: + +1. Check the non-safepoint CPU cost budget. +2. If unresolved candidates exist, apply candidate-search backoff. +3. Otherwise, run after a new GC epoch or after the adaptive fallback cadence expires. + +### Candidate-search backoff + +The source calls the focused candidate search the *canary* path. A new candidate or a resolved candidate resets its spacing multiplier to one. With this value, the next pass can start after the previous pass finishes. + +If a pass makes no candidate progress, the multiplier doubles up to its cap. The required spacing is: + +```text +spacing = multiplier × candidate-pass-duration EMA +``` + +The OOM ramp disables this backoff. The non-safepoint CPU cost budget still applies. + +### GC epoch and fallback cadence + +When no unresolved candidate controls scheduling, a completed GC authorizes a pass. The tracker compares the current GC-finish epoch with the epoch recorded by the previous pass. + +The fallback cadence also authorizes a pass. It starts at one second and the pause controller can adjust it between 10 milliseconds and four seconds. + +## Leak signal + +`LivenessTracker` bases the normal signal on sampled live-object populations by class. + +- It stores 30 GC-epoch values per class. +- It requires at least 10 values before trend evaluation. +- It requires five consecutive qualifying epochs by default. +- It requires three when a rising post-GC heap floor corroborates the signal. + +The aggregate heap-floor signal changes the hysteresis threshold. It does not identify a class. + +Candidate selection also returns qualifying thread identifiers. During an active candidate search, allocations by those threads bypass the normal liveness subsampling ratio. This change increases the chance that the tracker can correlate a leak tag with the walk. It does not change the admission of other threads. + +## Cost budgets + +The tracker has two `PainBudget` instances with different scopes. + +### Safepoint cost budget + +`_safepoint_pain_budget` gates a new search or a restart. Each pass adds its measured in-safepoint time to `_search_pain_ms`. When a search reaches a terminal state, the scheduler charges that accumulated value to the budget before it considers a restart. + +This budget does not stop a running search after a fixed cumulative duration. It controls when a later search can start. + +### CPU cost budget + +`_cpu_pain_budget` gates individual passes in a running search. Each pass charges the measured non-safepoint part of the pass. + +While unresolved candidates exist, the scheduler increases the refill rate by a factor of 100, capped at full wall-clock rate. Candidate backoff remains the main rate control for that path. + +### Meaning of a cost budget + +A cost budget delays work until its leaky bucket drains prior cost. It is not a hard CPU or pause-time quota. The current operation can finish after it exceeds a target. + +## Pause controller + +The tracker measures time spent inside safepointing JVMTI calls. It supplies that value to a PID controller after ordinary expansion passes. + +The controller adjusts: + +- The effective edge-admission budget. +- The fallback cadence when the budget reaches its floor or ceiling. + +Five consecutive passes at no more than half the pause target can start budget borrowing. A later pass that is not comfortably below the target removes the borrowed budget. A root-enumeration pass can also remove it. + +Root-enumeration passes do not update the PID controller because they use a separate edge budget. Their safepoint time still contributes to search cost and can revoke borrowed budget. + +The pause target controls resumable traversal phases. A callback asks JVMTI to stop when +it observes the deadline, but callback granularity and JVM work can cause overshoot. + +Root enumeration does not use the deadline. JVMTI provides no cursor that lets the next +attempt resume after an abort, and the admission budget does not count every examined +root. For this release, a root-enumeration pause is acceptable if it is shorter than +three seconds. [PROF-16284](https://datadoghq.atlassian.net/browse/PROF-16284) +tracks measurement and improvement. Neither the pause target nor this acceptance +threshold is a hard real-time guarantee. + +## OOM response + +`secondsToOOM()` independently projects exhaustion from confirmed rising JVM heap-floor and +container-memory trends. It returns the shorter valid estimate. A negative result means that +neither source provides a valid estimate. + +Two ranges have different purposes: + +- At less than 30 minutes, the OOM ramp increases the pause target, shortens the cadence, increases the edge-budget ceiling, and admits all liveness samples. +- At less than 5 minutes, `isUrgent()` latches an urgent episode. The episode can authorize one search without waiting for a class to complete normal hysteresis. + +The urgent latch clears after five consecutive checks at 10 minutes or more, or after five consecutive checks with no valid rising-trend estimate. This hysteresis prevents repeated search restarts when the estimate moves around the threshold. + +The ramp and the latch use different thresholds. Entering the ramp does not by itself activate the urgent search entitlement. + +## Terminal state and restart + +A search can become `COMPLETED` or `ABANDONED`. Before a restart, the tracker must confirm that it released all search-owned tags. + +The scheduler then: + +1. Charges the completed search's safepoint cost. +2. Clears that search-cost accumulator. +3. Checks the current leak signal. +4. Checks the safepoint cost budget. +5. Calls `restartSearch()` only when both checks pass. + +A running search does not restart itself. + +## Stop request + +`stopThread()` stores an abort request, signals the background thread, and joins it. The signal wakes the thread when it sleeps. + +If the thread is inside a JVMTI heap walk, the signal does not cancel the JVMTI call. `heapReferenceCallback()` checks the abort request on each callback and asks JVMTI to stop traversal. The stop latency therefore depends on callback progress. + +## Operational interpretation + +The scheduler aims to distribute work and react to measured cost. It cannot provide real-time guarantees. + +Use these signals when evaluating behavior: + +- In-safepoint time per pass. +- Non-safepoint pass time. +- Effective edge budget. +- Effective cadence. +- Candidate progress and backoff. +- Search outcome and abandonment reason.