Skip to content

fix(net): a pool held over capacity by its live edges still serves them - #9422

Draft
kixelated wants to merge 6 commits into
mainfrom
quest/m1/pool-pinned-debt
Draft

kixelated wants to merge 6 commits into
mainfrom
quest/m1/pool-pinned-debt

Conversation

@kixelated

@kixelated kixelated commented Nov 22, 2026 •

Copy link
Copy Markdown
Collaborator

Finishes quest/m1/pool-pinned-debt.md.

Problem

Sometimes a relay's cache target sits below what it can never evict: each live track's newest group. That happens with cache.headroom = "100%", or on any host whose available memory is below the headroom. The pool then stays over capacity for good, and two things go wrong:

  • Readers end. Readers' debt was forgiven only once used <= capacity, which never happens again. So every reader Record::settle picks ends with Lagged (TooFarBehind on the wire). On main, moq export fmp4 through such a relay loses its opus rendition this way every time.
  • The live edge is shed. Pool debt shed the front of a track's live edge even after the group finished. A new reader of a finished latest group, such as a catalog, then reads Evicted.

Approach

  • The pool pins what no eviction can reclaim. A new pinned counter covers each live edge's fixed costs (entry overhead and codec state), its content once the group finished, and each track's spare encoder. It is fed by the Footprint that already bills sessions' live_edges. A second FINAL flag bit keeps each count exact across races, as the EDGE bit already does.
  • Usage converges on max(capacity, pinned). The pool's excess over that target drives everything: debt accrual, forgiveness, held-head reclaim, and hold. A pool held past its capacity by pins alone owes nothing. A track's debt is capped at the unpinned bytes, which is all eviction could ever free.
  • Readers fold under any debt, but end only for usable room. Reader::ends now also needs Pool::room() > 0, meaning capacity above the pins. Ending a reader in a pool held past its capacity by pins frees room nothing may use.
  • Pool debt never sheds a finished live edge. A finished edge no longer grows, and a new reader starts from it. A live edge still arriving keeps shedding its front, as feat(net)!: shed group fronts past a 32 MB tail or under pool debt #6993 designed, so late joiners still ride flights. The session live_edges budget still sheds both kinds.

Decisions

The user was away, so I took these choices myself. Each is open to review:

  • Only finished live edges keep their content. A growing live edge still sheds its front under pool debt. Pinning every live edge would drop feat(net)!: shed group fronts past a 32 MB tail or under pool debt #6993's memory bound for long open groups, leaving the 32 MiB tail and the per-session live_edges budget as the only bounds. Peers have no live_edges bound by default.
  • The target is max(capacity, pinned), not capacity + pinned. Above the pins, capacity still counts live edges, so the headroom governor's math (used + available - headroom) is unchanged.
  • Records still fold under any debt. Folding costs a reader nothing and keeps records bounded in a pool held past its capacity by pins. Only the end needs room.
  • A track's debt is capped at the unpinned bytes, not the excess. Capping at the excess stopped a group larger than the excess from ever being evicted, which frame_only_writer_pays caught.
  • Spare encoders count as pinned. Only the track's end frees one.

Impact

  • Public API: none. Pool::pinned, excess, and room are crate-private.
  • Wire: none.
  • Behavior: a finished live edge is no longer shed by pool debt. In a pool whose capacity sits at or below its pins, no reader ends for debt. Everything else stays as on main while the capacity is above the pins.

Validation

  • Real surface. I ran moq-relay with [cache] headroom = "100%", moq import ts (lite-07, bbb5s_cbr.ts at real time), and moq export fmp4 for 10 s. The scripts are in the worktree's .tmp/repro.
    • main, 2 runs: the relay logs subscribed error track=1.opus err=lagged both times, the export logs rendition failed ... 1.opus ... too far behind, and it writes 66–67 KB.
    • This branch, 3 runs: no lagged and no fetch started. The export writes 73–95 KB. The opus rendition skips 6–11 groups of 100–400 ms that were evicted just after they left the live edge. That is the history loss the quest accepts under a zero target.
    • The same branch with headroom = "0": 104 KB, no skips.
  • Model test. a_pool_below_its_live_edges_still_serves_them fails on main with Evicted reading the catalog's latest frame, and passes here.
  • Unit tests. a_live_edge_pins_what_no_eviction_reclaims checks that the pins follow every add, shed, finish, and demote, and return to zero. a_debt_past_the_pins_ends_no_reader fails when the room() > 0 condition is removed, and shows the same debt ends the reader once there is room.
  • Checks. just check passes (11186 tests).

Caused by

  • moq-dev/moq#8476: added readers' debt, forgiven only once used <= capacity, which pinned live edges can make unreachable (found by person).
  • moq-dev/moq#6993: pool debt sheds the live edge's front even once the group finished, leaving a new reader nothing to start from (found by person).

The headroom governor becoming the default (moq-dev/moq#9269) exposed both on hosts with little free memory.

Alternatives

  • The governor clamping its target to the pinned bytes. The quest rejected it: it fixes the governor's case but not a static capacity below the same sum.
  • Pinning every live edge and dropping pool shedding of the live edge. Rejected for the memory bound described under Decisions.
  • Capping readers' debt at the room instead of gating the end on it. That forgave folding too, so a fragmented record grew unbounded in a pinned pool (a_reported_flood_folds_or_ends_the_reader caught it).

Follow-ups

None. The fetch started walk the quest observed belongs to lite06-fill-walk.

Models

Size: M
Implementer: Claude, claude-opus-5-5, unknown effort (arm B)

🤖 Generated with Claude Code

Review

Rewritten by quest review after each round.

  • Head: 2d97f2c, named in full once no reviewer reports a finding

  • Base: origin/main at 040a566

  • Risk: medium, It is a bug fix in shipped moq-net cache eviction: a relay pool held over capacity by pinned live edges now serves new readers their latest group and stops ending readers for unusable room, which could change what relays deliver and which readers end, but the public API, wire format, and auth are untouched and a revert undoes it all, so it is not high; the RISK.md doc and quest-plan deletion are low.

  • Classified: medium by Claude Code, claude-haiku-5-5, low effort, session 7886e894-6af0-4f9a-9918-6b21b064bcce

  • Reviewer: Codex, gpt-6.1-sol, high effort

  • Session: 01a20410-1d2e-7772-ae02-5d045dd0d2f1, round 1

  • Score: 4/10, The cache-pressure handling needs rework because reader records can grow without bound and shared payload retained by finished copies is undercounted as pinned.

  • Rounds: 1; first round 2 (high 1, medium 1); findings 2 (high 1, medium 1): fixed 0, refuted 0, open 2

  • Findings, open until the next round settles them:

    • High, rs/moq-net/src/model/cache.rs:1930: Preserve a memory bound for records when pins exhaust capacity
    • Medium, rs/moq-net/src/model/cache.rs:1476: Pin shared payload retained by a finished sibling

kixelated and others added 6 commits November 22, 2026 06:49
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…oom it can use

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… finish the quest

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated kixelated changed the title A pool held over capacity by its live edges still serves them fix(net): a pool held over capacity by its live edges still serves them Nov 22, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant