Repository navigation
Conversation
The 64 x 8 MiB default was tuned for Global Store only. On a network volume 8 x 64 MiB at the same 512 MiB in flight hydrates 2-2.8x faster: Qwen2.5-7B ~1420 vs ~730 MiB/s (4090 pod, n=4, ranges disjoint), Qwen2.5-3B ~1000 vs ~360 MiB/s. With the old default VolumeCache was slower than loading in place (3B: 23.1 s vs 21.2 s). On Global Store the two are within run-to-run noise (7B: pod 914 vs 823, 8-GPU H100 host 645 vs 723, Serverless 419 vs 514 MiB/s, all ranges overlapping) with zero ENOMEM/EIO/EAGAIN retries and no incomplete hydrates across 44 runs. The default chunk is now 512 MiB / hydrate_workers, capped at 64 MiB, so callers who set max_workers or hydrate_workers stay within the bytes-in-flight budget instead of reaching the ENOMEM zone (e.g. max_workers=32 gets 16 MiB ranges rather than 2 GiB in flight).
There was a problem hiding this comment.
🟢 Approval recommended
No active review issues remain for the VolumeCache hydration tuning changes.
No open issues 🎉
What changed in this PR
This PR tunes VolumeCache hydration for faster network-volume reads, changing the default from 64 workers × 8 MiB to 8 workers × 64 MiB while retaining 512 MiB in flight.
Changes:
- Derives default chunk size from the worker count, capped at 64 MiB, while preserving explicit chunk sizes.
- Adds tests for the new defaults, worker-dependent scaling, and explicit overrides.
- Updates documentation with the new defaults and Global Store tuning guidance.
| File | Description |
|---|---|
tests/test_serverless/test_utils/test_rp_volume_cache.py |
Tests defaults, chunk scaling, and explicit overrides. |
runpod/serverless/utils/rp_volume_cache.py |
Changes hydrate defaults and calculates worker-dependent chunk sizes. |
docs/serverless/volume_cache.md |
Documents defaults, memory budgeting, and Global Store tuning. |
Review effort: Balanced
briandconnelly
left a comment
briandconnelly
left a comment
There was a problem hiding this comment.
No issues found.
No actionable defects were found in the reviewed scope. Tests were assessed by reading them, not by running them.
What changed
The pull request changes VolumeCache’s hydrate defaults to 8 workers with 64 MiB ranges and scales the default range size to keep configured pools within a 512 MiB budget. It updates documentation and adds tests for worker-dependent sizing and explicit chunk-size overrides.
👾 Reviewed by review-agent (GPT-6.1 Sol) on behalf of briandconnelly
Commit b2d4d38 · 0 inline comments
Summary
Change VolumeCache's default hydrate setting from 64 workers × 8 MiB to 8 workers × 64 MiB, keeping the same 512 MiB in flight. The default chunk is now
512 MiB / hydrate_workers, capped at 64 MiB, so callers who setmax_workersorhydrate_workersstay within the budget.Why
The 64 × 8 MiB default from #621 was tuned for Global Store only. On a network volume it's 2–2.8× slower than 8 × 64 MiB, and it made VolumeCache slower than loading the model in place: in flash-model-cache-bench (Qwen2.5-3B, 4090, EU-RO-1), VolumeCache took 23.1 s against 21.2 s in place. On Flash 1.20 with whole-file copies it took 7.2 s.
Evidence
Hydrate throughput in MiB/s, median [range]. Stock
runpod==2.1.0with explicit settings, page cache evicted before each run, both settings run in alternating order each rep.Qwen2.5-7B (14.2 GiB, 25 files)
Qwen2.5-3B (5.76 GiB, 21 files)
Network volume: the ranges don't overlap. Global Store: the direction flips between arms and every range overlaps, so the difference is within run-to-run noise.
Retries and completeness: across all 44 hydrates in the 7B run, there were zero ENOMEM/EIO/EAGAIN retries and no incomplete hydrates. That covers the 8-GPU host where an ENOMEM storm was previously seen at more than 512 MiB in flight.
This branch's default path: I swapped this branch's code into the pod and ran 2 reps per volume with no explicit settings. The default resolved to 8 × 64 MiB, and the results matched the explicit numbers: network volume 1443 [1422–1463], Global Store 658 [349–967].
Caveat: the first hydrate on a fresh Serverless worker reading Global Store is slow whatever the setting (~150–350 MiB/s, cold host geesefs). Pooled cold first-hydrates hint 8 × 64 may be about 20% slower there (262 vs 332 MiB/s, n=4 each), but that isn't conclusive. The docs suggest trying
hydrate_workers=64on Global Store if that matters.Changes
rp_volume_cache.py:_HYDRATE_WORKERS = 8, plus_HYDRATE_BYTES_IN_FLIGHT = 512 MiBand_HYDRATE_MAX_CHUNK_SIZE = 64 MiB. The default chunk comes from_default_chunk_size(workers), and an explicithydrate_chunk_sizeis never rescaled. Docstring updated.docs/serverless/volume_cache.md: defaults table and the hydrate section.max_workers=32→ 16 MiB, and explicit chunk sizes left unchanged.Test plan
make quality-check: 2755 passed, coverage 94.27%