[XiaoHongbo-Hope]

Purpose

Reduce video-index I/O with a page directory and optional compression while keeping indexed reads for the format released in Paimon 2.1.0.

• Layout: Newly generated keyframe indexes use v2, with a page directory and raw or column-delta zlib pages. The released v1 layout remains the whole-index zlib stream; its 65,536-keyframe limit applies to one indexed source video, not the table. Video payloads, .video headers, and frame descriptors are unchanged.
• Reads: Readers dispatch on each index's version byte. Both released v1 and v2 use indexed seeking. Small v2 indexes are cached; large v2 indexes load only the relevant pages. Unknown versions or codecs retain scan fallback.
• Writes: New indexes use v2. Copying or compacting existing videos preserves each index's bytes and version, so v1 and v2 can coexist in one .video file. video.index.compression controls only newly generated v2 indexes (delta-zlib by default, or none).

The intermediate paged layout that also used version 1 was never released and is not a supported format.

Measurements

Three-run medians. Local index-only test: 1M keyframes; writes include fsync, reads include open + 64 random lookups.

Input	Encoding	Size	Write	Read	Bytes read
Regular	Raw	24.02 MB	104.5 ms	11.8 ms	6.21 MB
Regular	Delta-zlib	3.30 MB	214.4 ms	29.0 ms	0.88 MB
Irregular	Raw	24.02 MB	103.1 ms	11.9 ms	6.21 MB
Irregular	Delta-zlib	5.84 MB	293.2 ms	33.8 ms	1.52 MB

Compression saves I/O at a CPU cost. In three local DROID DataLoader runs (Python/PyAV, batch 32, 4 workers, 10 warmup + 64 measured batches), median throughput was 97.75 → 97.24 samples/s and CPU 5.76 → 5.75 cores; indexes shrank 90%. These indexes fit in cache; this does not measure large-index cache churn.

Separately, the directory-only OSS test reduced 64-lookup latency from 10.21 s to 4.02 s and GETs from 163 to 65.

Validation

Focused checks on the updated head: Python video-index tests 25 passed, Python video-format tests 25 passed, and Java VideoFileFormatTest 21 passed. The normal Maven test run (including style checks) and git diff --check passed.

Coverage includes an original Paimon 2.1.0 v1 index and .video fixture, indexed v1/v2 seeking, mixed versions in one .video file, byte-for-byte preservation on rewrite, large v2 indexes, unknown-version fallback, and corruption rejection.

[sundapeng]

Thanks for the PR! The page directory looks like the right direction to me, and from my local tests most of the improvement comes from it. But I have some concerns about the format design, mainly the versioning and the compression part.

1. The v1 layout changes again without bumping the version

This is the third v1 layout within one month (#9831: <BQII + whole-index zlib, #10891: <BQIQ + fixed pages, and now <BQIQB + codec + directory), all with version byte 1. I tried to read an index written by current master (00387fb) with this PR, and also the reverse direction. Both fail with Video metadata range is outside the video payload, because every field after the header is shifted by one byte. The error message is quite misleading.

On the read path VideoKeyframeIndex.open just throws. On the copy path both Python (video_format_writer.py:214) and Java (VideoFormatWriter.java:336) validate the copied index, so compaction / rewrite will fail as well.

I know none of these commits is released yet, but people building from master may already have such data. Since we already have a version byte, I think we should just use version 2 here (it costs nothing), and let readers report "unsupported keyframe index version" instead of misparsing. Maybe also reserve a flags or header-length field, so next time we don't need to break the layout again.

2. Most of the gain comes from the directory, not compression

I counted GETs with an instrumented read_at (synthetic keyframes: constant frame rate, GOP 2, random packet sizes; 512 random preceding + keyframes[...] lookups):

	master	PR none	PR zlib
65,536 keyframes, open	2 GET / 197 KB	1 GET / 1.57 MB	1 GET / 0.52 MB
65,536 keyframes, 512 lookups	346 GET / 76.9 MB	0	0
1M keyframes, 512 lookups	3,592 GET / 961 MB	487 GET / 47.7 MB	485 GET / 15.5 MB

Compression reduces the bytes by ~3x, but the GET count is the same. On OSS the latency is mostly decided by the number of requests; with a simple model (15 ms per GET + 100 MB/s, my assumption, not measured) it's about 7.8 s vs 7.4 s. For storage, the index is 24 bytes per keyframe while one keyframe usually covers tens of KB of video, so the saving is very small.

Also, once a codec is in the spec it's hard to change, and I'm not sure zlib over row-major int64 triples is a good choice. On the same data (GOP 2):

• zlib level 1, row-major (this PR): 3.1x
• DeltaVarintCompressor per column (we already have it in Java and Python for the .video footer): 4.0x
• per-column delta + zlib level 1: 7.8x

So my suggestion is to split this PR: land the directory with codec 0 first, and discuss compression separately. If we keep compression, maybe define the codec as per-column delta encoding. It would also be great if the benchmark could show the directory and compression separately.

3. Unsupported version / codec cannot be distinguished from corruption

If we add codec 2 later, old readers will fail with Invalid video keyframe index header or codec, on both read and compaction. But the index is only an accelerator, and we already have the scan path when there is no index. How about:

• unsupported version / codec: treat it as no index, log a warning, and copy it as is during compaction;
• checksum or monotonicity failure: still fail.

4. Directory caches can be simpler (minor)

Python keeps three caches: directory blocks (512 records, LRU of 2), descriptors (LRU of 512) and pages. The directory for 1M keyframes is only 16.6 KB and is usually already in the 64 KB prefix. If we read the whole directory at open, the binary search needs no I/O and the first two caches can be removed.

I also left a few smaller comments inline (page size wording in the spec, the eager-loading threshold, and the temp file in the Java validator).

WDYT?

[sundapeng]

This says "up to 4,096", but both readers assume every page except the last has exactly 4,096 entries: they compute the entry count as min(4096, count - p * 4096) and locate the page by divmod(index, 4096). So a writer which follows the spec and writes variable-size pages (e.g. a future Rust writer) will be rejected.

Maybe say "exactly 4,096 except the last page" here, or store page_entries in the header. The directory already has the first ordinal of each page, so variable-size pages are not hard to support.

[sundapeng]

Here the whole index is read when the encoded length <= 2 MiB, but load_small() only goes eager when the decoded size count * 24 <= 2 MiB. For compressed indexes in between, the whole encoded index stays in prefix, and pages are still decoded into the 1 MiB page cache. With 200K keyframes I see 1.58 MB held + 0.97 MB page cache, and it's not eager.

So the memory per open index goes from ~1.1 MB to ~2.5 MB, multiplied by max_open_videos (16 by default) and the number of DataLoader workers. Maybe use one threshold on the decoded size, or document this case.

[sundapeng]

This creates a temp file for every index only to read the directory twice. The caller's spool in VideoFormatWriter is already a RandomAccessFile, so maybe we can pass a seekable input here, or just keep the directory in memory. It is bounded by the index length, and only about 16.6 KB for 1M keyframes.

[JingsongLi]

Requirement fit: SUPPORTED. The page directory is connected to actual indexed seeking and provides a credible reduction in remote index I/O. I found no introduced correctness defect in the codec, directory bounds/CRC checks, Java/Python format handling, or mixed-version byte-preserving rewrites.

There is a production rollout condition that should be explicit in the video documentation: upgrade every video reader and rewrite/compaction worker before producing v2 indexes. I tested the exact release-2.1.0 implementation against bytes produced by this head: the new reader reads released v1 successfully, but the released reader rejects both compression=none and delta-zlib output with Invalid video keyframe index version or magic. Both options write version 2; none is not a legacy-format switch. The old collator propagates that error, and released Java rewrite validation also rejects v2. Once v2 is written, rolling back those workers to 2.1.0 is not safe without avoiding v2 data or providing a legacy/no-index output path. The intermediate unreleased paged-v1 baseline was already incompatible, so I am identifying a release/rollback condition, not attributing that entire incompatibility to this PR.

The current Native CI job also needs a green result or a demonstrated same-baseline/dependency explanation before treating the head as production-ready: https://github.com/apache/paimon/actions/runs/41286128293/job/124788039362. It reports 335 failures across vector/index/search suites with Unknown vindex option(s) errors. Those paths are outside this video diff, so I have not classified them as a video regression.

Local verification on this exact head: 21 Java format tests with normal JDK 8 Maven verify; 2 real Core/Flink integration tests for video rolling and Data Evolution BLOB updates with normal flink1 Maven packaging; and 68 Python video/index/format/collator/table cases, all passing with no skips after installing PyAV/Torch. The Python run includes a real 65,537-keyframe MP4 with sparse pixel-checked reads, both compression options through table writes, mixed-version rewrites and fallback/corruption cases. Configured Flake8 on all changed Python files and git diff --check pass. I did not rerun the OSS or DROID benchmarks; the supplied DROID figures do not establish a training-throughput improvement.