Repository navigation
[python] Support Episode video remux during LeRobot import - #10819
XiaoHongbo-Hope wants to merge 5 commits into
Conversation
JingsongLi
left a comment
JingsongLi
left a comment
There was a problem hiding this comment.
Found one timestamp-boundary issue in the opt-in Episode remux path; details inline.
| mapping = [(timestamp, ordinal) for ordinal, timestamp in enumerate(actual) | ||
| if start - Fraction(1, 10**7) <= timestamp < end - Fraction(1, 10**7)] |
There was a problem hiding this comment.
[P2] Preserve frames accepted by the importer's timestamp tolerance
This 1e-7 boundary filter can exclude a frame that the importer has already accepted with its 1e-4 timestamp tolerance. For a three-frame Episode at 30 FPS with float32 metadata bounds [4.8, 4.9), start becomes 4.800000190734863. Metadata validation and normal import both succeed, but this filter drops the source frame at exactly 4.8, so _EpisodeVideoSources.descriptors() raises Remux cannot map source timestamp 4.8 and aborts the remux import. I reproduced this on both PyAV 14.2.0 and 18.1.0.
Could the mapping use the source timestamps already resolved by the importer, or otherwise preserve its accepted boundary tolerance? Please add a regression test with float32 Episode bounds; the existing roundoff test only covers a much smaller error.
Purpose
LeRobot Episodes can share a long source video. Copying that video into multiple file groups duplicates unrelated frames and increases the cost of opening videos during training.
Changes
Add opt-in
write.lerobot.remux-episodes=trueto copy Episode video packets into separate MP4 payloads without re-encoding. Preserve decoding dependencies, normalize timestamps, and remap frame descriptors to the new payloads. Generated clips include keyframe indexes.The default import path is unchanged. PyAV is required; clips are bounded to 64 MiB and 65,536 packets, and ambiguous timestamps fail the import.
Validation
Packet-copy pixel checks with B-frames, Episode boundary roundoff and size limits; multi-Episode, two-camera import and training readback, including DataLoader workers.