Skip to content

feat(cloud): allow directory uploads as tar files - #5329

Merged
TomCC7 merged 6 commits into
mainfrom
cc/feat/lerobot-data-bundle
Nov 21, 2026
Merged

TomCC7 merged 6 commits into
mainfrom
cc/feat/lerobot-data-bundle

Conversation

@TomCC7

@TomCC7 TomCC7 commented Nov 18, 2026 •

Copy link
Copy Markdown
Member

Overview

Pass a directory to dimos data upload PATH to upload it as a tar archive.

Problem

Directory uploads currently require creating an archive by hand.

Changes

  • Package directories, including hidden files, and use the existing upload compression (lz4 by default). Ordinary file uploads are unchanged.
  • dimos data pull returns the .tar without unpacking it. Extract it manually with tar -xf ARCHIVE.tar.

Validation

96 local tests passed, 4 skipped at 030ee8ab3f. Review report.

@github-actions github-actions Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

go2 replay realtime (arm64)

Details
Benchmark suite Current: 030ee8a Previous: 1e8989d Ratio
peak memory 1738.43 MB 1755.098 MB 0.99
peak threads 365 threads 365 threads 1
network (transport) 2724.122 MB 2728.417 MB 1.00
disk write 1.66 MB 1.602 MB 1.04
instructions 138.226 G 137.431 G 1.01

This comment was automatically generated by workflow using github-action-benchmark.

@TomCC7 TomCC7 changed the title feat(cloud): transfer LeRobot dataset directories feat(cloud): transfer directories as plain tar Nov 18, 2026
@codecov

codecov Bot commented Nov 18, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 94.11765% with 6 lines in your changes missing coverage. Please review.
✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
dimos/cloud/data.py 85.71% 3 Missing and 1 partial ⚠️
dimos/cloud/test_data.py 97.29% 2 Missing ⚠️
@@           Coverage Diff           @@
##             main    #5329   +/-   ##
=======================================
  Coverage        ?   83.31%           
=======================================
  Files           ?     1963           
  Lines           ?   198265           
  Branches        ?    16569           
=======================================
  Hits            ?   165189           
  Misses          ?    28995           
  Partials        ?     4081           
Components Coverage Δ
Tests 94.01% <97.29%> (?)
Flag Coverage Δ
OS-macos-latest 79.34% <94.11%> (?)
OS-ubuntu-26.04-arm 78.93% <94.11%> (?)
OS-ubuntu-latest 79.38% <94.11%> (?)
Py-3.10 79.38% <94.11%> (?)
Py-3.11 79.36% <94.11%> (?)
Py-3.12 79.37% <94.11%> (?)
SelfHosted-Linux 40.53% <10.78%> (?)
SelfHosted-macOS 40.27% <10.78%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
dimos/cloud/test_data.py 99.55% <97.29%> (ø)
dimos/cloud/data.py 92.30% <85.71%> (ø)
🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@TomCC7

TomCC7 commented Nov 18, 2026 •

Copy link
Copy Markdown
Member Author

Review report

Comparison base (merge base): f9f7f4dd878e2639237fd2b6a35842d42f46f3a3 · Head: 030ee8ab3f7fee739b31210d656fd57298b1a0d4 · Updated: 2026-11-21

Review map (vs. main)

  • Modified dimos/cloud/data.py (+31/−12): Stage the tar outside the source tree, following only the selected root link. Write the tar directly; the staging context removes partial output on write failure. The existing upload codec and its staging check are unchanged.
  • Modified dimos/cloud/test_data.py (+87/−3): Check staging placement, symlinks, hidden contents, retry deduplication, partial-tar cleanup, and the exact .tar.lz4 name and wire format.
  • Modified docs/usage/cloud_data.md (+3/−1): Document directory uploads and the staging fallback. Pull still decodes the upload codec without extracting the tar.

Validation

  • At the head above, from the repository root with the local Python 3.12 project environment: python -m pytest --import-mode=importlib -o addopts='' -q dimos/cloud/test_data.py -k 'directory_upload or upload_guards or roundtrip_with_resume_and_manifest' — 9 passed.
  • python -m pytest --import-mode=importlib -o addopts='' -q -rs dimos/cloud dimos/cli/test_cloud.py experimental/gateway/tests/test_cloud.py — 96 passed, 4 skipped (mcap-ros2-support not installed). Ruff, commit hooks, and git diff --check passed.
  • Against the previous head dba36764e341abf819d6fd5091c163037d62c731, the new failure-path test hit the tar-size preflight instead of attempting the write; the compression round trips and ordinary-file control passed. Validation used local fixtures and fake cloud transport; no CI inspection or production uploads.

@TomCC7 TomCC7 changed the title feat(cloud): transfer directories as plain tar feat(cloud): allow directory uploads as tar files Nov 18, 2026
@TomCC7
TomCC7 marked this pull request as ready for review November 18, 2026 07:08
@greptile-apps

greptile-apps Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerView in GreptileConfidence Score: 3/5

[Medium impact] The change reaches the cloud data upload workflow for people who explicitly choose a directory.

Fix the missing directory contents and staging location before merging.

Findings

  1. P1 Directory links lose their contents ▶
  2. P1 Archives ignore the staging setting ▶
  3. P2 Temporary folders break upload retries ▶

Summary

Directory uploads now create a tar archive and send it through the existing file-upload flow.

  • Cloud uploads now package directories as named tar files.

Reviews (1) · Last reviewed commit: "refactor(cloud): make directory support ..." · Reviewed by Greptile

Comment thread dimos/cloud/data.py Resolved
Comment thread dimos/cloud/data.py Resolved
Comment thread dimos/cloud/data.py Resolved
@github-actions github-actions Bot added the ready-to-merge Required CI checks have passed on this PR label Nov 18, 2026
Comment thread dimos/cloud/data.py Resolved
Comment thread dimos/cloud/data.py Resolved
Comment thread dimos/cloud/test_data.py Resolved
@github-actions github-actions Bot removed the ready-to-merge Required CI checks have passed on this PR label Nov 21, 2026
@TomCC7
TomCC7 enabled auto-merge (squash) November 21, 2026 20:52
@TomCC7 TomCC7 added the backport:skip Skip creating a backport to any release branches label Nov 21, 2026
@TomCC7
TomCC7 merged commit 13debff into main Nov 21, 2026
46 of 47 checks passed
@TomCC7
TomCC7 deleted the cc/feat/lerobot-data-bundle branch November 21, 2026 21:09
@github-actions github-actions Bot added the ready-to-merge Required CI checks have passed on this PR label Nov 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport:skip Skip creating a backport to any release branches ready-to-merge Required CI checks have passed on this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants