Skip to content

feat(r1pro): integrate OpenArm and Joy-Con teleop with sim2 - #5212

Draft
Nabla7 wants to merge 730 commits into
mainfrom
pim/feat/r1pro-joylo-sim2
Draft

Nabla7 wants to merge 730 commits into
mainfrom
pim/feat/r1pro-joylo-sim2

Conversation

@Nabla7

@Nabla7 Nabla7 commented Nov 12, 2026 •

Copy link
Copy Markdown
Collaborator

Integrate Feetech OpenArm leaders, Joy-Cons and calibrated WebXR teleoperation with the shared R1 Pro coordinator, hardware/sim2 blueprints, compressed head/wrist cameras and Rust episode recording; preserve CC's #5017/#5019 and Krishna's #5177/#5182/#5198 contributions and use the shared sim2 #5033–#5041 stack, including #5038's Filament RGB-D replacement with the separate R1 color-only adapter removed; latest validation passes 97 camera/physics/calibration/registry checks and 12 input/recording checks plus targeted mypy and commit hooks, while two existing JoyLo timeout tests still expect immediate disable during the configured recovery grace period; full-resolution tote rendering measured 15.7 ms median for four color views and 10.0 ms for three depth views, excluding encoding/transport, on macOS; Linux graphics CI and physical motor-dynamics validation remain pending; setup and provenance: docs/development/r1pro-joylo.md.

Hermes and others added 30 commits November 9, 2026 21:01
…s its fixed edges

tf only ever went out from handle_coordinator_joint_state. The office scene's
manager has no controller -- dimos_graph shows coordinator_joint_state wired to
the three robot managers and nothing else -- so it never published anything, and
the viewer saw three robot trees and no 'world' to hang them off.

A thread now publishes the joints that cannot move at unchanged_tf_hz, through
the same _edges_to_send keepalive. Moving joints are left out: they need a real
joint state, and posing them at zero would be a lie.
Preserve the complete original VR layer, including merge-only resolutions, using its parent-to-head tree delta. Adapt both backends to explicit Twist input mapping and canonical JointState ports. Keep tuned pitch redundancy, Home preset, scoped triggers, Viser and the explicit real entrypoint; no hardware is launched.

Position holds remain position-only. Explicit inhibited_joints claim metadata reserves velocity-only resources so normal connection idle zeroes them, without a special-case hold tick gate or fake position command. Command-only grippers remain excluded by their owning declaration.
…pace

# Conflicts:
#	dimos/robot/urdf_manager.py
…exist before stop can run

Merging the static-tf publisher up: a namespaced manager must put its fixed
edges through frame() like its moving ones, or the office scene would publish
un-prefixed frames. The thread's fields are initialised before _check_mounts, so
the dispose() that a rejected mount triggers does not read them first -- the same
trap as the publisher that stop() touched before __init__ had made it.
A joint with no body0 keeps its offset in the child prim's own transform, not in
LocalPos0 -- _weld_frame writes LocalPos0 and LocalPos1 as zero on purpose. Reading
LocalPos0 for those joints dropped the placement, so the office mounts came out at
the origin and the G1s stood 0.755 m into the floor with their lidar scanning from
the wrong height. _welds_of already read the prim's placement for the same reason.
…ed stack

r1pro-openarm-joylo-real and r1pro-openarm-joylo-sim2 are now the same
composition (r1pro_joylo_inputs.py): operator devices, headset page, JoyLo,
R1ProTeleopCoordinator with its tasks, the recorder and the episode control.
Only the robot connection and the cameras differ. A test pins the
equivalence, and the hardware blueprints are structurally unchanged.

- sim2: SimCompressedCameraModule publishes color as JPEG (CompressedImage)
  and depth raw; SimCameraModule keeps raw color for every other simulation.
- The simulated head and wrist cameras have the sizes, rate, stream names
  and types R1SpatialSensors publishes (480x384 and 1280x720 at 5 Hz). Head
  intrinsics follow a local copy of the factory stereo calibration when one
  exists.
- The simulation records: r1pro_collection() with the same schema, and the
  same required --recorder.recording and --episodes.task.
- The simulation starts with no leader serial port (the leaders are then not
  read); hardware still requires one.
- Removed r1pro-joylo-sim2, r1pro-joylo-sim2-mock,
  r1pro-openarm-joylo-left-sim2, r1pro-openarm-joylo-office and the
  r1pro-openarm-joylo-collect alias. One leader is launch options now.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants