Skip to content

feat(manipulation): add experimental cuRobo V2 planning backend - #5365

Draft
TomCC7 wants to merge 3 commits into
mainfrom
cc/feat/curobo-v2-t7d6e128d
Draft

TomCC7 wants to merge 3 commits into
mainfrom
cc/feat/curobo-v2-t7d6e128d

Conversation

@TomCC7

@TomCC7 TomCC7 commented Nov 19, 2026 •

Copy link
Copy Markdown
Member

Why do we need this?

Evaluate scene-aware trajectory optimization as an opt-in alternative to RoboPlan. This draft is for offline/simulation review only: soft scene collision is not a physical safety barrier.

What is this feature?

Select world_backend=curobo and planner.backend=curobo together. The fixed-base xArm7 grasp URDF is used with a modeled open gripper, fitted link spheres and shared adjacency/config/SRDF exclusions. The shared xArm model config supplies the rigid link7 / xarm_gripper_base_link pair directly, without a dedicated SRDF. Existing exclusions and SRDF support for other models are unchanged. Pink still solves pose goals. V2 plan_cspace uses the world snapshot as a scene cost, not as a TrajOpt acceptance constraint; self-collision, limits and native success remain required. Native trajectory times and velocities go through existing materialization/execution. RoboPlan remains default. No changes to mapper, hardware or executor.

How to manually run this

From the checkout on Linux x86-64 with a supported NVIDIA GPU and driver:

uv sync --locked --python 3.12 --no-default-groups --extra manipulation --extra curobo

This pulls nvidia-curobo[cu12-torch] v0.9.0 from its pinned Git source with CUDA 12.8 PyTorch. See the manipulation guide for prerequisites. Select world_backend=curobo and planner.backend=curobo in the simulated xArm grasp blueprint; inspect planned results without execution. Cartesian segments and the full grasp workflow are not supported by this draft.

Verification

  • At 744dd854d: fresh locked install on Linux x86-64 with Python 3.12 and RTX 3090 (driver 590.48.01), using PyTorch 2.7.1+cu128. The standalone curobo extra also installed and imported successfully on Python 3.11 with CUDA available. Separate base and planning installs without development groups contained no cuRobo, Torch, or CUDA libraries; this does not change existing PyTorch selections in other extras or test groups.
  • At 744dd854d: 139 passed, 3 deselected in the focused CPU suite, including the real cuRobo scene/config parser, xArm blueprint, RoboPlan and SRDF/attachment regressions, and docs branding. The two xArm7 grasp compositions retained byte-identical prepared URDFs, all 15 collision links, and all 28 effective exclusions each versus 4ed8ad0dd.
  • At 744dd854d on RTX 3090: the native xArm7 home self-check passed. Empty-scene planning produced 55 timed samples; the same-endpoint detour improved sampled clearance from −24.7 mm to +9.8 mm (not a continuous safety guarantee). Unavoidable scene overlap remained accepted; non-exempt self-collision and an out-of-limit goal were rejected. Pink → native planning → timed materialization preserved 37 samples. No executor or hardware driver was started.
  • Prior evidence retained at 4ed8ad0dd: 387 passed, 31 deselected in the broader CPU/default-backend suite. The independent RTX 3090 qualification at that head also passed home, free-space planning, detour, soft overlap, self-collision and limits rejection, and Pink-to-timed materialization. That run used the earlier SRDF-backed mount declaration.
  • Related PRs refactor(perception): remove WorldBelief and the planner's object obstacles #5206 and feat(manipulation): add_surface, the work surface as one flat planner box #5355 touch adjacent manipulation/scene composition, but do not add this backend. Full grasp execution, robot hardware, and CI were not tested for this revision.

Which issue(s) does this PR close?

None.

Checklist

  • CLA — not attested on a human's behalf.

@github-actions github-actions Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

go2 replay realtime (arm64)

Details
Benchmark suite Current: 744dd85 Previous: 243fecd Ratio
peak memory 1761.289 MB 1784.68 MB 0.99
peak threads 384 threads 377 threads 1.02
network (transport) 2730.56 MB 2728.316 MB 1.00
disk write 1.922 MB 1.941 MB 0.99
instructions 138.955 G 138.933 G 1.00

This comment was automatically generated by workflow using github-action-benchmark.

@codecov

codecov Bot commented Nov 19, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 85.07209% with 176 lines in your changes missing coverage. Please review.
✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
dimos/manipulation/planning/world/curobo_world.py 73.87% 73 Missing and 20 partials ⚠️
dimos/manipulation/planning/world/curobo_model.py 71.20% 18 Missing and 18 partials ⚠️
...s/manipulation/planning/world/test_curobo_world.py 91.60% 22 Missing ⚠️
...s/manipulation/planning/planners/curobo_planner.py 81.08% 11 Missing and 10 partials ⚠️
dimos/manipulation/manipulation_module.py 66.66% 1 Missing and 1 partial ⚠️
dimos/manipulation/planning/factory.py 92.00% 0 Missing and 2 partials ⚠️
@@            Coverage Diff             @@
##             main    #5365      +/-   ##
==========================================
+ Coverage   84.76%   84.80%   +0.03%     
==========================================
  Files        1951     1958       +7     
  Lines      192276   193428    +1152     
  Branches    15683    15794     +111     
==========================================
+ Hits       162990   164039    +1049     
- Misses      25321    25433     +112     
+ Partials     3965     3956       -9     
Components Coverage Δ
Tests 96.76% <95.84%> (-0.01%) ⬇️
Flag Coverage Δ
OS-macos-latest 79.20% <85.07%> (+0.03%) ⬆️
OS-ubuntu-24.04 79.25% <85.07%> (+0.03%) ⬆️
OS-ubuntu-24.04-arm 78.79% <83.54%> (+0.03%) ⬆️
Py-3.10 79.25% <85.07%> (+0.04%) ⬆️
Py-3.11 79.22% <85.07%> (+0.02%) ⬆️
Py-3.12 79.23% <85.07%> (+0.02%) ⬆️
SelfHosted-macOS 40.15% <22.30%> (-0.11%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
dimos/manipulation/planning/planners/config.py 100.00% <100.00%> (ø)
...ipulation/planning/planners/test_curobo_planner.py 100.00% <100.00%> (ø)
...anipulation/planning/world/collision_exclusions.py 100.00% <100.00%> (ø)
...imos/manipulation/planning/world/roboplan_model.py 93.12% <100.00%> (+7.40%) ⬆️
...s/manipulation/planning/world/test_curobo_model.py 100.00% <100.00%> (ø)
dimos/manipulation/test_planning_factory.py 100.00% <100.00%> (ø)
...s/robot/manipulators/xarm/blueprints/test_grasp.py 100.00% <100.00%> (ø)
dimos/robot/manipulators/xarm/config.py 94.59% <100.00%> (+0.09%) ⬆️
dimos/manipulation/manipulation_module.py 82.58% <66.66%> (+4.48%) ⬆️
dimos/manipulation/planning/factory.py 87.50% <92.00%> (+9.50%) ⬆️
... and 4 more

... and 3 files with indirect coverage changes

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant