Shared capture infrastructure for teleoperation and egocentric data.
Isaac Capture provides shared infrastructure for collecting human and robot demonstration data. It brings together two components:
- Isaac Teleop supports robot-in-the-loop capture through teleoperation, including Teleop Session and Remote Assist.
- Ego Pipeline supports human and egocentric data capture through Ego Capture and Ego Post Processing.
Both components use a shared core of device plugins, retargeters, and data utilities for recording, replaying, and converting data. Isaac Teleop remains a component of Isaac Capture, with its existing functionality preserved.
- Unified stack for sim & real teleoperation
- Standardized device interface
- Flexible retargeting framework
- Currently supported use cases
- Use XR headsets for gripper / tri-finger hand manipulation
- Use XR headsets with gloves for dex-hand manipulation
- Seated full body loco-manipulation (Homie)
- Tracking based full body loco-manipulation with GR00T-WholeBodyControl (SONIC)
- Upcoming use cases
- Teleoperate using only non-XR devices (e.g. gamepad, Gello, haply, etc.)
- Teleoperate cloud based robotics simulations
- Remote teleoperation with camera streaming to the desktop
- Remote teleoperation with immersive camera streaming to XR headsets
- Capture egocentric data without a robot in the loop.
- Reconstruct hand and camera poses from egocentric video with the Ego Post Processing pipeline.
Our documentation page provides everything you need to get started, including detailed tutorials and step-by-step guides. Follow these links to learn more:
The latest NVIDIA Isaac Lab release, 3.0.0 EA, includes Isaac Capture, previously released as Isaac Teleop.
To get started, please refer to Isaac Lab's Installation guide for more details. Then follow the CloudXR teleoperation in Isaac Lab to get started with Teleoperation in Sim.
The default license is Apache-2.0. EgoPost is licensed under OpenMDW-1.1. Third-party materials may carry their own licenses.