Manipulation Sample Applications

Simple Joint Control with Jupyter Notebook

This sample provides interactive joint control using Jupyter Notebook. It’s a great starting point for working with the CompositeProto message used for manipulation components, including the LQR planner.

In the Isaac SDK repository, deploy the simple_joint_control package:

bob@desktop:~/isaac$ ./engine/build/ -p //apps/samples/manipulation:simple_joint_control-pkg -h localhost -d x86_64

Navigate to the deploy folder $HOME/deploy/<username>/simple_joint_control-pkg and open the Jupyter Notebook:

bob@desktop:~/deploy/bob$ jupyter notebook apps/samples/manipulation/simple_joint_control.ipynb

Your web browser should open the Jupyter notebook document. If it does not, search for a link on the console: It will look like http://localhost:8888/notebooks/simple_joint_control.ipynb. Open that link in your browser.

This sample has two parts. The first part, UR10 in Omniverse Isaac Sim, controls a simulated UR10 arm in Isaac Sim in Omniverse. The second part, Kinova Jaco Hardware, controls the Kinova generation 2 robotic arm hardware.

  • UR10 in Omniverse Isaac Sim: Follow the IsaacSim Omniverse documentation to start the simulator and open the stage at omni:/Isaac/Samples/Isaac_SDK/Scenario/ur10_basic.usd. Start the simulation and Robot Engine Bridge.

    In the Jupyter notebook, follow the cells to start the SDK application. Once it is connected to the simulator, you can use the sliders to move individual joints of the UR10 arm in simulation. The joint commands to reach the target joint angles are computed by the multi_joint_lqr_control subgraph, and the motion is limited by the maximum speed, minimum speed, and acceleration values set in the configuration of the MultiJointLqrPlanner component.

Jupyter notebook joint control widget (right) and UR10 in simulator (left)


The ur10 model in simulation by default has self collision disabled. To enable this, select ur10 in Stage, in the PhysX Properties window, check enableSelfCollisions. Note the LQR planner do not avoid self collision.

  • Kinova Jaco Hardware: Follow the instructions in the notebook to install the Kinova API and connect the arm to the workstation. Follow the cells to start the SDK application, and use the sliders to move individual joints on the Kinova arm.

Shuffle Box with Simulator

This sample shows how to make the robotic arm move through a set of predefined waypoints in joint angles and control the end effector (in this case, a gripper). The CompositeMetric and CompositeAtlas documentation explains in detail how to make the arm follow a predefined path.

Follow the IsaacSim Omniverse documentation to start the simulator and open the stage at omni:/Isaac/Samples/Isaac_SDK/Scenario/sortbot_sim.usd. Start the simulation and Robot Engine Bridge.

In the Isaac SDK repository, run the following:

bob@desktop:~/isaac$ bazel run apps/samples/manipulation:shuffle_box

The UR10 arm should repetitively pick up a pink box on one dolly and drop it off on the other side. Since the pick-up and drop-off waypoints are hard-coded joint angles, while the box position will shift slightly over time, after a number of iterations the arm will not be able to pick up the box due to misalignment.


The sample app also includes a pre-trained 3d pose estimation model for the pink box. The perception output can be visualized in Sight:

  1. Change the simulation camera resolution: click the Settings icon on the top left of the view port and change the ‘Render Resolution’ from ‘Viewport’ to ‘1280x720’.

  2. Switch the camera in simulation next to the Settings icon from Perspective to Camera / Camera . Open Sight at http://localhost:3000. You should see the CAD model of the box with the detected 3D pose overlayed on the RGB camera image.