Tutorial 1: Control Robot#

Objective: Learn how to send control commands to a robot in simulation.

What you’ll learn:

  • How to add a robot to your scenario

  • How to set joint position targets using set_dof_targets()

  • How to step the physics simulation with simulate()

  • Action formats (dictionary and tensor)

Prerequisites: Completed Tutorial 0: Static Scene

Estimated time: 20 minutes

Running the Tutorial#

python get_started/1_control_robot.py  --sim <simulator>

you can also render in the headless mode by adding --headless flag. By using this, there will be no window popping up and the rendering will also be faster.

By running the above command, you will give random control actions to the robot and it will automatically record a video.

Examples#

IsaacSim#

python get_started/1_control_robot.py  --sim isaacsim

Isaac Gym#

python get_started/1_control_robot.py  --sim isaacgym

Mujoco#

# For mac users, replace python with mjpython.
python get_started/1_control_robot.py  --sim mujoco --headless

Note that we find the non-headless mode of Mujoco is not stable. So we recommend using the headless mode.

Genesis#

python get_started/1_control_robot.py  --sim genesis

Note that we find the headless mode of Genesis is not stable. So we recommend using the non-headless mode.

Sapien#

python get_started/1_control_robot.py  --sim sapien3

Pybullet#

python get_started/1_control_robot.py  --sim pybullet

Newton#

python get_started/1_control_robot.py  --sim newton

You will get the following videos:

Isaac Sim

Isaac Gym

MuJoCo

Genesis

SAPIEN

PyBullet

Newton

Code Highlights#

Robot Control: Use handler.set_dof_targets(actions) to control robot joints. Actions follow the format:

actions = [
    {
        robot.name: {
            "dof_pos_target": {
                joint_name: (
                    torch.rand(1).item()
                    * (robot.joint_limits[joint_name][1] - robot.joint_limits[joint_name][0])
                    + robot.joint_limits[joint_name][0]
                )
                for joint_name in robot.joint_limits.keys()
            }
        }
    }
    for _ in range(scenario.num_envs)
]
handler.set_dof_targets(actions)
handler.simulate()

Key Points:

  • Can also use direct tensor actions instead of dictionary format ，Action order matches robot.actuators and robot.joint_limits configuration

  • set_dof_targets() only sets targets - call handler.simulate() to step physics

  • Unlike Task’s step(), Handler requires separate simulate() call