Sim Evaluation#

In this lesson, we’ll run the fine-tuned GR00T 1.7 policy in closed loop and evaluate it in the Isaac Lab-Arena Unitree G1 static apple-to-plate task.

To do that, we’ll:

Start the GR00T policy server from the standalone Isaac-GR00T checkout.
Run Isaac Lab-Arena as a client that queries the server over ZeroMQ.
Evaluate the policy in a single environment and optional parallel environments.
Interpret success-rate metrics and common server-client failure modes.

This lesson assumes you completed GR00T Fine-Tuning on Sim Data.

Start the GR00T Policy Server#

The server runs GR00T’s stack run_gr00t_server.py from the standalone Isaac-GR00T 1.7 checkout. We will start it before launching the client. Run the server outside Docker in the standalone Isaac-GR00T virtual environment created in the simulation overview.

These steps assume you completed GR00T Fine-Tuning on Sim Data and still have the standalone Isaac-GR00T environment available.

Continue in the standalone Isaac-GR00T terminal used for fine-tuning.

If you opened a new terminal, go back to the Isaac-GR00T directory:
```
cd $ISAAC_GR00T_DIR
```

Important

From previous steps, you should already be on the Isaac-GR00T commit from Setup: Isaac Lab-Arena.

But if you changed branches or pulled new changes during fine-tuning, restore the validated commit before starting evaluation:

cd $ISAAC_GR00T_DIR
git checkout 4b1dca9d88d2a0b9ea5a65aa61c82ff89f5c4f0e

Start the GR00T policy server.

Replace ~/IsaacLab-Arena with the path to your Arena repo if cloned somewhere else. Replace ${MODEL_PATH} with the fine-tuned checkpoint directory from training.

uv run python gr00t/eval/run_gr00t_server.py \
   --modality-config-path ~/IsaacLab-Arena/isaaclab_arena_gr00t/embodiments/g1/g1_sim_wbc_data_gr00t_n_1_7_config.py \
   --model-path ${MODEL_PATH} \
   --embodiment-tag NEW_EMBODIMENT \
   --device cuda \
   --host 0.0.0.0 \
   --port 5555

Once running, the server prints Server Ready and listening on 0.0.0.0:5555 to the terminal. This means it is ready for clients.

Run Single Environment Evaluation#

With the server running, launch the Arena client. The client side does not need GR00T dependencies, because it communicates to the server over ZeroMQ using the standard Base Arena container. Gr00tRemoteClosedloopPolicy is Arena’s client wrapper around the remote GR00T server.

Use the existing Arena container terminal from earlier simulation lessons, or start a new one from your Isaac Lab-Arena checkout if needed.
```
./docker/run_docker.sh
```

Confirm the dataset and models environment variables are still set.

If you’re in a new container shell, restore them:

export DATASET_DIR=/datasets/isaaclab_arena/static_apple_tutorial
export MODELS_DIR=/models/isaaclab_arena/static_apple_tutorial

Get the IP address of the host running the server, or localhost if it is the same machine.

Caution

Before running, make sure the model_path entry in isaaclab_arena_gr00t/policy/config/g1_static_apple_gr00t_closedloop_config.yaml points to a checkpoint directory you are serving, such as /models/isaaclab_arena/static_apple_tutorial/gn1x_tuned_static_apple for the Hugging Face checkpoint or /models/isaaclab_arena/static_apple_tutorial/static_apple_n17_finetune/checkpoint-20000 for a locally trained checkpoint.

This path must match the --model-path value passed to run_gr00t_server.py.

Run the policy in a single environment with visualization through the GUI. If not using localhost, make sure to replace --remote_host localhost with the IP address of the host running the server.

/isaac-sim/python.sh isaaclab_arena/evaluation/policy_runner.py \
   --viz kit \
   --policy_type isaaclab_arena_gr00t.policy.gr00t_remote_closedloop_policy.Gr00tRemoteClosedloopPolicy \
   --policy_config_yaml_path isaaclab_arena_gr00t/policy/config/g1_static_apple_gr00t_closedloop_config.yaml \
   --remote_host localhost --remote_port 5555 \
   --num_steps 600 \
   --enable_cameras \
   galileo_g1_static_pick_and_place \
   --object apple_01_objaverse_robolab \
   --destination clay_plates_hot3d_robolab \
   --embodiment g1_wbc_agile_joint

Tip

Timeout tuning is important here - we used 6 seconds to allow the robot to retry once or twice, but not to spend too much time if an episode is failing.

Note

The --num_steps value is 600 rather than 1500 because the static apple-to-plate episode has no walking phase and runs for roughly half as long as the loco-manipulation variant.

The 600-step command is intended as a quick test.

To estimate success rate more reliably, evaluate complete episodes instead of relying on one short rollout:

Use --num_episodes 100 for a quick estimate.
Use --num_episodes 1000 for a stronger estimate.

If you prefer --num_steps, this task’s 6-second timeout comes from episode_length_s=6.0 in galileo_g1_static_pick_and_place_environment.py. At 50 Hz control, that is about 300 environment steps per episode, so 100 episodes is roughly --num_steps 30000 and 1000 episodes is roughly --num_steps 300000.

At the end of evaluation, the console should show similar metrics:

[Rank 0/1] Metrics: {'success_rate': 1.0, 'object_moved_rate': 1.0, 'num_episodes': 1}

These metrics are computed over the entire evaluation process and depend on policy quality, dataset quality, and the number of evaluation steps.

Run Parallel Environments Evaluation#

Parallel evaluation is also supported by the policy runner. The command below assumes the GR00T policy server is still running.

Test the policy in 5 parallel environments with visualization through the GUI.

If not using localhost, make sure to replace --remote_host localhost with the IP address of the host running the server

/isaac-sim/python.sh isaaclab_arena/evaluation/policy_runner.py \
   --viz kit \
   --policy_type isaaclab_arena_gr00t.policy.gr00t_remote_closedloop_policy.Gr00tRemoteClosedloopPolicy \
   --policy_config_yaml_path isaaclab_arena_gr00t/policy/config/g1_static_apple_gr00t_closedloop_config.yaml \
   --remote_host localhost \
   --remote_port 5555 \
   --num_steps 600 \
   --num_envs 5 \
   --enable_cameras \
   galileo_g1_static_pick_and_place \
   --object apple_01_objaverse_robolab \
   --destination clay_plates_hot3d_robolab \
   --embodiment g1_wbc_agile_joint

Note

With the server-client architecture, the policy device is a server-side concern. The server places the policy on its own GPU through the --device flag passed to run_gr00t_server.py. The client’s --device flag controls Arena’s physics backend, not the policy.

During evaluation, the console should indicate which environments terminated and which truncated. terminated means the task-specific success condition fired or the six-second episode length was exceeded. truncated means a timeout occurred, if timeouts are enabled, such as the maximum episode length being exceeded.

Resetting policy for terminated env_ids: tensor([3], device='cuda:0') and truncated env_ids: tensor([], device='cuda:0', dtype=torch.int64)

At the end of evaluation, the console should show metrics similar to:

[Rank 0/1] Metrics: {'success_rate': 1.0, 'num_episodes': 5}

The success rate may not be 1.0 when more trials are evaluated. The number of episodes is higher than single-environment evaluation because of parallelization.

Review Evaluation Details#

Closed-loop policy inference uses the g1_wbc_agile_joint embodiment, which is different from g1_wbc_agile_pink used during teleoperation recording. During teleoperation, the upper body is controlled through target end-effector poses realized by PinkIK, and the lower body is controlled through AGILE WBC. The GR00T 1.7 policy trains on upper-body joint positions and lower-body WBC policy inputs, so evaluation uses the joint-control twin.

The evaluation commands above do not pass --device, so Arena defaults to its built-in physics backend. If your dataset was recorded on GPU physics, prefer --device cuda for both single and parallel runs to keep evaluation physics aligned with training. If it was recorded on CPU physics, add --device cpu for per-episode reproducibility.

The same-shelf placement makes the static variant slightly easier than the loco-manipulation apple-to-plate task: the destination plate is always within arm’s reach, so the policy never has to recover from a mistimed approach, and there are no intermediate locomotion phases that can drift off-course. The success criterion is the same contact-sensor termination used by the loco-manipulation variant: force_threshold=0.5 N and velocity_threshold=0.1 m/s, filtered to contacts with the --destination asset.

Debugging Common Server-Client Failure Modes#

ValueError: Invalid action shape, expected: 23, received: 50. means the client embodiment expects a 23-D PinkIK action, but the server is returning a 43-DoF joint chunk. Make sure the client uses --embodiment g1_wbc_agile_joint, not g1_wbc_agile_pink.
ModuleNotFoundError on the client side means the client’s --policy_type is wrong. Use isaaclab_arena_gr00t.policy.gr00t_remote_closedloop_policy.Gr00tRemoteClosedloopPolicy with --policy_config_yaml_path isaaclab_arena_gr00t/policy/config/g1_static_apple_gr00t_closedloop_config.yaml.
Action shape mismatch on the server, such as Action key 'left_arm''s horizon must be 40. Got 50, means the action modality registered at training time disagrees with the modality loaded by the server. Re-fine-tune at the same horizon or update the --modality-config-path passed to run_gr00t_server.py to match the checkpoint.

Key Takeaways#

Simulation Workflow Complete

We evaluated the fine-tuned GR00T 1.7 policy through Arena’s server-client architecture. This completes the simulation workflow for the static apple-to-plate task.