Reward Design and Evaluation#

When you train with reinforcement learning, the policy does not read your task description. The learning algorithm uses rewards to improve a policy that acts from observations. You’ll start by translating the behavior you want into a useful training reward. Then you’ll check whether the learned behavior actually solves the task. A policy can receive a high reward while still failing to complete the intended task.

The intended task is translated into weighted reward terms used for training. A frozen policy is then evaluated with task-level success measures, and observed failures inform the next reward-design experiment.

Reward design and task evaluation answer different questions. Keep both in the experimental loop.#

Reward Design and Task Success#

A task-success criterion is a testable statement about the desired outcome. For the gear-assembly robot, you might count an episode as successful only when the gear is fully seated on its shaft, its teeth mesh with the installed gear, and its speed remains below a chosen threshold for several control steps. This criterion describes the result you care about; it does not need to be smooth or differentiable.

A reward is the scalar feedback used to train the policy. A sparse reward might give a positive value only when the complete success criterion is met. This is faithful to the goal, but an untrained policy may almost never encounter it, which makes training slow. Reward shaping adds intermediate feedback for progress such as approaching (\(r\)), aligning (\(a\)), meshing (\(m\)), seating (\(s\)), and effort (\(e\)). These terms increase the frequency of nonzero feedback, which can speed up training:

\[r_{t} = w_{\mathrm{r}} r_{\mathrm{r},t} + w_{\mathrm{a}} r_{\mathrm{a},t} + w_{\mathrm{m}} r_{\mathrm{m},t} + w_{\mathrm{s}} r_{\mathrm{s},t} - w_{\mathrm{e}} c_{\mathrm{e},t}.\]

The first four terms reward progress, while the final term penalizes the robot for excessive effort (i.e., forces and torques). The weights (\(w\)) determine how strongly each part contributes.

Before you train, write down five characteristics of each term in your reward function:

  1. How can you measure it?

  2. What numerical range can it produce?

  3. Is it discrete (success or failure) or continuous (distance to target)?

  4. How important is it relative to other terms?

  5. How can you make it gradually more difficult to achieve?

Then do a simple accounting check. Compare the greatest return available from an incomplete behavior with the return for finishing the task. A policy should not earn more by hovering over the target, repeatedly triggering a milestone, holding the gear above the shaft forever, or ending the episode early.

Reward terms can also conflict. If you make a motion penalty too large, the robot may learn that standing still is safer than reaching. A reward paid for every step near the target may encourage the robot to wait instead of finish. A distance reward may bring the gear to the shaft without teaching tooth engagement or full seating. These are examples of reward exploitation: the policy follows the score you wrote, but not the intention you had.

To catch this problem, inspect representative episodes (what reward do you get when you manually control the robot?) and plot each reward component separately. A single total return can hide one large positive term and several important failures.

Tip

Training return values correlate with the question “how well did the policy optimize this reward?” On the other hand, success rates correlate with “did the robot complete the intended behavior?”

Evaluating Learned Policies#

During evaluation, hold the policy parameters fixed and test a named checkpoint under a defined set of conditions. Record enough information for another practitioner to repeat your result:

  • the checkpoint, task configuration, software version, and random seed;

  • the initial-condition distribution and any domain randomization;

  • whether actions are sampled or chosen deterministically;

  • the episode limit and the meanings of termination and truncation;

  • the number of independent episodes; and

  • the success criterion and other reported metrics.

Reproducibility lets you compare a new policy against the same baseline and investigate a surprising result without guessing which setup changed. A useful benchmark states the task, evaluation conditions, and success measure, then reports both the baseline and the new policy under those conditions.

Compare policies based on task-level measures such as success rate, time to completion, peak contact force, and the frequency of important failure modes. Rewards are a useful diagnostic, but they do not replace these measures. Visual playback can reveal behavior that aggregate metrics hide.

Compare policies under the same conditions. For example, in later lessons the state teacher and camera student policies will use the same physical success criterion, starting-state distribution, episode budget, and action meaning. Differences in their observations are part of the comparison. If you also change the evaluator, you cannot tell which change caused the result.

If your policy succeeds in \(m\) of \(n\) episodes, its measured success rate is \(\hat{p}=m/n\). This is an estimate, not a guarantee. Testing more episodes reduces sampling uncertainty, but it cannot reveal a failure condition that you never test. If you want to compare learning methods rather than two particular checkpoints, repeat training with multiple random seeds as well.

Diagnosing Failures#

If your policy isn’t behaving as you hoped, start by finding where things go wrong before changing a hyperparameter:

  1. Group failures by stage (e.g., reach, grasp, lift, reorient).

  2. Compare task success with reward components and critic predictions.

  3. Inspect representative trajectories from the dominant failure group.

  4. Change one justified part of the task, reward, observation, or training configuration and rerun the same evaluation.

This process turns training into a controlled experiment. A change is useful when it improves measured task performance across the conditions that matter, not only when it raises the training curve.

Review#

Check Your Understanding#

Does a higher training reward prove that a policy completes the task?

Answer

No. A policy can earn shaping rewards without completing the task. Evaluate task success separately from training reward.

Apply What You Learned#

How would you evaluate a policy that lifts the vial but rarely releases it into the rack? Define successful placement and the conditions and measurements you would use to compare policies.

Agent Tutor Prompt

Copy this prompt and include your answer:

Review my answer to this question: How would you evaluate a policy that lifts the vial but rarely releases it into the rack? Define successful placement and the conditions and measurements you would use to compare policies.

My answer: [paste your answer here].

Check my reasoning about direct success criterion, reward terms, playback, completed episodes, and fixed comparison conditions. Ask for missing context rather than assuming it. Identify what I explained well, point out one gap or misconception, and ask a follow-up question that helps me improve my answer. Let me revise it before supplying a complete answer.

Key Takeaways#

  • Reward shapes training behavior; task success needs its own measurement.

  • Evaluate complete episodes under recorded conditions.

  • Use Isaac Lab reward terms, ending flags, and playback together to diagnose failures.

Resources#