Forms of Learning#

Robots can learn in different ways, depending on the available data and what you want them to do. Let’s explore three common forms: supervised learning (including behavior cloning), unsupervised learning, and reinforcement learning.

In practice, a robot often combines several forms of learning with classical methods such as motion planning and feedback control.

Tip

You’ll see a mixture of diagrams, equations, and code snippets in this module. If one of these formats is unfamiliar, take your time with it. You don’t need to understand every detail right away. Focus on connecting the different ways of showing the same idea.

Supervised Learning#

Two rows of boxes with image, label, model, and bounding-box icons show labeled images training an object detector, then a new camera image passing through the trained detector to predict object locations.

During training, labeled images teach an object detector to locate objects; during use, the trained detector predicts object locations in a new camera image.#

Supervised learning trains a model on input–target pairs, such as images and annotated object locations. During training, a loss measures how much the predictions differ from the targets. A randomly initialized model nearly always produces poor predictions at first; training adjusts its parameters to reduce that loss and improve predictions.

We can describe a dataset \(\mathcal{D}\) with \(N\) labeled examples as

\[\mathcal{D} = \{(x_i, y_i)\}_{i=1}^{N},\]

where \(x_i\) is an input and \(y_i\) is its target (sometimes called a label). A model \(f_\theta\), with learned parameters \(\theta\), makes a prediction

\[\hat{y}_i = f_\theta(x_i).\]

Training seeks parameters that minimize the average loss over the dataset:

\[\theta^* = \underset{\theta}{\operatorname{arg\,min}}\; \frac{1}{N}\sum_{i=1}^{N}\mathcal{L}\left(\hat{y}_i , y_i\right).\]

Here, \(\mathcal{L}\) is the loss function. Parameters are typically updated with stochastic gradient descent (stochastic means involving randomness). In plain terms, training uses a small, randomly chosen batch of examples to determine which parameter changes would reduce the loss, then repeats that adjustment with new batches.

For object detection, one target \(y_i\) can contain multiple class labels and bounding boxes, as in the following example. An input–target pair might be represented like this:

 1object_detection_example = {
 2    "input": "rgb/frame_0042.png",
 3    "target": {
 4        "labels": ["package", "tray"],
 5        "boxes_xyxy_pixels": [
 6            [120, 80, 210, 190],
 7            [300, 110, 520, 360],
 8        ],
 9    },
10}

A standard supervised training loop repeatedly predicts targets, measures the loss, and updates the model parameters. The plural variable names below represent a batch of the notation above: inputs contains multiple values of \(x_i\) (the images identified by each example’s input field), targets contains values of \(y_i\), and predictions contains the corresponding values of \(\hat{y}_i\).

 1training_data = load_object_detection_data()
 2
 3model = initialize_model_randomly()
 4
 5for epoch in range(number_of_epochs):
 6    for inputs, targets in training_data:
 7        predictions = model(inputs)
 8        
 9        loss = loss_function(predictions, targets)
10
11        gradients = compute_gradients(loss, model.parameters)
12        
13        update_parameters(model.parameters, gradients)

At inference time, when you use the trained model, it predicts locations for a new image (prediction = model(new_image)). The difference is that at inference time we don’t know the true target. In robotics, the prediction is an intermediate result. Accurate object detection alone does not establish that the robot can grasp, transport, and release the object successfully.

Imitation Learning#

A person demonstrates a task, paired observations and expert actions train a policy, and the robot executes it autonomously.

Demonstrated actions provide targets for the policy to imitate.#

Imitation learning trains a policy to mimic expert behavior, usually represented as trajectories of observations and actions (a sequence of observations and the actions chosen at each step).

Behavior cloning is the simplest form of imitation learning: it treats demonstrated actions as targets (as in supervised learning) and trains a policy to predict the expert’s action from an observation. Broader imitation-learning methods may request additional expert feedback, address distribution shift, or infer a reward function from demonstrations before optimizing a policy.

For behavior cloning, the expert demonstrations can be written as a dataset of \(N\) observation–action pairs:

\[\mathcal{D}_E = \{(o_i, a_i^E)\}_{i=1}^{N},\]

where \(o_i\) is an observation and \(a_i^E\) is the action selected by the expert. A policy \(\pi_\theta\) predicts an action

\[\hat{a}_i = \pi_\theta(o_i).\]

Both \(f_\theta\) in the supervised-learning section and \(\pi_\theta\) here are parameterized functions. The different symbols describe their roles: \(f\) denotes a generic function that maps an input to a prediction, while \(\pi\) conventionally denotes a policy that maps an observation to an action. This does not require a different model architecture; either function could be represented by a neural network. The equation above denotes a deterministic policy. You might also come across stochastic policies, which are commonly written as \(\pi_\theta(a \mid o)\).

Behavior cloning minimizes the disagreement between the predicted and expert actions:

\[\theta^* = \underset{\theta}{\operatorname{arg\,min}}\; \frac{1}{N}\sum_{i=1}^{N} \mathcal{L}\left(\hat{a}_i, a_i^E\right).\]

For example, a package-picking demonstration might pair camera and robot-state observations \(o_i\) with the expert’s next action \(a_i^E\):

 1demonstration_step = {
 2    "observation": {
 3        "camera": "rgb/frame_0128.png",
 4        "joint_positions_rad": [0.12, -0.83, 1.04, 0.31, -0.22, 0.48],
 5        "gripper_width_m": 0.042,
 6    },
 7    "expert_action": {
 8        "joint_position_targets_rad": [0.14, -0.79, 1.01, 0.35, -0.20, 0.46],
 9        "gripper_width_target_m": 0.018,
10    },
11}

Although this is a made-up example, notice that the joint_positions_rad and joint_position_targets_rad values are similar, suggesting that the expert is commanding a small change in joint positions based on the camera frame. The behavior-cloning loop resembles supervised learning with the expert actions serving as targets:

 1demonstration_data = load_expert_demonstrations()
 2
 3policy = initialize_policy_randomly()
 4
 5for epoch in range(number_of_epochs):
 6    for observations, expert_actions in demonstration_data:
 7        predicted_actions = policy(observations)
 8        
 9        loss = imitation_loss(predicted_actions, expert_actions)
10
11        gradients = compute_gradients(loss, policy.parameters)
12        
13        update_parameters(policy.parameters, gradients)

For continuous actions such as joint-position targets, imitation_loss might use mean squared error. For discrete actions such as open or close, it might use cross-entropy loss.

The main difficulty in learning from an expert is distribution shift: the robot’s own mistakes can take it into situations absent from the demonstrations. An expert may never miss a grasp, so successful demonstrations alone may contain no recovery behavior. Demonstrations that include recovery from mistakes can help the policy learn what to do in those situations.

Unsupervised Learning#

Unlabeled robot sensor observations are transformed into a feature space, where an unsupervised algorithm discovers clusters and an outlier without target labels or rewards.

Unlabeled observations can reveal clusters and outliers in the data.#

Unsupervised learning finds patterns in data without labeled targets or rewards. An unsupervised dataset therefore contains only inputs:

\[\mathcal{D} = \{x_i\}_{i=1}^{N}.\]

There is no provided target that specifies the correct result for each input. In robotics, \(x_i\) might be a camera frame, a point cloud, or a window of joint, force, and torque measurements (a window is a consecutive segment of time-series measurements). Common goals include grouping similar observations, learning compact representations, and identifying unusual readings.

For example, consider grouping robot sensor windows by similarity. Each input \(x_i\) can be converted into a feature vector \(z_i = g(x_i)\), where \(g\) is either fixed preprocessing or a learned encoder.

K-means is a well-known unsupervised clustering algorithm that divides feature vectors into a chosen number of groups, \(K\). Starting with \(K\) candidate centers, it alternates between assigning each feature vector to its nearest center and replacing each center with the mean of the features assigned to it. These steps repeat until the centers stop changing appreciably. The corresponding objective is

\[\underset{\{c_i\},\{\mu_k\}}{\operatorname{minimize}}\; \frac{1}{N}\sum_{i=1}^{N}\left\|z_i - \mu_{c_i}\right\|_2^2,\]

where \(c_i\) is the cluster assigned to example \(i\) and \(\mu_k\) is the center of cluster \(k\). The objective favors compact clusters; it does not say what those clusters mean. After inspection, a robotics engineer might find that they correspond to free motion, expected contact, and slipping, but those names were not supplied during training.

One unlabeled sensor window might look like this:

 1sensor_window = {
 2    "input": {
 3        "sample_times_s": [0.00, 0.01, 0.02],
 4        "joint_velocity_rad_s": [
 5            [0.18, -0.04, 0.11, 0.07, -0.02, 0.03],
 6            [0.19, -0.03, 0.12, 0.06, -0.01, 0.03],
 7            [0.16, -0.02, 0.10, 0.05, -0.01, 0.02],
 8        ],
 9        "gripper_force_n": [1.2, 1.5, 2.8],
10        "camera_frames": [
11            "rgb/frame_0192.png",
12            "rgb/frame_0193.png",
13            "rgb/frame_0194.png",
14        ],
15    },
16}

There is no target field, just a collection of data that can be processed by a feature extractor or encoder. A standard K-means loop can group features derived from many such inputs:

 1sensor_windows = load_unlabeled_sensor_windows()
 2
 3features = extract_features(sensor_windows)
 4
 5centers = initialize_cluster_centers(features, number_of_clusters=3)
 6
 7for iteration in range(number_of_iterations):
 8    assignments = assign_to_nearest_center(features, centers)
 9    
10    new_centers = average_features_in_each_cluster(features, assignments)
11
12    converged = have_converged(centers, new_centers)
13    
14    centers = new_centers
15    
16    if converged:
17        break

The resulting assignments can help organize a dataset or flag observations far from every cluster center as possible anomalies. A system can use those flags to request a closer check of a measurement. Other unsupervised methods learn lower-dimensional representations instead of explicit clusters. Closely related, self-supervised learning creates its training targets from the data itself.

Unlike reinforcement learning, unsupervised learning does not require the robot to explore an environment and receive rewards. This makes it useful when sensor data is plentiful but annotations are scarce. However, a discovered pattern is not automatically meaningful for the robot’s task, so the learned clusters or representations must still be evaluated and connected to a downstream objective like grasping or navigation.

Reinforcement Learning#

A policy acts in an environment; recorded transitions and rewards support parameter updates. Miss, lift, and place illustrate possible outcomes.

The robot interacts with an environment, and the resulting transitions and rewards guide updates to its policy.#

Reinforcement learning is explained in more detail on the next page. Here, you’ll get a high-level overview and see how it compares to the other forms of learning.

Reinforcement learning trains a policy through interaction with an environment. The robot takes actions, observes their outcomes, and receives rewards that indicate how well those outcomes support the desired task. The learning algorithm uses this experience to improve future behavior, so the reward must accurately represent the intended goal.

Unlike supervised learning and behavior cloning, the experience does not contain the correct action for each observation. Unlike unsupervised learning, the experience does not contain only inputs. Instead, policy updates favor actions that led to greater reward over time.

For the gear-assembly task, success means fully seating the gear with its teeth meshed. You could reward the policy only when it completes that task. But an initially untrained policy may rarely succeed, so most attempts would receive no reward and provide little guidance about which actions help. This is a sparse reward.

To help the policy learn, you might also reward progress: aligning the already-grasped gear, engaging its teeth with the installed gear, and seating it. However, if you reward too much for close proximity, the policy might learn to hover above the shaft without ever meshing the teeth. The reward must be designed carefully to encourage completion of the task.

The robot’s experience can be collected in simulation or the real world. However, learning requires exploration and mistakes, which can be costly in the real world. Simulation allows for safe exploration and faster learning through parallelization. The next page develops the details of transitions, episodes, returns, and policy updates.

Comparison of Learning Forms#

Form of Learning

Training Data

Learning Signal

What It Learns

Robot Example

Main Challenge

Supervised learning

Input–target pairs

Loss between predictions and targets

A mapping from inputs to predictions

Detect packages and trays from labeled camera images

Accuracy doesn’t necessarily lead to good behaviors.

Behavior cloning

Expert observation–action pairs

Loss between predicted and expert actions

A policy that predicts expert actions from observations

Reproduce a demonstrated package-picking task

Mistakes can lead to situations absent from the demonstrations (distribution shift).

Unsupervised learning

Unlabeled observations

An objective that captures structure, such as distances to cluster centers

Clusters or compact representations

Group sensor windows by similarity or flag unusual readings

Discovered patterns need interpretation and evaluation for the robot’s task.

Reinforcement learning

Experience from environment interaction

Rewards accumulated over time

A policy that chooses actions to increase return

Learn to align, mesh, and seat a gear

Sparse or poorly designed rewards can hinder learning or encourage unintended behavior.

Review#

Check Your Understanding#

What learning signal does reinforcement learning use?

Answer

Reinforcement learning uses rewards from interaction with the environment rather than supplied correct actions.

Apply What You Learned#

For one robot task, how would supervised, imitation, and reinforcement learning differ in the data or feedback they need? Describe the training inputs, targets or rewards, and where each source of feedback would come from.

Agent Tutor Prompt

Copy this prompt and include your answer:

Review my answer to this question: For one robot task, how would supervised, imitation, and reinforcement learning differ in the data or feedback they need? Describe the training inputs, targets or rewards, and where each source of feedback would come from.

My answer: [paste your answer here].

Check my reasoning about inputs, targets or rewards, and the overlap between demonstrations and labeled examples. Ask for missing context rather than assuming it. Identify what I explained well, point out one gap or misconception, and ask a follow-up question that helps me improve my answer. Let me revise it before supplying a complete answer.

Key Takeaways#

  • Supervised learning fits targets; behavior cloning fits expert actions.

  • Unsupervised learning finds structure without supplied targets.

  • In Isaac Lab, reinforcement learning gathers simulated interaction and rewards to update a policy.