> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/gym/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/gym/_mcp/server.

# Multi-Reward Verification

> Score a rollout on multiple reward components for richer evaluation and multi-objective RL

**Goal**: Learn how to design a verifier to return multiple reward components for a single rollout, useful for both evaluation and multi-objective RL.

This page assumes you know how a resources server's `verify()` method receives a rollout and returns a reward. For background, see [Environment Components](/about/architecture) for how the resources server fits alongside the agent and model servers, and the [Single-Step Environment](/environment-tutorials/single-step-environment) tutorial for a basic `verify()` walkthrough.

---

## When to Use Multiple Rewards

Use multiple reward components when:

* **You care about several independent objectives at once** — e.g. *did the agent pick the right action*, *were its arguments well-formed*, *did it follow the output format* — and collapsing them into one number would hide which objective failed.
* You want to diagnose **how an agent fails during evaluation**, not just whether it passed.
* You plan to train with a **multi-objective algorithm** such as GDPO, which normalizes each objective separately.

If a single pass/fail signal captures everything you care about, a scalar `reward` is simpler — don't add components you won't use.

---

## The Verifier Contract

A multi-reward verifier is a normal resources server with one change to what `verify()` returns. Return **three forms** of the same scores:

| Form                                         | Who reads it                                                     |
| -------------------------------------------- | ---------------------------------------------------------------- |
| Scalar `reward`                              | GRPO and other single-reward consumers; overall score in reports |
| `reward_components` dict                     | GDPO and other multi-objective trainers                          |
| Top-level numeric fields (one per objective) | Aggregate metrics (per-objective pass rates)                     |

The recipe is three steps:

### 1. Scaffold a resources server

Skip if you already have one. Otherwise generate the app, config, and test layout:

```bash
gym env init --resources-server my_server
```

### 2. Subclass `BaseMultiRewardVerifyResponse` with top-level component fields

`BaseMultiRewardVerifyResponse` defines the shared multi-reward contract: a required `{objective_name: score}` dict (`reward_components`). Use the same objective keys for every task in an environment. Environments that only return a scalar reward should continue to use `BaseVerifyResponse`.

Subclass it and declare **one top-level float field per objective** — don't leave the subclass empty. Aggregate metrics computes a stat for every top-level numeric field, but does not descend into nested dicts, so a key inside `reward_components` alone won't appear in your metrics.

```python
from nemo_gym.base_resources_server import BaseMultiRewardVerifyResponse


class MultiRewardResponse(BaseMultiRewardVerifyResponse):
    # One top-level field per objective so aggregate metrics can profile each.
    correctness: float = 0.0
    schema_valid: float = 0.0
    format: float = 0.0
```

### 3. Set the scalar `reward`

Every verify response carries a scalar `reward`, and most consumers read it rather than the components: aggregate metrics reports it as the overall score for evaluation, and single-reward trainers (e.g. GRPO) use it directly. A weighted sum of the components is commonly used as the scalar reward.

---

## Try It

We'll make the contract concrete with the [`example_tool_call_multireward`](https://github.com/NVIDIA-NeMo/Gym/tree/main/resources_servers/example_tool_call_multireward) server, which asks the agent to call `get_weather` for a city and scores the rollout on three independent `{0, 1}` components: did the agent call the right tool, were its arguments well-formed, and did it emit exactly one call with no extra text. Here each rollout is a single tool call, but the pattern applies to multi-step trajectories too.

### 1. Carry the ground truth into `verify()`

The example data carries an `expected_call` describing the tool call the agent should make:

```json
{"responses_create_params": {"input": [{"content": "You are a helpful assistant. When the user asks about the weather, respond with exactly one get_weather tool call and no other text.", "role": "developer"}, {"content": "what's the weather in San Francisco?", "role": "user"}], "tools": [{"name": "get_weather", "parameters": {"type": "object", "properties": {"city": {"type": "string", "description": "The city to get the weather for."}}, "required": ["city"], "additionalProperties": false}, "strict": true, "type": "function", "description": "Get the current weather for a city."}]}, "expected_call": {"name": "get_weather", "arguments": {"city": "San Francisco"}}}
```

Subclass `BaseVerifyRequest` to declare any extra per-task fields your scoring needs:

```python
from typing import Any, Dict

from pydantic import Field

from nemo_gym.base_resources_server import BaseVerifyRequest


class ToolCallMultiRewardVerifyRequest(BaseVerifyRequest):
    expected_call: Dict[str, Any] = Field(default_factory=dict)
```

The subclass is required for `expected_call` to survive Pydantic parsing: parsing into `BaseVerifyRequest` directly would silently drop any field not defined on the base class.

### 2. Score each objective independently

Compute one score per objective. Keeping them independent is what lets evaluation and GDPO treat them separately. The example uses three `{0, 1}` checks:

| Component      | Measures                                                                         |
| -------------- | -------------------------------------------------------------------------------- |
| `correctness`  | A predicted call matches the expected tool name and arguments.                   |
| `schema_valid` | The call's arguments parse as a JSON object containing every required parameter. |
| `format`       | Exactly one tool call was emitted, with no extra assistant text.                 |

```python
async def verify(self, body: ToolCallMultiRewardVerifyRequest) -> ToolCallMultiRewardVerifyResponse:
    predicted_calls = self._extract_function_calls(body)

    # format: exactly one tool call and no extra assistant prose.
    format_score = 1.0 if len(predicted_calls) == 1 and not self._has_assistant_text(body) else 0.0

    # schema_valid: the first call's arguments are a valid object with all required params.
    schema_score = 0.0
    if predicted_calls:
        first = predicted_calls[0]
        parsed, is_valid = self._parse_arguments(first["arguments"])
        required = self._required_params(body, first["name"])
        schema_score = 1.0 if is_valid and all(k in parsed for k in required) else 0.0

    # correctness: some predicted call matches the expected name + arguments.
    correctness_score = 1.0 if any(self._call_matches(c, body.expected_call) for c in predicted_calls) else 0.0

    # ... continues in "Return all three forms" below
```

The `self._…` helpers (extracting calls, parsing arguments, checking required params, matching the expected call) are defined on the server — see `app.py`. What matters for the pattern is that each objective produces its own score.

### 3. Return all three forms

The example's response class inherits the required `reward_components` dict from `BaseMultiRewardVerifyResponse` and adds the three top-level component fields. The scalar `reward` is inherited through `BaseVerifyResponse`, so it isn't redeclared here — `verify()` computes its value (the sum) in the return below.

```python
class ToolCallMultiRewardVerifyResponse(BaseMultiRewardVerifyResponse):
    correctness: float = 0.0
    schema_valid: float = 0.0
    format: float = 0.0
```

Continuing the same `verify()` method body from step 2, assemble the three forms and return them:

```python
    # ... continued from "Score each objective" (same verify() method body)
    reward_components = {
        "correctness": correctness_score,
        "schema_valid": schema_score,
        "format": format_score,
    }

    return ToolCallMultiRewardVerifyResponse(
        **body.model_dump(),
        reward=sum(reward_components.values()),   # scalar for single-reward consumers
        reward_components=reward_components,      # dict for all reward components
        correctness=correctness_score,            # top-level fields for aggregate metrics
        schema_valid=schema_score,
        format=format_score,
    )
```

See [`resources_servers/example_tool_call_multireward/app.py`](https://github.com/NVIDIA-NeMo/Gym/blob/main/resources_servers/example_tool_call_multireward/app.py) for the full implementation, including the argument-parsing and call-matching helpers.

---

## Use Multi-Reward Verification for Evaluation

Collect rollouts as with any other environment:

```bash
gym env start \
    --model-type openai_model \
    --resources-server example_tool_call_multireward

gym eval run --no-serve --agent example_tool_call_multireward_simple_agent \
    --input resources_servers/example_tool_call_multireward/data/example.jsonl \
    --output resources_servers/example_tool_call_multireward/data/example_rollouts.jsonl \
    --limit null --num-repeats null --concurrency null
```

Because `correctness`, `schema_valid`, and `format` are top-level numeric fields, the [Aggregate Metrics](/evaluation/aggregate-metrics) step reports a separate mean (pass rate) for each one alongside the summed `reward`. After `gym eval run`, those stats land next to your rollouts as `<output>_aggregate_metrics.json` — for the command above, that is `resources_servers/example_tool_call_multireward/data/example_rollouts_aggregate_metrics.json`. The file has an entry per component (illustrative):

```json
[
  {
    "agent_ref": {"name": "example_tool_call_multireward_simple_agent"},
    "agent_metrics": {
      "mean/reward": 2.6,
      "mean/correctness": 0.8,
      "mean/schema_valid": 1.0,
      "mean/format": 0.8
    },
    "key_metrics": {
      "mean/reward": 2.6
    }
  }
]
```

That breakdown tells you *where* the agent struggles. Above, the model always emits schema-valid calls (`schema_valid` = 1.0) and usually picks the right city (`correctness` = 0.8), but sometimes wraps the call in extra prose (`format` = 0.8) — detail a single conflated score would hide.

---

## Use Multi-Reward Verification for Multi-Objective RL (E.g. GDPO)

The same verifier feeds multi-objective training: a GDPO-style algorithm reads the per-objective scores from `reward_components` and normalizes each component independently, rather than collapsing them into one number. A GRPO baseline instead reads the summed `reward` and therefore cannot distinguish two rollouts with the same total but different composition (e.g. correct-but-malformed vs. wrong-but-clean) — the advantage collapse that GDPO is designed to fix.

How `reward_components` reaches the trainer depends on the training framework's NeMo Gym integration; see [Integrate RL Frameworks](/contribute/rl-framework-integration) for current support.