Multi-Reward Verification

When to Use Multiple Rewards

Use multiple reward components when:

• You care about several independent objectives at once — e.g. did the agent pick the right action, were its arguments well-formed, did it follow the output format — and collapsing them into one number would hide which objective failed.
• You want to diagnose how an agent fails during evaluation, not just whether it passed.
• You plan to train with a multi-objective algorithm such as GDPO, which normalizes each objective separately.

If a single pass/fail signal captures everything you care about, a scalar reward is simpler — don’t add components you won’t use.

The Verifier Contract

A multi-reward verifier is a normal resources server with one change to what verify() returns. Return three forms of the same scores:

The recipe is three steps:

1. Scaffold a resources server

Skip if you already have one. Otherwise generate the app, config, and test layout:

$
gym env init --resources-server my_server

2. Subclass BaseMultiRewardVerifyResponse with top-level component fields

BaseMultiRewardVerifyResponse defines the shared multi-reward contract: a required {objective_name: score} dict (reward_components). Use the same objective keys for every task in an environment. Environments that only return a scalar reward should continue to use BaseVerifyResponse.

Subclass it and declare one top-level float field per objective — don’t leave the subclass empty. Aggregate metrics computes a stat for every top-level numeric field, but does not descend into nested dicts, so a key inside reward_components alone won’t appear in your metrics.

1
from nemo_gym.base_resources_server import BaseMultiRewardVerifyResponse
2
3
4
class MultiRewardResponse(BaseMultiRewardVerifyResponse):
5
    # One top-level field per objective so aggregate metrics can profile each.
6
    correctness: float = 0.0
7
    schema_valid: float = 0.0
8
    format: float = 0.0

3. Set the scalar reward

Every verify response carries a scalar reward, and most consumers read it rather than the components: aggregate metrics reports it as the overall score for evaluation, and single-reward trainers (e.g. GRPO) use it directly. A weighted sum of the components is commonly used as the scalar reward.

Try It

We’ll make the contract concrete with the example_tool_call_multireward server, which asks the agent to call get_weather for a city and scores the rollout on three independent {0, 1} components: did the agent call the right tool, were its arguments well-formed, and did it emit exactly one call with no extra text. Here each rollout is a single tool call, but the pattern applies to multi-step trajectories too.

1. Carry the ground truth into verify()

The example data carries an expected_call describing the tool call the agent should make:

1
{"responses_create_params": {"input": [{"content": "You are a helpful assistant. When the user asks about the weather, respond with exactly one get_weather tool call and no other text.", "role": "developer"}, {"content": "what's the weather in San Francisco?", "role": "user"}], "tools": [{"name": "get_weather", "parameters": {"type": "object", "properties": {"city": {"type": "string", "description": "The city to get the weather for."}}, "required": ["city"], "additionalProperties": false}, "strict": true, "type": "function", "description": "Get the current weather for a city."}]}, "expected_call": {"name": "get_weather", "arguments": {"city": "San Francisco"}}}

Subclass BaseVerifyRequest to declare any extra per-task fields your scoring needs:

1
from typing import Any, Dict
2
3
from pydantic import Field
4
5
from nemo_gym.base_resources_server import BaseVerifyRequest
6
7
8
class ToolCallMultiRewardVerifyRequest(BaseVerifyRequest):
9
    expected_call: Dict[str, Any] = Field(default_factory=dict)

The subclass is required for expected_call to survive Pydantic parsing: parsing into BaseVerifyRequest directly would silently drop any field not defined on the base class.

2. Score each objective independently

Compute one score per objective. Keeping them independent is what lets evaluation and GDPO treat them separately. The example uses three {0, 1} checks:

1
async def verify(self, body: ToolCallMultiRewardVerifyRequest) -> ToolCallMultiRewardVerifyResponse:
2
    predicted_calls = self._extract_function_calls(body)
3
4
    # format: exactly one tool call and no extra assistant prose.
5
    format_score = 1.0 if len(predicted_calls) == 1 and not self._has_assistant_text(body) else 0.0
6
7
    # schema_valid: the first call's arguments are a valid object with all required params.
8
    schema_score = 0.0
9
    if predicted_calls:
10
        first = predicted_calls[0]
11
        parsed, is_valid = self._parse_arguments(first["arguments"])
12
        required = self._required_params(body, first["name"])
13
        schema_score = 1.0 if is_valid and all(k in parsed for k in required) else 0.0
14
15
    # correctness: some predicted call matches the expected name + arguments.
16
    correctness_score = 1.0 if any(self._call_matches(c, body.expected_call) for c in predicted_calls) else 0.0
17
18
    # ... continues in "Return all three forms" below

The self._… helpers (extracting calls, parsing arguments, checking required params, matching the expected call) are defined on the server — see app.py. What matters for the pattern is that each objective produces its own score.

3. Return all three forms

The example’s response class inherits the required reward_components dict from BaseMultiRewardVerifyResponse and adds the three top-level component fields. The scalar reward is inherited through BaseVerifyResponse, so it isn’t redeclared here — verify() computes its value (the sum) in the return below.

1
class ToolCallMultiRewardVerifyResponse(BaseMultiRewardVerifyResponse):
2
    correctness: float = 0.0
3
    schema_valid: float = 0.0
4
    format: float = 0.0

Continuing the same verify() method body from step 2, assemble the three forms and return them:

1
    # ... continued from "Score each objective" (same verify() method body)
2
    reward_components = {
3
        "correctness": correctness_score,
4
        "schema_valid": schema_score,
5
        "format": format_score,
6
    }
7
8
    return ToolCallMultiRewardVerifyResponse(
9
        **body.model_dump(),
10
        reward=sum(reward_components.values()),   # scalar for single-reward consumers
11
        reward_components=reward_components,      # dict for all reward components
12
        correctness=correctness_score,            # top-level fields for aggregate metrics
13
        schema_valid=schema_score,
14
        format=format_score,
15
    )

See resources_servers/example_tool_call_multireward/app.py for the full implementation, including the argument-parsing and call-matching helpers.

Use Multi-Reward Verification for Evaluation

Collect rollouts as with any other environment:

$
gym env start \
>
    --model-type openai_model \
>
    --resources-server example_tool_call_multireward
$
$
gym eval run --no-serve --agent example_tool_call_multireward_simple_agent \
>
    --input resources_servers/example_tool_call_multireward/data/example.jsonl \
>
    --output resources_servers/example_tool_call_multireward/data/example_rollouts.jsonl \
>
    --limit null --num-repeats null --concurrency null

Because correctness, schema_valid, and format are top-level numeric fields, the Aggregate Metrics step reports a separate mean (pass rate) for each one alongside the summed reward. After gym eval run, those stats land next to your rollouts as <output>_aggregate_metrics.json — for the command above, that is resources_servers/example_tool_call_multireward/data/example_rollouts_aggregate_metrics.json. The file has an entry per component (illustrative):

1
[
2
  {
3
    "agent_ref": {"name": "example_tool_call_multireward_simple_agent"},
4
    "agent_metrics": {
5
      "mean/reward": 2.6,
6
      "mean/correctness": 0.8,
7
      "mean/schema_valid": 1.0,
8
      "mean/format": 0.8
9
    },
10
    "key_metrics": {
11
      "mean/reward": 2.6
12
    }
13
  }
14
]

That breakdown tells you where the agent struggles. Above, the model always emits schema-valid calls (schema_valid = 1.0) and usually picks the right city (correctness = 0.8), but sometimes wraps the call in extra prose (format = 0.8) — detail a single conflated score would hide.

Use Multi-Reward Verification for Multi-Objective RL (E.g. GDPO)

The same verifier feeds multi-objective training: a GDPO-style algorithm reads the per-objective scores from reward_components and normalizes each component independently, rather than collapsing them into one number. A GRPO baseline instead reads the summed reward and therefore cannot distinguish two rollouts with the same total but different composition (e.g. correct-but-malformed vs. wrong-but-clean) — the advantage collapse that GDPO is designed to fix.

How reward_components reaches the trainer depends on the training framework’s NeMo Gym integration; see Integrate RL Frameworks for current support.