`nemo_rl.experience.reward_penalties`#

Shared reward calculations and logging for verified capture rows.

Capture observations retain each valid row’s final reward until its group is consumed, allowing exact pooled medians, histograms and agent reward logging. Optional full-result JSON is retained only when full-result logging is enabled.

Module Contents#

Classes#

`RewardChecks`	Scored-text and original-prompt checks saved with a rollout attempt.
`CapturePenalty`	One supported penalty’s configuration, counter and logging contract.
`EffortRewardSettings`	Active effort parameters that affect saved reward interpretation.
`CaptureRewardSettings`	Semantic checkpoint identity; user-config extras are never persisted.
`RewardLogContext`	Agent identity and optional full-result table data, without raw reward.
`FinalizedReward`	A valid finalized sample’s reward, retained until group consumption.

Functions#

`has_duplicated_reasoning`	Detect nonempty reasoning copied into immediately adjacent content.
`has_empty_final_answer`	Detect missing supported final content, except a final function call.
`effort_shaping_enabled`	Return whether the configured low-effort adjustment is active.
`is_low_effort`	Classify the original rollout input using the standard last-user rule.
`shape_effort_reward`	Return shaped reward and length term, preserving negative reward semantics.
`has_unwanted_tokens`	Check full generated sequences, including their terminal tokens.
`parse_reward_log_context`	Validate the optional log context stored with an unfinished attempt.
`compute_reward_checks`	Evaluate available checks before sealing, without changing the raw reward.
`finalize_capture_reward`	Shape, then penalize. Token IDs and spans must already be verified by Gym.
`aggregate_capture_reward_metrics`	Pool per-group finalizer counts and per-row rewards over valid rows.
`capture_reward_result_tables`	Build the existing agent full-result table rows using final rewards.

Data#

`CAPTURE_PENALTIES`
`CAPTURE_PENALTY_METRICS`

API#

nemo_rl.experience.reward_penalties.has_duplicated_reasoning( output_items: list[dict[str, Any]], ) → bool[source]#: Detect nonempty reasoning copied into immediately adjacent content.

nemo_rl.experience.reward_penalties.has_empty_final_answer( output_items: list[dict[str, Any]], ) → bool[source]#: Detect missing supported final content, except a final function call.

nemo_rl.experience.reward_penalties.effort_shaping_enabled( config: nemo_rl.experience.rollouts.EffortLevelsConfig | None, ) → bool[source]#: Return whether the configured low-effort adjustment is active.

nemo_rl.experience.reward_penalties.is_low_effort( row: dict[str, Any], config: nemo_rl.experience.rollouts.EffortLevelsConfig, ) → bool[source]#: Classify the original rollout input using the standard last-user rule.

nemo_rl.experience.reward_penalties.shape_effort_reward( reward: float, *, length: int, config: nemo_rl.experience.rollouts.EffortLevelsConfig, ) → tuple[float, float][source]#: Return shaped reward and length term, preserving negative reward semantics.

nemo_rl.experience.reward_penalties.has_unwanted_tokens( generations: collections.abc.Iterable[collections.abc.Container[int]], unwanted_ids: collections.abc.Sequence[int], ) → bool[source]#: Check full generated sequences, including their terminal tokens.

class nemo_rl.experience.reward_penalties.RewardChecks[source]#

Scored-text and original-prompt checks saved with a rollout attempt.

duplicated_reasoning: bool#: None

empty_final_answer: bool#: None

low_effort: bool#: None

__post_init__() → None[source]#

class nemo_rl.experience.reward_penalties.CapturePenalty[source]#

One supported penalty’s configuration, counter and logging contract.

name: str#: None

flag: str#: None

metric: str#: None

enabled: collections.abc.Callable[[nemo_rl.algorithms.grpo.RewardPenaltyConfig], bool]#: None

text_match: collections.abc.Callable[[nemo_rl.experience.reward_penalties.RewardChecks], bool] | None#: None

nemo_rl.experience.reward_penalties.CAPTURE_PENALTIES#: ()

class nemo_rl.experience.reward_penalties.EffortRewardSettings[source]#

Active effort parameters that affect saved reward interpretation.

low_weight: float#: None

low_penalty: float#: None

low_ub: int#: None

low_string: str#: None

class nemo_rl.experience.reward_penalties.CaptureRewardSettings[source]#

Semantic checkpoint identity; user-config extras are never persisted.

enabled_penalties: tuple[str, ...]#: None

unwanted_tokens: tuple[int, ...]#: None

effort: nemo_rl.experience.reward_penalties.EffortRewardSettings | None#: None

classmethod from_configs( penalties: nemo_rl.algorithms.grpo.RewardPenaltyConfig, effort: nemo_rl.experience.rollouts.EffortLevelsConfig | None, ) → nemo_rl.experience.reward_penalties.CaptureRewardSettings[source]#

to_state() → dict[str, Any][source]#: Serialize version 1 using the existing config field names.

classmethod from_state( value: object, ) → nemo_rl.experience.reward_penalties.CaptureRewardSettings[source]#

Validate a version-1 record written by to_state.

Sidecars always write version 1; the live config never passes through this path (see from_configs). Existing model defaults fill absent config fields. A future change to reward semantics must explicitly version/migrate this projection.

require_compatible( current: nemo_rl.experience.reward_penalties.CaptureRewardSettings, ) → None[source]#: Reject changed reward rules with actionable field names.

class nemo_rl.experience.reward_penalties.RewardLogContext[source]#

Agent identity and optional full-result table data, without raw reward.

agent_name: str#: None

full_result_json: str | None#: None

__post_init__() → None[source]#

class nemo_rl.experience.reward_penalties.FinalizedReward[source]#

A valid finalized sample’s reward, retained until group consumption.

sample_id: str#: None

rollout_id: str#: None

reward: float#: None

log_context: nemo_rl.experience.reward_penalties.RewardLogContext | None#: None

__post_init__() → None[source]#

classmethod from_state( value: object, ) → nemo_rl.experience.reward_penalties.FinalizedReward[source]#: Validate checkpoint observations before using them for metrics.

nemo_rl.experience.reward_penalties.parse_reward_log_context( value: object, ) → nemo_rl.experience.reward_penalties.RewardLogContext | None[source]#: Validate the optional log context stored with an unfinished attempt.

nemo_rl.experience.reward_penalties.compute_reward_checks( result: dict[str, Any], row: dict[str, Any], effort_config: nemo_rl.experience.rollouts.EffortLevelsConfig | None, ) → nemo_rl.experience.reward_penalties.RewardChecks[source]#: Evaluate available checks before sealing, without changing the raw reward.

nemo_rl.experience.reward_penalties.finalize_capture_reward( reward: float, *, checks: nemo_rl.experience.reward_penalties.RewardChecks | None, penalty_config: nemo_rl.algorithms.grpo.RewardPenaltyConfig | None, effort_config: nemo_rl.experience.rollouts.EffortLevelsConfig | None, token_ids: list[int], link_spans: list[tuple[str, int, int]], ) → tuple[float, dict[str, int], dict[str, float]][source]#: Shape, then penalize. Token IDs and spans must already be verified by Gym.

nemo_rl.experience.reward_penalties.CAPTURE_PENALTY_METRICS#: None

nemo_rl.experience.reward_penalties.aggregate_capture_reward_metrics( metrics: dict[str, list[float]], observations: collections.abc.Sequence[nemo_rl.experience.reward_penalties.FinalizedReward], ) → dict[str, Any][source]#

Pool per-group finalizer counts and per-row rewards over valid rows.

metrics holds the per-group finalizer metrics of the consumed groups; observations holds one retained reward per valid row of those groups. Rates are computed over valid rows, never averaged across groups.

nemo_rl.experience.reward_penalties.capture_reward_result_tables( observations: collections.abc.Sequence[nemo_rl.experience.reward_penalties.FinalizedReward], ) → dict[str, list[list[str]]][source]#: Build the existing agent full-result table rows using final rewards.

nemo_rl.experience.reward_penalties#