`nemo_rl.algorithms.loss.clipped_pg`#

Module Contents#

`materialize_scalar_metrics`	Copy scalar metrics to the host in one synchronization.
`clipped_pg_diagnostic_metrics`	Compute optional diagnostics for the full metric set.
`clipped_pg_actor_objective`	Compute and reduce the actor objective in one compilable region.

nemo_rl.algorithms.loss.clipped_pg.materialize_scalar_metrics( metrics: dict[str, torch.Tensor], dtype: torch.dtype, ) → dict[str, float]#: Copy scalar metrics to the host in one synchronization.

nemo_rl.algorithms.loss.clipped_pg.clipped_pg_diagnostic_metrics( *, curr_logprobs: torch.Tensor, prev_logprobs: torch.Tensor, generation_logprobs: torch.Tensor, ratios: torch.Tensor, ratios_clamped: torch.Tensor, mask: torch.Tensor, global_valid_toks: torch.Tensor, reference_policy_kl_type: str, ) → dict[str, torch.Tensor]#: Compute optional diagnostics for the full metric set.

nemo_rl.algorithms.loss.clipped_pg.clipped_pg_actor_objective( curr_logprobs: torch.Tensor, *, prev_logprobs: torch.Tensor, advantages: torch.Tensor, token_mask: torch.Tensor, mask: torch.Tensor, sample_mask: torch.Tensor, importance_weights: Optional[torch.Tensor], global_valid_toks: torch.Tensor, global_valid_seqs: torch.Tensor, ratio_clip_min: float, ratio_clip_max: float, ratio_clip_c: float, disable_ppo_ratio: bool, force_on_policy_ratio: bool, sequence_level_importance_ratios: bool, use_cispo: bool, token_level_loss: bool, ) → tuple[torch.Tensor, torch.Tensor, torch.Tensor]#: Compute and reduce the actor objective in one compilable region.