`nemo_rl.experience.failures`#

Rollout failure taxonomy for the SingleController path.

A failed rollout attempt is one of exactly two things, and the distinction drives the whole retry policy:

Infra (:class:RolloutInfraFailure) — the prompt is fine, the fleet is not. A dead generation shard, a timeout, a dropped connection. Re-dispatching the same prompt is expected to succeed because the retry lands on a different shard.
Data (:class:RolloutDataFailure) — deterministic and prompt-specific. A prompt longer than the engine’s max_model_len, non-contiguous tokens after tokenization. Another shard fails the same way, so the retry budget is small and exhausting it is reported rather than absorbed.

func:: classify_rollout_failure maps an arbitrary exception onto that split. Anything not recognized as infrastructure is treated as data — an unrecognized exception is more likely a real bug than a transient blip, and the data path surfaces it loudly instead of retrying it into silence.

Classify where the information still exists. An exception that crosses a Ray actor boundary arrives stripped: Ray pickles the cause, and anything unpicklable (aiohttp’s CIMultiDictProxy headers, for one) is replaced by a bare error carrying neither type nor .status. Driver-side classification is then guessing. So failures raised inside an actor are converted to a picklable member of this taxonomy before they cross —

func:: http_status_is_infra exists to keep that decision identical on both sides. See _typed_gym_failure in nemo_rl/environments/nemo_gym.py.

Module Contents#

Classes#

Retry policy bucket for a failed rollout attempt.

Functions#

`http_status_is_infra`	Whether an HTTP status means the endpoint is unwell rather than the request bad.
`_http_status`	Return the HTTP status carried by an aiohttp response error, if this is one.
`_is_infra`	Return whether this single exception (ignoring its cause chain) is infra.
`classify_rollout_failure`	Bucket a rollout exception into `INFRA` or `DATA`.

Data#

`_AIOHTTP_INFRA_TYPES`
`_CLIENT_RESPONSE_ERROR`
`_MAX_CAUSE_DEPTH`
`_RETRIABLE_HTTP_STATUSES`
`_INFRA_TYPES`
`_INFRA_TYPE_NAMES`

API#

nemo_rl.experience.failures._AIOHTTP_INFRA_TYPES: tuple[type[BaseException], ...]#: None

nemo_rl.experience.failures._CLIENT_RESPONSE_ERROR: Optional[type[BaseException]]#: None

nemo_rl.experience.failures._MAX_CAUSE_DEPTH: Final[int]#: 8

nemo_rl.experience.failures._RETRIABLE_HTTP_STATUSES: Final[frozenset[int]]#: ‘frozenset(…)’

exception nemo_rl.experience.failures.RolloutFailure#

Bases: Exception

Base class for failures that terminate a single rollout attempt.

Initialization

Initialize self. See help(type(self)) for accurate signature.

exception nemo_rl.experience.failures.RolloutInfraFailure#

Bases: nemo_rl.experience.failures.RolloutFailure

The generation or environment infrastructure could not serve this rollout.

The prompt is not implicated. Re-dispatching it is expected to succeed once a healthy shard is selected.

Initialization

Initialize self. See help(type(self)) for accurate signature.

exception nemo_rl.experience.failures.RolloutTimeout#

Bases: nemo_rl.experience.failures.RolloutInfraFailure

A rollout, generation turn, or environment step exceeded its deadline.

Initialization

Initialize self. See help(type(self)) for accurate signature.

exception nemo_rl.experience.failures.GenerationUnavailable#

Bases: nemo_rl.experience.failures.RolloutInfraFailure

The selected generation shard could not serve the request.

Initialization

Initialize self. See help(type(self)) for accurate signature.

exception nemo_rl.experience.failures.NoHealthyShards#

Bases: nemo_rl.experience.failures.RolloutInfraFailure

No generation shard is currently eligible to serve traffic.

Initialization

Initialize self. See help(type(self)) for accurate signature.

exception nemo_rl.experience.failures.GymTransportError#

Bases: nemo_rl.experience.failures.RolloutInfraFailure

NeMo-Gym failed at the transport layer rather than returning a rollout.

Initialization

Initialize self. See help(type(self)) for accurate signature.

exception nemo_rl.experience.failures.RolloutDataFailure#

Bases: nemo_rl.experience.failures.RolloutFailure

This prompt cannot be rolled out, and another shard would fail identically.

Usually a configuration problem — most often policy.max_total_sequence_length exceeding the generation engine’s max_model_len.

Initialization

Initialize self. See help(type(self)) for accurate signature.

exception nemo_rl.experience.failures.RolloutRedispatchExhausted#

Bases: RuntimeError

A prompt exhausted its infrastructure retry budget.

Deliberately not a :class:RolloutFailure: it is terminal for the run rather than for one attempt, so the per-attempt retry loop must not catch it. Reaching it means the prompt failed across repeated shard selections, which indicates fleet-wide failure rather than a bad prompt.

Initialization

Initialize self. See help(type(self)) for accurate signature.

exception nemo_rl.experience.failures.RolloutStall#

Bases: RuntimeError

Rollouts are in flight but none has committed within the watchdog deadline.

Initialization

Initialize self. See help(type(self)) for accurate signature.

class nemo_rl.experience.failures.FailureClass#

Bases: str, enum.Enum

Retry policy bucket for a failed rollout attempt.

Initialization

Initialize self. See help(type(self)) for accurate signature.

INFRA#: ‘infra’

DATA#: ‘data’

nemo_rl.experience.failures._INFRA_TYPES: Final[tuple[type[BaseException], ...]]#: ()

nemo_rl.experience.failures._INFRA_TYPE_NAMES: Final[frozenset[str]]#: ‘frozenset(…)’

nemo_rl.experience.failures.http_status_is_infra(status: int) → bool#

Whether an HTTP status means the endpoint is unwell rather than the request bad.

5xx and the retriable 4xx are infrastructure; any other 4xx describes the request itself, so another shard would answer it identically.

Public because NeMo-Gym has to make this same call on the raising side of a Ray actor boundary (see nemo_rl/environments/nemo_gym.py), and the two copies of the decision must not drift apart.

nemo_rl.experience.failures._http_status(exc: BaseException) → Optional[int]#: Return the HTTP status carried by an aiohttp response error, if this is one.

nemo_rl.experience.failures._is_infra(exc: BaseException) → bool#: Return whether this single exception (ignoring its cause chain) is infra.

nemo_rl.experience.failures.classify_rollout_failure( exc: BaseException, ) → nemo_rl.experience.failures.FailureClass#

Bucket a rollout exception into INFRA or DATA.

An explicit :class:RolloutDataFailure always wins, so callers that know a failure is prompt-specific can say so and not have it re-read as infrastructure. Otherwise the exception and its __cause__ chain are checked against the infrastructure table; anything unrecognized is DATA so that unexpected exceptions fail loudly instead of being retried into silence.

Parameters:: exc – The exception raised by a rollout attempt.
Returns:: The :class:FailureClass governing this failure’s retry budget.