Failure observations

This page is for version Main (default). For other versions, use one of these documentation indexes: - Main (default): https://docs.nvidia.com/nemo/gym/main/llms.txt - 0.7.0: https://docs.nvidia.com/nemo/gym/v0.7.0/llms.txt - 0.6.0: https://docs.nvidia.com/nemo/gym/v0.6.0/llms.txt - 0.5.1: https://docs.nvidia.com/nemo/gym/v0.5.1/llms.txt - 0.5.0: https://docs.nvidia.com/nemo/gym/v0.5.0/llms.txt - 0.4.0: https://docs.nvidia.com/nemo/gym/v0.4.0/llms.txt - 0.3.0: https://docs.nvidia.com/nemo/gym/v0.3.0/llms.txt - 0.2.1: https://docs.nvidia.com/nemo/gym/v0.2.1/llms.txt

Gym records an observed failure as one NgFailureEvent exception. The same object is raised and serialized with to_record(); from_record() reconstructs it. It names the boundary that observed it (model, harness, agent, resources_server, environment or collection), its kind, stage, HTTP status and the event ID of its explicit cause. Existing handlers, outcome policy, retry budgets and scoring remain in control.

The flow for one failure:

1. A boundary raises NgFailureEvent(...) from error. The event ID is assigned here.
2. failure_report_for turns the explicit cause chain (raise ... from) into a FailureReport: failure_id names the current failure, cause_id links causes. Implicit Python context is not a cause.
3. The server middleware sends the report in the X-NG-Failure-Report reply header, on success and failure replies, with the failures observed while serving the request (for example a model attempt that a retry recovered).
4. The caller’s raise_boundary_failure raises the reconstructed remote failure, preserving event IDs and explicit causes. Its http_error keeps this hop’s status and body separately. raise_for_status remains the HTTP exception compatibility path for callers whose retry or routing logic depends on ClientResponseError.
5. Collection saves the report beside the existing result or failure record (_ng_failure_events, _ng_failure_id).

An environment uses handled_episode_failure(...) with event fields, its local terminal decision and separate partial evidence. It returns an NgFailureEvent; the response adapter constructs EpisodeFailure after cleanup, including late observations and trajectory evidence. Diagnostics never borrow a cause’s kind. The v1 EpisodeFailure keeps its existing four fields; failure_report travels beside it in the response.

Pickle and cloudpickle retain canonical Gym causes, captured IDs and selected SDK facts. Foreign exceptions do not become boundary-less events: their type, bounded selected message and status go in the nearest Gym event’s foreign_error. Gym causes below a foreign intermediary remain linked. Exception type names stay data; decoding never imports them. Every event requires an explicit boundary and a registered or namespaced kind, including recovered reports.

There are no boundary-specific exception subclasses. Catch NgFailureEvent and select by boundary, failure_kind and, where needed, operation. A judge call records judge_failed above its provider cause; standalone parser JudgeErrors remain supported by the judge failsafe.

The optional foreign_error field holds diagnostics that an adapter selected from an SDK error: its type, a scalar message and its HTTP status. SDK objects, request bodies and response payloads are not serialized. A long message keeps a visible [truncated] marker.

aiohttp rejects a reply header line over 8190 bytes. A report over 4096 bytes is therefore not sent: the header carries {"omitted": true}, every hop above keeps the flag, and Collection saves _ng_failure_events_omitted. The real status and body still arrive. This diagnostic loss does not change the task outcome. Old records with no failure report keep their meaning. The final saved-record migration remains a separate change.