> This page is for version 0.7.0. > For other versions, use one of these documentation indexes: > - Main (default): https://docs.nvidia.com/nemo/gym/main/llms.txt > - 0.7.0: https://docs.nvidia.com/nemo/gym/v0.7.0/llms.txt > - 0.6.0: https://docs.nvidia.com/nemo/gym/v0.6.0/llms.txt > - 0.5.1: https://docs.nvidia.com/nemo/gym/v0.5.1/llms.txt > - 0.5.0: https://docs.nvidia.com/nemo/gym/v0.5.0/llms.txt > - 0.4.0: https://docs.nvidia.com/nemo/gym/v0.4.0/llms.txt > - 0.3.0: https://docs.nvidia.com/nemo/gym/v0.3.0/llms.txt > - 0.2.1: https://docs.nvidia.com/nemo/gym/v0.2.1/llms.txt > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/nemo/gym/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/gym/_mcp/server. # nemo_gym.verification.judge Shared LLM-as-judge failure abstraction. A failed judge call is a distinct outcome, not a wrong answer. Resources servers issue judge calls through `call_judge`, or wrap a bespoke call in `reraise_judge_errors`; `judge_failsafe` wraps every verify endpoint so a judge failure becomes a row tagged `_ng_failure_class="judge_failed"`, which rollout_collection routes to `_failures.jsonl` — excluded from the aggregate metrics and from the file-based re-aggregation (`gym eval aggregate`), and retained on resume without regenerating its answer. Judge-only recovery requires a supported reverification path; manifest-backed runs currently reject reverification. Boundary: a failed *call* (transport/timeout/auth/HTTP) → `judge_failed` event → sidecar; a *received-but-unparseable* response is a legitimate wrong answer (let the parser score it, don't raise), unless the benchmark explicitly treats an invalid judge verdict as an unusable grade. FrontierScience Research uses that exception for missing/invalid final numeric rubric verdicts: retry through the judge-failure sidecar rather than assign zero points to the model answer; ordinary resume does not regenerate that answer. Completion checks must still reject a partial evaluation. Empty output is a per-benchmark call, so servers differ. Wrap the call, not the scoring around it: `reraise_judge_errors` relabels every exception it sees, so covering a whole `verify()` would misfile ordinary bugs (a KeyError in prompt assembly, say) as judge failures. ## Module Contents ### Classes | Name | Description | | ----------------------------------------------------------- | ------------------------------------------------------------------------------------------------- | | [`JudgeError`](#nemo_gym-verification-judge-JudgeError) | A parser-declared judge failure; judge_failsafe routes the row to the failures sidecar. | | [`JudgeGrading`](#nemo_gym-verification-judge-JudgeGrading) | Judge facts of one reviewed LLM-judge grading execution; set only by `observe_llm_judge_grading`. | ### Functions | Name | Description | | ------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------- | | [`_is_retryable_judge_error`](#nemo_gym-verification-judge-_is_retryable_judge_error) | Whether another attempt can succeed: a 408, 429 or 5xx reply, a timeout or a lost connection. | | [`_judge_error`](#nemo_gym-verification-judge-_judge_error) | The judge failure for a failed judge call; the caller raises it `from error`. | | [`_judge_http_status`](#nemo_gym-verification-judge-_judge_http_status) | The status of the judge reply that failed, from this hop's HTTP error; `None` when no reply came. | | [`_judge_retry_wait_s`](#nemo_gym-verification-judge-_judge_retry_wait_s) | Full-jitter exponential backoff after failed attempt `attempt` (1-based). | | [`_reraise_judge_errors`](#nemo_gym-verification-judge-_reraise_judge_errors) | - | | [`call_judge`](#nemo_gym-verification-judge-call_judge) | POST to a judge model server and parse the reply; report failure as a `judge_failed` event. | | [`is_judge_failure`](#nemo_gym-verification-judge-is_judge_failure) | Whether `error` is a judge failure: the `JudgeError` a parser raises, or a `judge_failed` event. | | [`is_judge_provider_stop`](#nemo_gym-verification-judge-is_judge_provider_stop) | Whether `error` is a `judge_provider_stopped` event. | | [`judge_call_failed`](#nemo_gym-verification-judge-judge_call_failed) | The resources server boundary's `judge_failed`; what was observed stays on the explicit cause. | | [`judge_failsafe`](#nemo_gym-verification-judge-judge_failsafe) | Wrap verify() so a judge failure returns a sidecar-routed row (reward 0.0, the | | [`judge_provider_stopped`](#nemo_gym-verification-judge-judge_provider_stopped) | The resources server boundary's stop of a judge provider that refuses to grade (quota or credentials). | | [`normalize_math_judge_verdict`](#nemo_gym-verification-judge-normalize_math_judge_verdict) | Canonicalize whitespace inside A=B/A!=B markers without changing other text. | | [`record_parsed_judge_verdict`](#nemo_gym-verification-judge-record_parsed_judge_verdict) | Mark that the reviewed call site took its explicitly parsed-verdict branch for the active judge grading. | | [`reraise_judge_errors`](#nemo_gym-verification-judge-reraise_judge_errors) | Await a judge call; preserve judge failures and wrap other errors as `judge_failed` events. | ### Data [`JUDGE_CALL_MAX_ATTEMPTS`](#nemo_gym-verification-judge-JUDGE_CALL_MAX_ATTEMPTS) [`JUDGE_FAILURES`](#nemo_gym-verification-judge-JUDGE_FAILURES) [`JUDGE_PROVIDER_STOP_KINDS`](#nemo_gym-verification-judge-JUDGE_PROVIDER_STOP_KINDS) [`JUDGE_RETRY_BASE_WAIT_S`](#nemo_gym-verification-judge-JUDGE_RETRY_BASE_WAIT_S) [`JUDGE_RETRY_MAX_WAIT_S`](#nemo_gym-verification-judge-JUDGE_RETRY_MAX_WAIT_S) [`ResponseT`](#nemo_gym-verification-judge-ResponseT) [`_RETRYABLE_JUDGE_STATUSES`](#nemo_gym-verification-judge-_RETRYABLE_JUDGE_STATUSES) [`_judge_grading`](#nemo_gym-verification-judge-_judge_grading) ### API ```python class nemo_gym.verification.judge.JudgeError() ``` Exception **Bases:** `Exception` A parser-declared judge failure; judge_failsafe routes the row to the failures sidecar. ```python class nemo_gym.verification.judge.JudgeGrading( dependencies: list[nemo_gym.trajectory.trajectory_record.ModelCallRef] = list(), unidentified_calls: int = 0, parsed_dependencies: set[int] = set() ) ``` Dataclass Judge facts of one reviewed LLM-judge grading execution; set only by `observe_llm_judge_grading`. **`dependencies`** `list[ModelCallRef] = field(default_factory=list)` --- **`parsed_dependencies`** `set[int] = field(default_factory=set)` --- **`unidentified_calls`** `int = 0` --- ```python nemo_gym.verification.judge._is_retryable_judge_error( error: BaseException ) -> bool ``` Whether another attempt can succeed: a 408, 429 or 5xx reply, a timeout or a lost connection. Other 4xx replies (auth, a bad request) and replies that do not parse fail the same way again. ```python nemo_gym.verification.judge._judge_error( error: Exception, operation_id: str | None = None, attempt_id: str | None = None ) -> nemo_gym.verification.judge.JudgeError | nemo_gym.core.typed_exceptions.NgFailureEvent ``` The judge failure for a failed judge call; the caller raises it `from error`. A call that failed over HTTP, could not connect or timed out is a typed `judge_failed`. The kind is not derived from the error's status: a policy that needs the status reads it on the cause. `call_judge` names each of its attempts with `operation_id` and `attempt_id` and records the reply's status. ```python nemo_gym.verification.judge._judge_http_status( error: BaseException ) -> int | None ``` The status of the judge reply that failed, from this hop's HTTP error; `None` when no reply came. ```python nemo_gym.verification.judge._judge_retry_wait_s( attempt: int ) -> float ``` Full-jitter exponential backoff after failed attempt `attempt` (1-based). ```python nemo_gym.verification.judge._reraise_judge_errors( coro: typing.Awaitable[typing.Any], operation_id: str | None = None, attempt_id: str | None = None ) -> typing.Any ``` async ```python nemo_gym.verification.judge.call_judge( server_client: nemo_gym.client.server_client.ServerClient, server_name: str, url_path: str, json: typing.Any, response_model: typing.Type[nemo_gym.verification.judge.ResponseT], timeout: aiohttp.ClientTimeout | None = None, max_attempts: int = JUDGE_CALL_MAX_ATTEMPTS ) -> nemo_gym.verification.judge.ResponseT ``` async POST to a judge model server and parse the reply; report failure as a `judge_failed` event. `url_path` is the judge's API surface (`/v1/responses` or `/v1/chat/completions`), paired with the matching `response_model`. The status check comes before parsing so an auth or server error reports the HTTP failure itself, not a validation error against the error body. The enclosing verify execution supplies the invocation identity; all its judge calls and retries share that identity. Standalone calls do not invent a verifier execution. A 408, 429 or 5xx reply, a timeout or a lost connection is tried again, up to `max_attempts` attempts (default `JUDGE_CALL_MAX_ATTEMPTS`; 1 for a caller that owns its retry loop) with full-jitter exponential backoff, before the failure masks the row; other 4xx replies and unparseable replies fail at once. Each failed attempt is observed as its own `judge_failed` event (one `operation_id` per call, the attempt in `attempt_id`, the reply's `http_status`), so a row a retry scored and a row a final failure masked both keep every attempt. ```python nemo_gym.verification.judge.is_judge_failure( error: BaseException ) -> bool ``` Whether `error` is a judge failure: the `JudgeError` a parser raises, or a `judge_failed` event. ```python nemo_gym.verification.judge.is_judge_provider_stop( error: BaseException ) -> bool ``` Whether `error` is a `judge_provider_stopped` event. ```python nemo_gym.verification.judge.judge_call_failed( message: str, operation: str, sdk_error: nemo_gym.core.failure_records.SdkErrorDetails | None = None, operation_id: str | None = None, attempt_id: str | None = None, http_status: int | None = None ) -> nemo_gym.core.typed_exceptions.NgFailureEvent ``` The resources server boundary's `judge_failed`; what was observed stays on the explicit cause. ```python nemo_gym.verification.judge.judge_failsafe( verify_fn: typing.Callable ) -> typing.Callable ``` Wrap verify() so a judge failure returns a sidecar-routed row (reward 0.0, the routing keys, the request's `response` carried) instead of propagating. functools.wraps keeps verify's signature so FastAPI injects the same params (`body`, and `request` for servers that take it); `*args, **kwargs` pass them straight through. ```python nemo_gym.verification.judge.judge_provider_stopped( message: str, failure_kind: str, operation: str ) -> nemo_gym.core.typed_exceptions.NgFailureEvent ``` The resources server boundary's stop of a judge provider that refuses to grade (quota or credentials). `judge_failsafe` routes it like a judge failure, so the row keeps its saved answer and recovery grades it again from that answer; the kind tells recovery and the run that no attempt can succeed until the operator acts. ```python nemo_gym.verification.judge.normalize_math_judge_verdict( text: str ) -> str ``` Canonicalize whitespace inside A=B/A!=B markers without changing other text. ```python nemo_gym.verification.judge.record_parsed_judge_verdict() -> None ``` Mark that the reviewed call site took its explicitly parsed-verdict branch for the active judge grading. Call it only where the grade comes from a valid judge verdict; fallback, no-verdict and caught-error branches must not call it, so their grading stays unknown. ```python nemo_gym.verification.judge.reraise_judge_errors( coro: typing.Awaitable[typing.Any] ) -> typing.Any ``` async Await a judge call; preserve judge failures and wrap other errors as `judge_failed` events. Custom handlers catch `JUDGE_FAILURES` and select with `is_judge_failure`; an `except JudgeError` handler alone catches only standalone parser failures. ```python nemo_gym.verification.judge.JUDGE_CALL_MAX_ATTEMPTS = 3 ``` Attempts of one `call_judge` call (the first one included) before its failure masks the row. ```python nemo_gym.verification.judge.JUDGE_FAILURES = (JudgeError, NgFailureEvent) ``` Types to catch around a judge call; narrow with `is_judge_failure` where other Gym failures can occur. ```python nemo_gym.verification.judge.JUDGE_PROVIDER_STOP_KINDS = (PROVIDER_QUOTA_EXHAUSTED, PROVIDER_CREDENTIALS_REJECTED) ``` The kinds of a judge provider that cannot grade until an operator acts: its quota resets or its key is fixed. ```python nemo_gym.verification.judge.JUDGE_RETRY_BASE_WAIT_S = 2.0 ``` ```python nemo_gym.verification.judge.JUDGE_RETRY_MAX_WAIT_S = 30.0 ``` ```python nemo_gym.verification.judge.ResponseT = TypeVar('ResponseT', bound=BaseModel) ``` ```python nemo_gym.verification.judge._RETRYABLE_JUDGE_STATUSES = frozenset({408, 429}) ``` ```python nemo_gym.verification.judge._judge_grading: ContextVar[JudgeGrading | None] = ContextVar('judge_grading', default=None) ```