nemo_gym.verification.judge

Shared LLM-as-judge failure abstraction.

A failed judge call is a distinct outcome, not a wrong answer. Resources servers issue judge calls through call_judge, or wrap a bespoke call in reraise_judge_errors; judge_failsafe wraps every verify endpoint so a judge failure becomes a row tagged _ng_failure_class="judge_failed", which rollout_collection routes to <output>_failures.jsonl — excluded from the aggregate metrics and from the file-based re-aggregation (gym eval aggregate), and retained on resume without regenerating its answer. Judge-only recovery requires a supported reverification path; manifest-backed runs currently reject reverification.

Boundary: a failed call (transport/timeout/auth/HTTP) → judge_failed event → sidecar; a received-but-unparseable response is a legitimate wrong answer (let the parser score it, don’t raise), unless the benchmark explicitly treats an invalid judge verdict as an unusable grade. FrontierScience Research uses that exception for missing/invalid final numeric rubric verdicts: retry through the judge-failure sidecar rather than assign zero points to the model answer; ordinary resume does not regenerate that answer. Completion checks must still reject a partial evaluation. Empty output is a per-benchmark call, so servers differ.

Wrap the call, not the scoring around it: reraise_judge_errors relabels every exception it sees, so covering a whole verify() would misfile ordinary bugs (a KeyError in prompt assembly, say) as judge failures.

Module Contents

Classes

Name	Description
`JudgeError`	A parser-declared judge failure; judge_failsafe routes the row to the failures sidecar.
`JudgeGrading`	Judge facts of one reviewed LLM-judge grading execution; set only by `observe_llm_judge_grading`.

Functions

Name	Description
`_is_retryable_judge_error`	Whether another attempt can succeed: a 408, 429 or 5xx reply, a timeout or a lost connection.
`_judge_error`	The judge failure for a failed judge call; the caller raises it `from error`.
`_judge_http_status`	The status of the judge reply that failed, from this hop’s HTTP error; `None` when no reply came.
`_judge_retry_wait_s`	Full-jitter exponential backoff after failed attempt `attempt` (1-based).
`_reraise_judge_errors`	-
`call_judge`	POST to a judge model server and parse the reply; report failure as a `judge_failed` event.
`is_judge_failure`	Whether `error` is a judge failure: the `JudgeError` a parser raises, or a `judge_failed` event.
`is_judge_provider_stop`	Whether `error` is a `judge_provider_stopped` event.
`judge_call_failed`	The resources server boundary’s `judge_failed`; what was observed stays on the explicit cause.
`judge_failsafe`	Wrap verify() so a judge failure returns a sidecar-routed row (reward 0.0, the
`judge_provider_stopped`	The resources server boundary’s stop of a judge provider that refuses to grade (quota or credentials).
`normalize_math_judge_verdict`	Canonicalize whitespace inside A=B/A!=B markers without changing other text.
`record_parsed_judge_verdict`	Mark that the reviewed call site took its explicitly parsed-verdict branch for the active judge grading.
`reraise_judge_errors`	Await a judge call; preserve judge failures and wrap other errors as `judge_failed` events.

Data

JUDGE_CALL_MAX_ATTEMPTS

JUDGE_FAILURES

JUDGE_PROVIDER_STOP_KINDS

JUDGE_RETRY_BASE_WAIT_S

JUDGE_RETRY_MAX_WAIT_S

ResponseT

_RETRYABLE_JUDGE_STATUSES

_judge_grading

API

class nemo_gym.verification.judge.JudgeError()

Exception

Bases: Exception

A parser-declared judge failure; judge_failsafe routes the row to the failures sidecar.

class nemo_gym.verification.judge.JudgeGrading(
    dependencies: list[nemo_gym.trajectory.trajectory_record.ModelCallRef] = list(),
    unidentified_calls: int = 0,
    parsed_dependencies: set[int] = set()
)

Dataclass

Judge facts of one reviewed LLM-judge grading execution; set only by observe_llm_judge_grading.

dependencies

list[ModelCallRef] = field(default_factory=list)

parsed_dependencies

set[int] = field(default_factory=set)

unidentified_calls

int = 0

nemo_gym.verification.judge._is_retryable_judge_error(
    error: BaseException
) -> bool

Whether another attempt can succeed: a 408, 429 or 5xx reply, a timeout or a lost connection.

Other 4xx replies (auth, a bad request) and replies that do not parse fail the same way again.

nemo_gym.verification.judge._judge_error(
    error: Exception,
    operation_id: str | None = None,
    attempt_id: str | None = None
) -> nemo_gym.verification.judge.JudgeError | nemo_gym.core.typed_exceptions.NgFailureEvent

The judge failure for a failed judge call; the caller raises it from error.

A call that failed over HTTP, could not connect or timed out is a typed judge_failed. The kind is not derived from the error’s status: a policy that needs the status reads it on the cause. call_judge names each of its attempts with operation_id and attempt_id and records the reply’s status.

nemo_gym.verification.judge._judge_http_status(
    error: BaseException
) -> int | None

The status of the judge reply that failed, from this hop’s HTTP error; None when no reply came.

nemo_gym.verification.judge._judge_retry_wait_s(
    attempt: int
) -> float

Full-jitter exponential backoff after failed attempt attempt (1-based).

nemo_gym.verification.judge._reraise_judge_errors(
    coro: typing.Awaitable[typing.Any],
    operation_id: str | None = None,
    attempt_id: str | None = None
) -> typing.Any

async

nemo_gym.verification.judge.call_judge(
    server_client: nemo_gym.client.server_client.ServerClient,
    server_name: str,
    url_path: str,
    json: typing.Any,
    response_model: typing.Type[nemo_gym.verification.judge.ResponseT],
    timeout: aiohttp.ClientTimeout | None = None,
    max_attempts: int = JUDGE_CALL_MAX_ATTEMPTS
) -> nemo_gym.verification.judge.ResponseT

async

POST to a judge model server and parse the reply; report failure as a judge_failed event.

url_path is the judge’s API surface (/v1/responses or /v1/chat/completions), paired with the matching response_model. The status check comes before parsing so an auth or server error reports the HTTP failure itself, not a validation error against the error body. The enclosing verify execution supplies the invocation identity; all its judge calls and retries share that identity. Standalone calls do not invent a verifier execution.

A 408, 429 or 5xx reply, a timeout or a lost connection is tried again, up to max_attempts attempts (default JUDGE_CALL_MAX_ATTEMPTS; 1 for a caller that owns its retry loop) with full-jitter exponential backoff, before the failure masks the row; other 4xx replies and unparseable replies fail at once. Each failed attempt is observed as its own judge_failed event (one operation_id per call, the attempt in attempt_id, the reply’s http_status), so a row a retry scored and a row a final failure masked both keep every attempt.

nemo_gym.verification.judge.is_judge_failure(
    error: BaseException
) -> bool

Whether error is a judge failure: the JudgeError a parser raises, or a judge_failed event.

nemo_gym.verification.judge.is_judge_provider_stop(
    error: BaseException
) -> bool

Whether error is a judge_provider_stopped event.

nemo_gym.verification.judge.judge_call_failed(
    message: str,
    operation: str,
    sdk_error: nemo_gym.core.failure_records.SdkErrorDetails | None = None,
    operation_id: str | None = None,
    attempt_id: str | None = None,
    http_status: int | None = None
) -> nemo_gym.core.typed_exceptions.NgFailureEvent

The resources server boundary’s judge_failed; what was observed stays on the explicit cause.

nemo_gym.verification.judge.judge_failsafe(
    verify_fn: typing.Callable
) -> typing.Callable

Wrap verify() so a judge failure returns a sidecar-routed row (reward 0.0, the routing keys, the request’s response carried) instead of propagating. functools.wraps keeps verify’s signature so FastAPI injects the same params (body, and request for servers that take it); *args, **kwargs pass them straight through.

nemo_gym.verification.judge.judge_provider_stopped(
    message: str,
    failure_kind: str,
    operation: str
) -> nemo_gym.core.typed_exceptions.NgFailureEvent

The resources server boundary’s stop of a judge provider that refuses to grade (quota or credentials).

judge_failsafe routes it like a judge failure, so the row keeps its saved answer and recovery grades it again from that answer; the kind tells recovery and the run that no attempt can succeed until the operator acts.

nemo_gym.verification.judge.normalize_math_judge_verdict(
    text: str
) -> str

Canonicalize whitespace inside A=B/A!=B markers without changing other text.

nemo_gym.verification.judge.record_parsed_judge_verdict() -> None

Mark that the reviewed call site took its explicitly parsed-verdict branch for the active judge grading.

Call it only where the grade comes from a valid judge verdict; fallback, no-verdict and caught-error branches must not call it, so their grading stays unknown.

nemo_gym.verification.judge.reraise_judge_errors(
    coro: typing.Awaitable[typing.Any]
) -> typing.Any

async

Await a judge call; preserve judge failures and wrap other errors as judge_failed events.

Custom handlers catch JUDGE_FAILURES and select with is_judge_failure; an except JudgeError handler alone catches only standalone parser failures.

nemo_gym.verification.judge.JUDGE_CALL_MAX_ATTEMPTS = 3

Attempts of one call_judge call (the first one included) before its failure masks the row.

nemo_gym.verification.judge.JUDGE_FAILURES = (JudgeError, NgFailureEvent)

Types to catch around a judge call; narrow with is_judge_failure where other Gym failures can occur.

nemo_gym.verification.judge.JUDGE_PROVIDER_STOP_KINDS = (PROVIDER_QUOTA_EXHAUSTED, PROVIDER_CREDENTIALS_REJECTED)

The kinds of a judge provider that cannot grade until an operator acts: its quota resets or its key is fixed.

nemo_gym.verification.judge.JUDGE_RETRY_BASE_WAIT_S = 2.0

nemo_gym.verification.judge.JUDGE_RETRY_MAX_WAIT_S = 30.0

nemo_gym.verification.judge.ResponseT = TypeVar('ResponseT', bound=BaseModel)

nemo_gym.verification.judge._RETRYABLE_JUDGE_STATUSES = frozenset({408, 429})

nemo_gym.verification.judge._judge_grading: ContextVar[JudgeGrading | None] = ContextVar('judge_grading', default=None)