nemo_gym.judge | NeMo Gym

Shared LLM-as-judge failure abstraction.

A failed judge call is a distinct outcome, not a wrong answer. Resources servers issue judge calls through call_judge, or wrap a bespoke call in reraise_judge_errors; judge_failsafe wraps every verify endpoint so a JudgeError becomes a row tagged _ng_failure_class="judge_failed", which rollout_collection routes to <output>_failures.jsonl — excluded from the aggregate metrics and from the file-based re-aggregation (gym eval aggregate), and retryable on resume.

Boundary: a failed call (transport/timeout/auth/HTTP) → JudgeError → sidecar; a received-but-unparseable response is a legitimate wrong answer (let the parser score it, don’t raise). Empty output is a per-benchmark call, so servers differ.

Wrap the call, not the scoring around it: reraise_judge_errors relabels every exception it sees, so covering a whole verify() would misfile ordinary bugs (a KeyError in prompt assembly, say) as judge failures.

Module Contents

Classes

Name	Description
`JudgeError`	A judge call failed; judge_failsafe routes the row to the failures sidecar.

Functions

Name	Description
`call_judge`	POST to a judge model server and parse the reply; raise JudgeError on failure.
`judge_failsafe`	Wrap verify() so a JudgeError returns a sidecar-routed row (reward 0.0, the
`reraise_judge_errors`	Await a judge call; re-raise any exception as JudgeError (recorded verbatim).

Data

ResponseT

API

class nemo_gym.judge.JudgeError()

Exception

Bases: Exception

A judge call failed; judge_failsafe routes the row to the failures sidecar.

nemo_gym.judge.call_judge(
    server_client: nemo_gym.server_utils.ServerClient,
    server_name: str,
    url_path: str,
    json: typing.Any,
    response_model: typing.Type[nemo_gym.judge.ResponseT]
) -> nemo_gym.judge.ResponseT

async

POST to a judge model server and parse the reply; raise JudgeError on failure.

url_path is the judge’s API surface (/v1/responses or /v1/chat/completions), paired with the matching response_model. The status check comes before parsing so an auth or server error reports the HTTP failure itself, not a validation error against the error body.

nemo_gym.judge.judge_failsafe(
    verify_fn: typing.Callable
) -> typing.Callable

Wrap verify() so a JudgeError returns a sidecar-routed row (reward 0.0, the routing keys, the request’s response carried) instead of propagating. functools.wraps keeps verify’s signature so FastAPI injects the same params (body, and request for servers that take it); *args, **kwargs pass them straight through.

nemo_gym.judge.reraise_judge_errors(
    coro: typing.Awaitable[typing.Any]
) -> typing.Any

async

Await a judge call; re-raise any exception as JudgeError (recorded verbatim).

nemo_gym.judge.ResponseT = TypeVar('ResponseT', bound=BaseModel)