> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/gym/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/gym/_mcp/server.

# nemo_gym.judge

Shared LLM-as-judge failure abstraction.

A failed judge call is a distinct outcome, not a wrong answer. Resources servers
issue judge calls through `call_judge`, or wrap a bespoke call in
`reraise_judge_errors`; `judge_failsafe` wraps every verify endpoint so a
JudgeError becomes a row tagged `_ng_failure_class="judge_failed"`, which
rollout\_collection routes to `<output>_failures.jsonl` — excluded from the
aggregate metrics and from the file-based re-aggregation (`gym eval
aggregate`), and retryable on resume.

Boundary: a failed *call* (transport/timeout/auth/HTTP) → JudgeError → sidecar; a
*received-but-unparseable* response is a legitimate wrong answer (let the parser
score it, don't raise). Empty output is a per-benchmark call, so servers differ.

Wrap the call, not the scoring around it: `reraise_judge_errors` relabels every
exception it sees, so covering a whole `verify()` would misfile ordinary bugs
(a KeyError in prompt assembly, say) as judge failures.

## Module Contents

### Classes

| Name                                       | Description                                                                  |
| ------------------------------------------ | ---------------------------------------------------------------------------- |
| [`JudgeError`](#nemo_gym-judge-JudgeError) | A judge call failed; judge\_failsafe routes the row to the failures sidecar. |

### Functions

| Name                                                           | Description                                                                    |
| -------------------------------------------------------------- | ------------------------------------------------------------------------------ |
| [`call_judge`](#nemo_gym-judge-call_judge)                     | POST to a judge model server and parse the reply; raise JudgeError on failure. |
| [`judge_failsafe`](#nemo_gym-judge-judge_failsafe)             | Wrap verify() so a JudgeError returns a sidecar-routed row (reward 0.0, the    |
| [`reraise_judge_errors`](#nemo_gym-judge-reraise_judge_errors) | Await a judge call; re-raise any exception as JudgeError (recorded verbatim).  |

### Data

[`ResponseT`](#nemo_gym-judge-ResponseT)

### API

```python
class nemo_gym.judge.JudgeError()
```

Exception

**Bases:** `Exception`

A judge call failed; judge\_failsafe routes the row to the failures sidecar.

```python
nemo_gym.judge.call_judge(
    server_client: nemo_gym.server_utils.ServerClient,
    server_name: str,
    url_path: str,
    json: typing.Any,
    response_model: typing.Type[nemo_gym.judge.ResponseT]
) -> nemo_gym.judge.ResponseT
```

async

POST to a judge model server and parse the reply; raise JudgeError on failure.

`url_path` is the judge's API surface (`/v1/responses` or
`/v1/chat/completions`), paired with the matching `response_model`. The
status check comes before parsing so an auth or server error reports the HTTP
failure itself, not a validation error against the error body.

```python
nemo_gym.judge.judge_failsafe(
    verify_fn: typing.Callable
) -> typing.Callable
```

Wrap verify() so a JudgeError returns a sidecar-routed row (reward 0.0, the
routing keys, the request's `response` carried) instead of propagating.
functools.wraps keeps verify's signature so FastAPI injects the same params
(`body`, and `request` for servers that take it); `*args, **kwargs` pass
them straight through.

```python
nemo_gym.judge.reraise_judge_errors(
    coro: typing.Awaitable[typing.Any]
) -> typing.Any
```

async

Await a judge call; re-raise any exception as JudgeError (recorded verbatim).

```python
nemo_gym.judge.ResponseT = TypeVar('ResponseT', bound=BaseModel)
```