> This page is for version 0.2.1.
> For other versions, use one of these documentation indexes:
> - Main (default): https://docs.nvidia.com/nemo/gym/main/llms.txt
> - 0.7.0: https://docs.nvidia.com/nemo/gym/v0.7.0/llms.txt
> - 0.6.0: https://docs.nvidia.com/nemo/gym/v0.6.0/llms.txt
> - 0.5.1: https://docs.nvidia.com/nemo/gym/v0.5.1/llms.txt
> - 0.5.0: https://docs.nvidia.com/nemo/gym/v0.5.0/llms.txt
> - 0.4.0: https://docs.nvidia.com/nemo/gym/v0.4.0/llms.txt
> - 0.3.0: https://docs.nvidia.com/nemo/gym/v0.3.0/llms.txt
> - 0.2.1: https://docs.nvidia.com/nemo/gym/v0.2.1/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/gym/llms.txt.

# nemo_gym.orchestration.executors.slurm.judge_job

Judge-only jobs: grade a run's saved answers again on a CPU or served-judge profile.

## Module Contents

### Classes

| Name                                                                       | Description                                                                                  |
| -------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------- |
| [`JudgeJobs`](#nemo_gym-orchestration-executors-slurm-judge_job-JudgeJobs) | The judge-only jobs of `SlurmExecutor`: grade a run's saved answers again, never its policy. |

### Functions

| Name                                                                     | Description                                                                            |
| ------------------------------------------------------------------------ | -------------------------------------------------------------------------------------- |
| [`_stopped`](#nemo_gym-orchestration-executors-slurm-judge_job-_stopped) | The outcome of a judge-only job that was not queued: `decision` stopped with `reason`. |

### Data

[`_COLLECTION_ONLY_RUN_KEYS`](#nemo_gym-orchestration-executors-slurm-judge_job-_COLLECTION_ONLY_RUN_KEYS)

[`_JOURNAL_JUDGE_JOB`](#nemo_gym-orchestration-executors-slurm-judge_job-_JOURNAL_JUDGE_JOB)

### API

```python
class nemo_gym.orchestration.executors.slurm.judge_job.JudgeJobs()
```

The judge-only jobs of `SlurmExecutor`: grade a run's saved answers again, never its policy.

Mixed into `SlurmExecutor`, whose `recovery_evidence`, `_saved_config` and `installed_commit` its
methods use.

```python
nemo_gym.orchestration.executors.slurm.judge_job.JudgeJobs._queue_judge_job(
    conn: nemo_gym.orchestration.executors.connection.Connection,
    record: nemo_gym.orchestration.jobs.SubmissionRecord,
    benchmark: str,
    decision: nemo_gym.eval.recovery.decision.RecoveryDecision,
    evidence: nemo_gym.eval.recovery.evidence.CollectedEvidence,
    config: nemo_gym.orchestration.api.SubmitConfig,
    judge_partition: str | None,
    judge_qos: str | None = None,
    repin: str | None,
    repin_reason: str | None,
    acknowledge_failed_judge_jobs: bool = False,
    acknowledge_reason: str | None = None
) -> nemo_gym.eval.recovery.decision.RecoveryOutcome
```

Queue the judge-only job of a run that is not CPU-only, on the judge execution profile derived from its
saved config (`judge_profile`), and journal it: the served-judge profile when the run serves services beside
its policy (its judge on a node pool of its own), else the CPU profile.

The job's command is the same `gym eval reverify --judge-failed-only --settle-into-source` as a CPU-only
run's, with the run's saved settings, the policy's `--model-type` so the run's config composes as it did
(the policy server it names is never called) and a receipt the job writes when it ends. Under the resubmit
lock, no job of the run may wait or run in its job dir, no allocation of the chain may be live, and the saved
sources, resolved config and chain accounting must be as the evidence saw them; then the script is staged
beside the run's and queued with `--no-requeue`, and its id, partition, walltime, rows and receipt path are
appended to the job dir's judge-job journal. The job is not a segment of the policy chain: nothing is added
to `.gym_job_chain` and no retry or walltime budget is spent or needed. When the last
`JUDGE_JOB_FAILURE_BOUND` judge-only jobs ended without settling a saved answer, nothing is queued and the
stop names them and the way on: `acknowledge_failed_judge_jobs` with a nonempty `acknowledge_reason` appends
one line (time, user, reason, those jobs' ids) to the acknowledgement journal once the new job is accepted by the
scheduler, so exactly one more job queues; the acknowledgement covers only the jobs it names.

```python
nemo_gym.orchestration.executors.slurm.judge_job.JudgeJobs.reverify_judge_failures(
    record: nemo_gym.orchestration.jobs.SubmissionRecord,
    benchmark: str,
    decision: nemo_gym.eval.recovery.decision.RecoveryDecision,
    judge_partition: str | None = None,
    judge_qos: str | None = None,
    repin: str | None = None,
    repin_reason: str | None = None,
    acknowledge_failed_judge_jobs: bool = False,
    acknowledge_reason: str | None = None
) -> nemo_gym.eval.recovery.decision.RecoveryOutcome
```

Queue a judge-only job that grades the run's selected saved answers again, never the policy.

A CPU-only submission (no service, no policy model, CPU node pools only) runs it on its own compute with the
chain's remaining walltime allowance, as before. Any other submission, one that served its policy on GPUs,
runs it on a judge execution profile derived from its saved config (`_queue_judge_job`): the run's own judge
and verifier servers on one node of a CPU partition (`judge_partition`, a saved CPU pool or the launcher's
reader placement, with `judge_qos` or a QOS derived for it), or, for a run that serves its judge on a node
pool of its own, on that saved pool with those services started and the run's QOS, for the run's segment
walltime bounded by the partition's MaxTime, at the run's pinned Gym commit or `repin`.
`acknowledge_failed_judge_jobs` (with `acknowledge_reason`) lets one more judge-only job queue when the last
ones ended without settling a saved answer (see `_queue_judge_job`). Either way the job settles through
`gym eval reverify --judge-failed-only --settle-into-source`. This never starts a policy service or turns an
unavailable judge into a model run.

```python
nemo_gym.orchestration.executors.slurm.judge_job._stopped(
    decision: nemo_gym.eval.recovery.decision.RecoveryDecision,
    reason: str,
    refusal: str | None = None
) -> nemo_gym.eval.recovery.decision.RecoveryOutcome
```

The outcome of a judge-only job that was not queued: `decision` stopped with `reason`.

```python
nemo_gym.orchestration.executors.slurm.judge_job._COLLECTION_ONLY_RUN_KEYS = ('limit', 'output_jsonl_fpath', 'materialized_inputs_jsonl_fpath', 'resume_from_...
```

```python
nemo_gym.orchestration.executors.slurm.judge_job._JOURNAL_JUDGE_JOB = "import json,sys; entry=json.loads(sys.argv[2]); entry['job_id']=sys.argv[3] or ...
```