Skip to navigation

nemo_gym.orchestration.executors.slurm.judge_job

View as MarkdownOpen in Claude

Judge-only jobs: grade a run’s saved answers again on a CPU or served-judge profile.

Module Contents

Classes

NameDescription
JudgeJobsThe judge-only jobs of SlurmExecutor: grade a run’s saved answers again, never its policy.

Functions

NameDescription
_stoppedThe outcome of a judge-only job that was not queued: decision stopped with reason.

Data

_COLLECTION_ONLY_RUN_KEYS

_JOURNAL_JUDGE_JOB

API

class nemo_gym.orchestration.executors.slurm.judge_job.JudgeJobs()

The judge-only jobs of SlurmExecutor: grade a run’s saved answers again, never its policy.

Mixed into SlurmExecutor, whose recovery_evidence, _saved_config and installed_commit its methods use.

nemo_gym.orchestration.executors.slurm.judge_job.JudgeJobs._queue_judge_job(
benchmark: str,
judge_partition: str | None,
judge_qos: str | None = None,
repin: str | None,
repin_reason: str | None,
acknowledge_failed_judge_jobs: bool = False,
acknowledge_reason: str | None = None

Queue the judge-only job of a run that is not CPU-only, on the judge execution profile derived from its saved config (judge_profile), and journal it: the served-judge profile when the run serves services beside its policy (its judge on a node pool of its own), else the CPU profile.

The job’s command is the same gym eval reverify --judge-failed-only --settle-into-source as a CPU-only run’s, with the run’s saved settings, the policy’s --model-type so the run’s config composes as it did (the policy server it names is never called) and a receipt the job writes when it ends. Under the resubmit lock, no job of the run may wait or run in its job dir, no allocation of the chain may be live, and the saved sources, resolved config and chain accounting must be as the evidence saw them; then the script is staged beside the run’s and queued with --no-requeue, and its id, partition, walltime, rows and receipt path are appended to the job dir’s judge-job journal. The job is not a segment of the policy chain: nothing is added to .gym_job_chain and no retry or walltime budget is spent or needed. When the last JUDGE_JOB_FAILURE_BOUND judge-only jobs ended without settling a saved answer, nothing is queued and the stop names them and the way on: acknowledge_failed_judge_jobs with a nonempty acknowledge_reason appends one line (time, user, reason, those jobs’ ids) to the acknowledgement journal once the new job is accepted by the scheduler, so exactly one more job queues; the acknowledgement covers only the jobs it names.

nemo_gym.orchestration.executors.slurm.judge_job.JudgeJobs.reverify_judge_failures(
benchmark: str,
judge_partition: str | None = None,
judge_qos: str | None = None,
repin: str | None = None,
repin_reason: str | None = None,
acknowledge_failed_judge_jobs: bool = False,
acknowledge_reason: str | None = None

Queue a judge-only job that grades the run’s selected saved answers again, never the policy.

A CPU-only submission (no service, no policy model, CPU node pools only) runs it on its own compute with the chain’s remaining walltime allowance, as before. Any other submission, one that served its policy on GPUs, runs it on a judge execution profile derived from its saved config (_queue_judge_job): the run’s own judge and verifier servers on one node of a CPU partition (judge_partition, a saved CPU pool or the launcher’s reader placement, with judge_qos or a QOS derived for it), or, for a run that serves its judge on a node pool of its own, on that saved pool with those services started and the run’s QOS, for the run’s segment walltime bounded by the partition’s MaxTime, at the run’s pinned Gym commit or repin. acknowledge_failed_judge_jobs (with acknowledge_reason) lets one more judge-only job queue when the last ones ended without settling a saved answer (see _queue_judge_job). Either way the job settles through gym eval reverify --judge-failed-only --settle-into-source. This never starts a policy service or turns an unavailable judge into a model run.

nemo_gym.orchestration.executors.slurm.judge_job._stopped(
reason: str,
refusal: str | None = None

The outcome of a judge-only job that was not queued: decision stopped with reason.

nemo_gym.orchestration.executors.slurm.judge_job._COLLECTION_ONLY_RUN_KEYS = ('limit', 'output_jsonl_fpath', 'materialized_inputs_jsonl_fpath', 'resume_from_...
nemo_gym.orchestration.executors.slurm.judge_job._JOURNAL_JUDGE_JOB = "import json,sys; entry=json.loads(sys.argv[2]); entry['job_id']=sys.argv[3] or ...