nemo_gym.orchestration.executors.slurm.judge_job
nemo_gym.orchestration.executors.slurm.judge_job
Judge-only jobs: grade a run’s saved answers again on a CPU or served-judge profile.
Module Contents
Classes
Functions
Data
API
The judge-only jobs of SlurmExecutor: grade a run’s saved answers again, never its policy.
Mixed into SlurmExecutor, whose recovery_evidence, _saved_config and installed_commit its
methods use.
Queue the judge-only job of a run that is not CPU-only, on the judge execution profile derived from its
saved config (judge_profile), and journal it: the served-judge profile when the run serves services beside
its policy (its judge on a node pool of its own), else the CPU profile.
The job’s command is the same gym eval reverify --judge-failed-only --settle-into-source as a CPU-only
run’s, with the run’s saved settings, the policy’s --model-type so the run’s config composes as it did
(the policy server it names is never called) and a receipt the job writes when it ends. Under the resubmit
lock, no job of the run may wait or run in its job dir, no allocation of the chain may be live, and the saved
sources, resolved config and chain accounting must be as the evidence saw them; then the script is staged
beside the run’s and queued with --no-requeue, and its id, partition, walltime, rows and receipt path are
appended to the job dir’s judge-job journal. The job is not a segment of the policy chain: nothing is added
to .gym_job_chain and no retry or walltime budget is spent or needed. When the last
JUDGE_JOB_FAILURE_BOUND judge-only jobs ended without settling a saved answer, nothing is queued and the
stop names them and the way on: acknowledge_failed_judge_jobs with a nonempty acknowledge_reason appends
one line (time, user, reason, those jobs’ ids) to the acknowledgement journal once the new job is accepted by the
scheduler, so exactly one more job queues; the acknowledgement covers only the jobs it names.
Queue a judge-only job that grades the run’s selected saved answers again, never the policy.
A CPU-only submission (no service, no policy model, CPU node pools only) runs it on its own compute with the
chain’s remaining walltime allowance, as before. Any other submission, one that served its policy on GPUs,
runs it on a judge execution profile derived from its saved config (_queue_judge_job): the run’s own judge
and verifier servers on one node of a CPU partition (judge_partition, a saved CPU pool or the launcher’s
reader placement, with judge_qos or a QOS derived for it), or, for a run that serves its judge on a node
pool of its own, on that saved pool with those services started and the run’s QOS, for the run’s segment
walltime bounded by the partition’s MaxTime, at the run’s pinned Gym commit or repin.
acknowledge_failed_judge_jobs (with acknowledge_reason) lets one more judge-only job queue when the last
ones ended without settling a saved answer (see _queue_judge_job). Either way the job settles through
gym eval reverify --judge-failed-only --settle-into-source. This never starts a policy service or turns an
unavailable judge into a model run.
The outcome of a judge-only job that was not queued: decision stopped with reason.