nemo_gym.orchestration.executors.judge_profile

The execution profile of a judge-only job that grades a run’s saved answers again.

A run that served its policy on GPUs has no allocation a judge-only job could reuse: its node pools are GPU pools and its services are the policy’s. The profile derived here keeps what grading needs from the saved submission config, the driver with its env, mounts, container and pinned install, so the run’s own judge and verifier servers start as they did, and replaces what grading does not need. It is one of two kinds.

A run that serves nothing beside its policy gets the CPU profile: no service, no policy model, one node of a CPU partition and a bounded walltime. The partition comes from the saved config when it declares a CPU node pool (gpus_per_node 0), else from the launcher’s reader placement (NEMO_GYM_REMOTE_READER_SRUN, the CPU partition the heavy readers already run on), else from --judge-partition; without one the profile is refused, naming the three. The run’s QOS belongs to its GPU allocation and does not follow (a GPU QOS can refuse a job without GPUs): the judge job takes --judge-qos, else the saved CPU pool’s own QOS, else the QOS the reader placement names (-q/--qos) for that partition, else the partition’s default.

A run that serves its judge as a service of its own, on a node pool of its own beside the policy’s (a vLLM judge its judge server reaches through GYM_POOL_*_NODES), gets the served-judge profile: those services on their saved node pools, with their saved shape and partition, and no policy service, policy model or policy pool. It is a GPU job like the run, so it keeps the run’s job-level QOS; --judge-partition and --judge-qos name a CPU placement and are not used (a launcher that passes them on every recovery would put the GPU pool on a CPU partition or QOS). A service that runs on the policy’s own nodes has no served-judge profile: the judge job does not start the policy’s allocation.

Either way the walltime is the run’s segment walltime bounded by the partition’s MaxTime, or a default when neither is known.

Module Contents

Classes

Name	Description
`JudgeExecutionProfile`	Where and for how long a judge-only job of a run grades its saved answers again, and what it starts.
`JudgeProfileUnsupported`	No judge execution profile can be derived from the submission: the message says why and what would allow

Functions

Name	Description
`_reader_step_option`	-
`_service_pools`	The node pools a saved service runs on; None for a part that takes the whole allocation (no `node_pool`),
`_slurm_duration`	-
`bound_judge_walltime`	The judge job’s walltime: the run’s segment walltime `segment`, bounded by the partition’s MaxTime in
`choose_judge_partition`	The CPU partition a judge-only job of this submission queues on, and where the name came from.
`is_cpu_only_submission`	Whether the submission already runs on CPU nodes only: no service, no policy model, and every node pool with
`judge_execution_profile`	The profile of a judge-only job of this submission on `partition` (see `choose_judge_partition`), with the
`judge_job_config`	The submission config of a judge-only job of `benchmark`: the run’s driver (its env, mounts, container and
`reader_step_partition`	The partition the reader step’s srun arguments (`NEMO_GYM_REMOTE_READER_SRUN`) name, if any.
`reader_step_qos`	The QOS the reader step’s srun arguments (`NEMO_GYM_REMOTE_READER_SRUN`) name, if any.
`served_judge_placement`	The saved services beside the policy a judge-only job of this submission starts, and the saved node pools
`served_judge_profile`	The served-judge profile of a submission that serves its judge beside its policy (see

Data

CPU_PLACEMENT_UNUSED

CPU_PROFILE

DEFAULT_JUDGE_WALLTIME

JUDGE_POOL_NAME

JUDGE_WALLTIME_REASON_DEFAULT

PARTITION_DEFAULT_QOS

SERVED_JUDGE_PROFILE

_ALLOCATION_SHAPE_DIRECTIVES

_NO_PROFILE

_NO_SERVED_PROFILE

_SERVED_SHAPE_DIRECTIVES

API

class nemo_gym.orchestration.executors.judge_profile.JudgeExecutionProfile(
    partition: str,
    partition_source: str,
    walltime: str,
    walltime_bounds: tuple[str, ...],
    compute: nemo_gym.orchestration.api.SlurmComputeConfig,
    qos: str | None = None,
    qos_source: str = PARTITION_DEFAULT_QOS,
    kind: str = CPU_PROFILE,
    pools: tuple[str, ...] = (JUDGE_POOL_NAME,),
    services: tuple[str, ...] = (),
    unused: str | None = None
)

Dataclass

Where and for how long a judge-only job of a run grades its saved answers again, and what it starts.

compute

SlurmComputeConfig

kind

str = CPU_PROFILE

partition

str

partition_source

str

pools

tuple[str, ...] = (JUDGE_POOL_NAME,)

qos

str | None = None

qos_source

str = PARTITION_DEFAULT_QOS

services

tuple[str, ...] = ()

unused

str | None = None

walltime

str

walltime_bounds

tuple[str, ...]

nemo_gym.orchestration.executors.judge_profile.JudgeExecutionProfile.report() -> dict[str, typing.Any]

class nemo_gym.orchestration.executors.judge_profile.JudgeProfileUnsupported()

Bases: ValueError

No judge execution profile can be derived from the submission: the message says why and what would allow one.

nemo_gym.orchestration.executors.judge_profile._reader_step_option(
    args: str | None,
    short: str,
    long: str
) -> str | None

nemo_gym.orchestration.executors.judge_profile._service_pools(
    name: str,
    service: nemo_gym.orchestration.api.ServiceConfig
) -> set[str | None]

The node pools a saved service runs on; None for a part that takes the whole allocation (no node_pool), the driver’s node (a Ray cluster without node_pools) or the first node (a prefill/decode router).

nemo_gym.orchestration.executors.judge_profile._slurm_duration(
    seconds: int
) -> str

nemo_gym.orchestration.executors.judge_profile.bound_judge_walltime(
    segment: str | None,
    limit: dict[str, typing.Any] | None
) -> tuple[str, tuple[str, ...]]

The judge job’s walltime: the run’s segment walltime segment, bounded by the partition’s MaxTime in limit (as _partition_time_limit reports it), or the default when neither is known; with what bounded it.

nemo_gym.orchestration.executors.judge_profile.choose_judge_partition(
    config: nemo_gym.orchestration.api.SubmitConfig,
    cluster: str,
    partition: str | None = None,
    reader_srun: str | None = None
) -> tuple[str, str]

The CPU partition a judge-only job of this submission queues on, and where the name came from.

partition (--judge-partition) wins; else the first saved node pool with gpus_per_node 0 names it; else the reader step’s arguments (reader_srun, the value of NEMO_GYM_REMOTE_READER_SRUN). Only a submission that serves nothing beside its policy model takes the CPU profile; one that does takes the served-judge profile (served_judge_placement).

nemo_gym.orchestration.executors.judge_profile.is_cpu_only_submission(
    config: nemo_gym.orchestration.api.SubmitConfig
) -> bool

Whether the submission already runs on CPU nodes only: no service, no policy model, and every node pool with gpus_per_node 0. Such a run’s own compute is its judge execution profile.

nemo_gym.orchestration.executors.judge_profile.judge_execution_profile(
    config: nemo_gym.orchestration.api.SubmitConfig,
    cluster: str,
    partition: str,
    partition_source: str,
    limit: dict[str, typing.Any] | None,
    qos: str | None = None,
    reader_srun: str | None = None
) -> nemo_gym.orchestration.executors.judge_profile.JudgeExecutionProfile

The profile of a judge-only job of this submission on partition (see choose_judge_partition), with the partition’s MaxTime limit bounding the walltime (see bound_judge_walltime).

The derived compute keeps the account, forwarded env and job-level directives of the run, less the directives that shape the original allocation, request its GPUs or name its QOS; a saved CPU pool of the partition lends its own directives to the judge node. The job’s QOS is qos (--judge-qos), else the saved CPU pool’s, else the reader placement’s (reader_srun, the value of NEMO_GYM_REMOTE_READER_SRUN) when that placement names this partition or none, else the partition’s default.

nemo_gym.orchestration.executors.judge_profile.judge_job_config(
    config: nemo_gym.orchestration.api.SubmitConfig,
    cluster: str,
    benchmark: str,
    profile: nemo_gym.orchestration.executors.judge_profile.JudgeExecutionProfile,
    command: str,
    gym_ref: str | None = None
) -> nemo_gym.orchestration.api.SubmitConfig

The submission config of a judge-only job of benchmark: the run’s driver (its env, mounts, container and pinned install, at gym_ref when given) runs command in place of gym eval run, with no policy model, no preparation, no resume chain, on the profile’s compute. It starts the profile’s services, the saved judge services of a served-judge profile, and none of a CPU profile.

nemo_gym.orchestration.executors.judge_profile.reader_step_partition(
    args: str | None
) -> str | None

The partition the reader step’s srun arguments (NEMO_GYM_REMOTE_READER_SRUN) name, if any.

nemo_gym.orchestration.executors.judge_profile.reader_step_qos(
    args: str | None
) -> str | None

The QOS the reader step’s srun arguments (NEMO_GYM_REMOTE_READER_SRUN) name, if any.

nemo_gym.orchestration.executors.judge_profile.served_judge_placement(
    config: nemo_gym.orchestration.api.SubmitConfig,
    cluster: str
) -> tuple[tuple[str, ...], dict[str, nemo_gym.orchestration.api.NodePool]] | None

The saved services beside the policy a judge-only job of this submission starts, and the saved node pools they run on, in the saved order; None when it serves nothing beside its policy (the CPU profile).

A service on a node of the policy’s, its pool or the whole allocation, is refused, naming it: the judge job does not start the policy’s allocation, and a separate node pool for that service would allow it.

nemo_gym.orchestration.executors.judge_profile.served_judge_profile(
    config: nemo_gym.orchestration.api.SubmitConfig,
    cluster: str,
    limit: dict[str, typing.Any] | None,
    partition: str | None = None,
    qos: str | None = None
) -> nemo_gym.orchestration.executors.judge_profile.JudgeExecutionProfile

The served-judge profile of a submission that serves its judge beside its policy (see served_judge_placement), with the kept pools’ partition MaxTime limit bounding the walltime.

The derived compute is the run’s, with only the kept node pools in their saved shape (nodes, tasks, GPUs, partition and directives) and the bounded walltime; its account, forwarded env and job-level directives stay, less those sizing the original allocation, which the kept pools state for themselves. It is a GPU job like the run: the QOS stays the run’s job-level one, else the kept pools’, else the partition’s default. partition (--judge-partition) and qos (--judge-qos) name a CPU placement and are recorded as not used.

nemo_gym.orchestration.executors.judge_profile.CPU_PLACEMENT_UNUSED = '--judge-partition/--judge-qos apply to the CPU judge profile only; not used'

nemo_gym.orchestration.executors.judge_profile.CPU_PROFILE = 'cpu'

nemo_gym.orchestration.executors.judge_profile.DEFAULT_JUDGE_WALLTIME = '04:00:00'

nemo_gym.orchestration.executors.judge_profile.JUDGE_POOL_NAME = 'judge'

nemo_gym.orchestration.executors.judge_profile.JUDGE_WALLTIME_REASON_DEFAULT = f'the default judge walltime {DEFAULT_JUDGE_WALLTIME}: the run records no segmen...

nemo_gym.orchestration.executors.judge_profile.PARTITION_DEFAULT_QOS = "the partition's default"

nemo_gym.orchestration.executors.judge_profile.SERVED_JUDGE_PROFILE = 'served-judge'

nemo_gym.orchestration.executors.judge_profile._ALLOCATION_SHAPE_DIRECTIVES = frozenset({'partition', 'nodes', 'ntasks-per-node', 'gpus', 'gpus-per-node', 'gp...

nemo_gym.orchestration.executors.judge_profile._NO_PROFILE = 'this submission has no existing CPU-only judge execution profile and none can b...

nemo_gym.orchestration.executors.judge_profile._NO_SERVED_PROFILE = 'this submission has no served-judge execution profile'

nemo_gym.orchestration.executors.judge_profile._SERVED_SHAPE_DIRECTIVES = _ALLOCATION_SHAPE_DIRECTIVES - {'gres', 'qos'}