nemo_gym.orchestration.executors.kubernetes_script

Renders the Kubernetes Job manifest for one benchmark.

v1 scope: single node, one Job per benchmark, one pod. Model/ray services become native sidecar containers (initContainers with restartPolicy: Always, k8s >= 1.29) so kubelet enforces start-before-driver ordering and readiness via probes, instead of the hand-rolled bash health-check loop slurm_script.py needs on Slurm. The driver is the pod’s single regular container.

Module Contents

Functions

Name	Description
`_command_env`	Env a `command` benchmark gets in place of `gym eval run` arguments — mirrors
`_dns_label`	Kubernetes object/container names are RFC 1123 labels: lowercase alnum and ’-’ only, <=63
`_driver_command`	-
`_env_list`	-
`_gpu_resources`	-
`_manifest_write_commands`	Preamble lines that write the resolved config / job manifest onto the mounted PVC.
`_output_volume_mount`	-
`_probe`	-
`_reject_unsupported_mounts`	-
`_scale_quantity`	Scale a Kubernetes memory quantity (“32Gi”) by an integer factor (“64Gi” for factor=2).
`_sidecar_containers`	-
`_vllm_command`	-
`build_job_manifest`	-
`job_name`	-

Data

ARTIFACTS_DIRNAME

DRIVER_MEMORY_REQUEST

_READINESS_FAILURE_THRESHOLD

_READINESS_PERIOD_SECONDS

API

nemo_gym.orchestration.executors.kubernetes_script._command_env(
    benchmark: nemo_gym.orchestration.api.BenchmarkRunConfig,
    run_dir: str
) -> dict[str, str]

Env a command benchmark gets in place of gym eval run arguments — mirrors slurm_script.py’s _command_env, kept local to avoid coupling the two executor modules.

nemo_gym.orchestration.executors.kubernetes_script._dns_label(
    value: str
) -> str

Kubernetes object/container names are RFC 1123 labels: lowercase alnum and ’-’ only, <=63 chars, must start/end alphanumeric. Gym service and benchmark names allow ’.’, ’_’ (e.g. vllm_model, tau2.airline), which are valid there but rejected by the k8s API server, so every name that becomes a k8s name goes through this first.

nemo_gym.orchestration.executors.kubernetes_script._driver_command(
    config: nemo_gym.orchestration.api.SubmitConfig,
    benchmark_name: str,
    benchmark: nemo_gym.orchestration.api.BenchmarkRunConfig,
    run_dir: str,
    manifest_writes: list[str]
) -> list[str]

nemo_gym.orchestration.executors.kubernetes_script._env_list(
    env: dict[str, str]
) -> list[dict[str, str]]

nemo_gym.orchestration.executors.kubernetes_script._gpu_resources(
    gpu_count: int,
    memory_per_gpu: str
) -> dict[str, typing.Any]

nemo_gym.orchestration.executors.kubernetes_script._manifest_write_commands(
    resolved_config: str,
    manifest: str,
    run_dir: str
) -> list[str]

Preamble lines that write the resolved config / job manifest onto the mounted PVC.

Stands in for Connection.write_text (Slurm) / a ConfigMap (rejected — see kubernetes.py): the k8s Job name is chosen before submission, so both files’ content is fully known up front and can be embedded directly in the driver container’s own startup command.

nemo_gym.orchestration.executors.kubernetes_script._output_volume_mount(
    output_path: str
) -> dict[str, typing.Any]

nemo_gym.orchestration.executors.kubernetes_script._probe(
    path: str,
    port: int,
    period: int,
    failure_threshold: int
) -> dict[str, typing.Any]

nemo_gym.orchestration.executors.kubernetes_script._reject_unsupported_mounts(
    mounts: list[str]
) -> None

nemo_gym.orchestration.executors.kubernetes_script._scale_quantity(
    quantity: str,
    factor: int
) -> str

Scale a Kubernetes memory quantity (“32Gi”) by an integer factor (“64Gi” for factor=2).

nemo_gym.orchestration.executors.kubernetes_script._sidecar_containers(
    config: nemo_gym.orchestration.api.SubmitConfig,
    compute: nemo_gym.orchestration.api.KubernetesComputeConfig
) -> list[dict[str, typing.Any]]

nemo_gym.orchestration.executors.kubernetes_script._vllm_command(
    service: nemo_gym.orchestration.api.VllmServiceConfig,
    port: int
) -> list[str]

nemo_gym.orchestration.executors.kubernetes_script.build_job_manifest(
    config: nemo_gym.orchestration.api.SubmitConfig,
    benchmark_name: str,
    benchmark: nemo_gym.orchestration.api.BenchmarkRunConfig,
    compute: nemo_gym.orchestration.api.KubernetesComputeConfig,
    run_dir: pathlib.Path,
    name: str,
    gym_job_id: str,
    resolved_config: str,
    manifest: str
) -> dict[str, typing.Any]

nemo_gym.orchestration.executors.kubernetes_script.job_name(
    gym_job_id: str,
    benchmark_name: str
) -> str

nemo_gym.orchestration.executors.kubernetes_script.ARTIFACTS_DIRNAME = 'artifacts'

nemo_gym.orchestration.executors.kubernetes_script.DRIVER_MEMORY_REQUEST = '2Gi'

nemo_gym.orchestration.executors.kubernetes_script.GPU_RESOURCE_KEY = 'nvidia.com/gpu'

nemo_gym.orchestration.executors.kubernetes_script.LOGS_DIRNAME = 'logs'

nemo_gym.orchestration.executors.kubernetes_script.OUTPUT_VOLUME_NAME = 'gym-output'

nemo_gym.orchestration.executors.kubernetes_script.SHM_SIZE = '4Gi'

nemo_gym.orchestration.executors.kubernetes_script.SHM_VOLUME_NAME = 'dshm'

nemo_gym.orchestration.executors.kubernetes_script._QUANTITY_RE = re.compile('^(\\d+)([A-Za-z]*)$')

nemo_gym.orchestration.executors.kubernetes_script._READINESS_FAILURE_THRESHOLD = 3

nemo_gym.orchestration.executors.kubernetes_script._READINESS_PERIOD_SECONDS = 5