nemo_gym.orchestration.executors.script_templates

Module Contents

Functions

Name	Description
`bash_var`	-
`escape_for_single_quoted_block`	Make `body` safe to embed inside a single-quoted `bash -c '...'` block.
`render_driver_entrypoint`	Render the srun entrypoint for the driver step.
`render_gym_cmd`	Render a bash array with each argument on its own line.
`render_health_check`	`host` is emitted verbatim so it may be a shell expansion: a service pinned to
`render_ray_prelude`	-
`render_repo_checkout`	Render an &&-chained command that installs git if missing, then clones and checks out `ref`.
`render_vllm_ray_symmetric_run`	Render the Ray head/worker bootstrap that wraps a single vLLM instance’s TP/PP command so
`render_write_file_from_base64`	Render a command that reconstructs `content` at `dest_path` via a base64 round-trip.

Data

ENSURE_RAY_INSTALLED

RAY_SERVE_QUEUE_LENGTH_RESPONSE_DEADLINE_S

RAY_SYMMETRIC_RUN_CLUSTER_WAIT_TIMEOUT_S

_HEALTH_WAIT_MULTI

_RAY_PRELUDE

_VLLM_RAY_SYMMETRIC_RUN

API

nemo_gym.orchestration.executors.script_templates.bash_var(
    name: str
) -> str

nemo_gym.orchestration.executors.script_templates.escape_for_single_quoted_block(
    body: str
) -> str

Make body safe to embed inside a single-quoted bash -c '...' block.

POSIX shells do not nest single quotes: an inner quote ENDS the outer string rather than nesting in it. Every value that had to be quoted — a Hydra override containing a space, or any of the JSON blobs vLLM flags take (--hf-overrides '{"architectures":[...]}') — would otherwise break out of the block and word-split. Both failure modes have been observed on real submissions: Hydra rejecting +multistage.stages=[{num_tasks: on its own, and /usr/bin/env: Argument list too long from a multi-node vLLM command whose JSON flags reopened the quoting.

'"'"' is the standard end-quote / literal-quote / reopen-quote sequence. It leaves $VAR and $(( )) untouched, which matters: the inner shell is the one meant to expand them.

nemo_gym.orchestration.executors.script_templates.render_driver_entrypoint(
    repo: str | None,
    ref: str | None,
    prepare_cmd: str | None
) -> str

Render the srun entrypoint for the driver step.

When either gym_install or prepare is needed, wraps everything in a single bash -c so prepare and run happen in the same srun step and container.

nemo_gym.orchestration.executors.script_templates.render_gym_cmd(
    subcommand: str,
    var_name: str,
    args: list[str]
) -> str

Render a bash array with each argument on its own line.

nemo_gym.orchestration.executors.script_templates.render_health_check(
    name: str,
    port: int,
    path: str,
    timeout: int,
    host: str = 'localhost'
) -> str

host is emitted verbatim so it may be a shell expansion: a service pinned to a node pool answers on that pool’s head, not on the node running this script.

nemo_gym.orchestration.executors.script_templates.render_ray_prelude() -> str

nemo_gym.orchestration.executors.script_templates.render_repo_checkout(
    repo: str,
    ref: str,
    dest: str | None = None
) -> str

Render an &&-chained command that installs git if missing, then clones and checks out ref.

dest is emitted verbatim so it may be a shell expression (the driver passes "$GYM_SRC/gym" to clone outside the job directory); omit it to clone into a directory named after the repo in the current one.

nemo_gym.orchestration.executors.script_templates.render_vllm_ray_symmetric_run(
    inner_cmd: str,
    total_nodes: int,
    resource_flags: str
) -> str

Render the Ray head/worker bootstrap that wraps a single vLLM instance’s TP/PP command so it spans multiple Slurm nodes.

Uses ray symmetric-run when available (Ray >= 2.50), which starts/joins a Ray cluster across every task and runs the entrypoint only on the elected head node. Containers with an older Ray pin fall back to manually starting head/worker Ray processes, keyed on Slurm’s per-node task rank ($SLURM_NODEID).

nemo_gym.orchestration.executors.script_templates.render_write_file_from_base64(
    content: str,
    dest_path: str
) -> str

Render a command that reconstructs content at dest_path via a base64 round-trip.

nemo_gym.orchestration.executors.script_templates.ENSURE_RAY_INSTALLED = 'command -v ray >/dev/null 2>&1 || pip install -q "ray[default]"'

nemo_gym.orchestration.executors.script_templates.RAY_SERVE_QUEUE_LENGTH_RESPONSE_DEADLINE_S = 1.0

nemo_gym.orchestration.executors.script_templates.RAY_SYMMETRIC_RUN_CLUSTER_WAIT_TIMEOUT_S = 600

nemo_gym.orchestration.executors.script_templates._HEALTH_WAIT_MULTI = '# Wait for {name} (try multiple health endpoints)\necho "Waiting for {name} at ...

nemo_gym.orchestration.executors.script_templates._RAY_PRELUDE = '# Resolve the head node IP for multi-node vLLM services (spanning nodes via Ray...

nemo_gym.orchestration.executors.script_templates._VLLM_RAY_SYMMETRIC_RUN = "bash -lc '\n    export RAY_SYMMETRIC_RUN_CLUSTER_WAIT_TIMEOUT=" + str(RAY_SYMME...