nemo_gym.orchestration.executors.script_templates

Make body safe to embed inside a single-quoted bash -c '...' block.

POSIX shells do not nest single quotes: an inner quote ENDS the outer string rather than nesting in it. Every value that had to be quoted — a Hydra override containing a space, or any of the JSON blobs vLLM flags take (--hf-overrides '{"architectures":[...]}') — would otherwise break out of the block and word-split. Both failure modes have been observed on real submissions: Hydra rejecting +multistage.stages=[{num_tasks: on its own, and /usr/bin/env: Argument list too long from a multi-node vLLM command whose JSON flags reopened the quoting.

'"'"' is the standard end-quote / literal-quote / reopen-quote sequence. It leaves $VAR and $(( )) untouched, which matters: the inner shell is the one meant to expand them.

Render the srun entrypoint for the driver step.

When either gym_install or prepare is needed, wraps everything in a single bash -c so prepare and run happen in the same srun step and container.

command replaces the gym eval run invocation for a benchmark whose harness is not Gym’s own runner. It is emitted verbatim, so it may be a multi-line script; everything before it (the install, and prepare) is unchanged.

extras are optional-dependency groups installed with the checkout; the server venvs Gym builds copy their telemetry packages from this process, so the driver is where telemetry has to be installed for any server to have it. ray is always added: gym needs Ray to start a server that declares ray_enabled = True. An extra the checkout does not define is dropped with a notice, since uv errors on it.

record_install_failure writes driver_install_failed to the file named by $GYM_DRIVER_FAILURE_FILE when the install exits with a failure status (1 to 128; a signal is not an install failure), for the guard of the next segment. It replaces the EXIT trap, so it cannot be combined with cleanup.

private_uv gives the install its own uv binary inside its temporary directory (a copy of a working uv, else a fresh install there), keeps uv’s cache on the host’s local disk in this user’s own directory only, and gives the install its own managed Pythons. Concurrent installs on other hosts replace the shared ~/.local/bin/uv and cache entries on a network home, and a uv running from those files then dies with SIGBUS.

cleanup removes the temporary checkout ($GYM_SRC) through an EXIT trap when the step exits.

export_tail (see render_export_tail) takes the place of exec "$@", so the exports run in the same step and container as the run, where gym is installed.

Render the part of the driver script that runs gym eval run, then each export of its rollouts.

exports pairs an exporter name with its shell-quoted override arguments for EXPORT_STEP_MODULE. The run is a child, not exec’d, so the script can go on to the exports; TERM, which is what the driver gets at the time limit, is passed on to it, so an interrupted run behaves as it did when it was exec’d. The run’s exit code goes to <bench_dir>/eval.exit and is the script’s exit code: the exports are reported in their own result files and cannot change it. They run only after a successful run, so a partial run is never exported, and an export that already succeeded is not repeated (the export step decides that from its result file).

Render a bash array with each argument on its own line.

host is emitted verbatim so it may be a shell expansion: a service pinned to a node pool answers on that pool’s head, not on the node running this script.

Render an &&-chained command that installs git if missing, then clones and checks out ref.

dest is emitted verbatim so it may be a shell expression (the driver passes "$GYM_SRC/gym" to clone outside the job directory); omit it to clone into a directory named after the repo in the current one.

Render the Ray head/worker bootstrap that wraps a single vLLM instance’s TP/PP command so it spans multiple Slurm nodes.

Uses ray symmetric-run when available (Ray >= 2.50), which starts/joins a Ray cluster across every task and runs the entrypoint only on the elected head node. Containers with an older Ray pin fall back to manually starting head/worker Ray processes, keyed on Slurm’s per-node task rank ($SLURM_NODEID).

Render a command that reconstructs content at dest_path via a base64 round-trip.