> This page is for version 0.6.0.
> For other versions, use one of these documentation indexes:
> - Main (default): https://docs.nvidia.com/nemo/gym/main/llms.txt
> - 0.7.0: https://docs.nvidia.com/nemo/gym/v0.7.0/llms.txt
> - 0.6.0: https://docs.nvidia.com/nemo/gym/v0.6.0/llms.txt
> - 0.5.1: https://docs.nvidia.com/nemo/gym/v0.5.1/llms.txt
> - 0.5.0: https://docs.nvidia.com/nemo/gym/v0.5.0/llms.txt
> - 0.4.0: https://docs.nvidia.com/nemo/gym/v0.4.0/llms.txt
> - 0.3.0: https://docs.nvidia.com/nemo/gym/v0.3.0/llms.txt
> - 0.2.1: https://docs.nvidia.com/nemo/gym/v0.2.1/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/gym/llms.txt.

# nemo_gym.orchestration.executors.slurm.script

## Module Contents

### Functions

| Name                                                                                                                                              | Description                                                                                                   |
| ------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------- |
| [`_build_nodes`](#nemo_gym-orchestration-executors-slurm-script-_build_nodes)                                                                     | Nodes the service command is written for; a per-node server is a single-node command.                         |
| [`_build_ray_command`](#nemo_gym-orchestration-executors-slurm-script-_build_ray_command)                                                         | `ray start` for one node of a ray service: the head, or with `worker` a node joining it.                      |
| [`_build_router_command`](#nemo_gym-orchestration-executors-slurm-script-_build_router_command)                                                   | The vllm-router invocation fronting a prefill/decode pair.                                                    |
| [`_build_service_command`](#nemo_gym-orchestration-executors-slurm-script-_build_service_command)                                                 | -                                                                                                             |
| [`_build_vllm_command`](#nemo_gym-orchestration-executors-slurm-script-_build_vllm_command)                                                       | -                                                                                                             |
| [`_build_vllm_multi_instance_multi_node_command`](#nemo_gym-orchestration-executors-slurm-script-_build_vllm_multi_instance_multi_node_command)   | -                                                                                                             |
| [`_build_vllm_ray_command`](#nemo_gym-orchestration-executors-slurm-script-_build_vllm_ray_command)                                               | -                                                                                                             |
| [`_build_vllm_ray_serve_command`](#nemo_gym-orchestration-executors-slurm-script-_build_vllm_ray_serve_command)                                   | -                                                                                                             |
| [`_build_vllm_single_instance_multi_node_command`](#nemo_gym-orchestration-executors-slurm-script-_build_vllm_single_instance_multi_node_command) | -                                                                                                             |
| [`_collector_command`](#nemo_gym-orchestration-executors-slurm-script-_collector_command)                                                         | Run a node-local copy of the collector binary, then append its node-local jsonl into `&lt;job dir&gt;/otel/`. |
| [`_command_env`](#nemo_gym-orchestration-executors-slurm-script-_command_env)                                                                     | Environment a `command` benchmark gets in place of `gym eval run` arguments.                                  |
| [`_driver_node`](#nemo_gym-orchestration-executors-slurm-script-_driver_node)                                                                     | The node the driver runs on in a multi-node job: the policy's first node.                                     |
| [`_escape_for_double_quoted_bash`](#nemo_gym-orchestration-executors-slurm-script-_escape_for_double_quoted_bash)                                 | Escape text for safe embedding inside a double-quoted bash string ("...").                                    |
| [`_health_check_host`](#nemo_gym-orchestration-executors-slurm-script-_health_check_host)                                                         | Where this service answers its health probe: the node it runs on, as \_service_nodelist places it.            |
| [`_kv_transfer_flag`](#nemo_gym-orchestration-executors-slurm-script-_kv_transfer_flag)                                                           | The --kv-transfer-config for one tier of a prefill/decode pair.                                               |
| [`_nixl_pre_command`](#nemo_gym-orchestration-executors-slurm-script-_nixl_pre_command)                                                           | Exports NIXL needs before a tier's `vllm serve` starts.                                                       |
| [`_node_totals`](#nemo_gym-orchestration-executors-slurm-script-_node_totals)                                                                     | -                                                                                                             |
| [`_places_services`](#nemo_gym-orchestration-executors-slurm-script-_places_services)                                                             | Whether the script declares the allocation's host array to place services with.                               |
| [`_pool_directive`](#nemo_gym-orchestration-executors-slurm-script-_pool_directive)                                                               | The one value every pool agrees on for `attribute`.                                                           |
| [`_pool_head`](#nemo_gym-orchestration-executors-slurm-script-_pool_head)                                                                         | Shell expansion for the first hostname of the pool starting at `offset`.                                      |
| [`_pool_of`](#nemo_gym-orchestration-executors-slurm-script-_pool_of)                                                                             | -                                                                                                             |
| [`_pool_offsets`](#nemo_gym-orchestration-executors-slurm-script-_pool_offsets)                                                                   | Each pool's (first node index, node count) within the allocation.                                             |
| [`_probe_host`](#nemo_gym-orchestration-executors-slurm-script-_probe_host)                                                                       | How the batch script, which runs on the allocation's first node, reaches `node`.                              |
| [`_ray_steps`](#nemo_gym-orchestration-executors-slurm-script-_ray_steps)                                                                         | The srun step names of a ray service, as \_render_ray_service names them.                                     |
| [`_ray_worker_ranges`](#nemo_gym-orchestration-executors-slurm-script-_ray_worker_ranges)                                                         | Each spanned pool's worker nodes as (first index, count): every node but the head.                            |
| [`_render_collector_health_check`](#nemo_gym-orchestration-executors-slurm-script-_render_collector_health_check)                                 | -                                                                                                             |
| [`_render_collector_service`](#nemo_gym-orchestration-executors-slurm-script-_render_collector_service)                                           | The collector's srun step. Started before the model services so the scrape covers their                       |
| [`_render_collector_shutdown`](#nemo_gym-orchestration-executors-slurm-script-_render_collector_shutdown)                                         | Run after the driver: one more scrape interval so the final counters are seen, then a                         |
| [`_render_directives`](#nemo_gym-orchestration-executors-slurm-script-_render_directives)                                                         | -                                                                                                             |
| [`_render_lustre_md_stats_start`](#nemo_gym-orchestration-executors-slurm-script-_render_lustre_md_stats_start)                                   | Define the `lustre_md_stats` snapshot and take the `start` one, before any service starts.                    |
| [`_render_pool_directives`](#nemo_gym-orchestration-executors-slurm-script-_render_pool_directives)                                               | One set of directives for the whole allocation, not one per pool.                                             |
| [`_render_pool_nodes`](#nemo_gym-orchestration-executors-slurm-script-_render_pool_nodes)                                                         | Declare the allocation's hosts, and export each node pool's, for --nodelist.                                  |
| [`_render_ray_head_addresses`](#nemo_gym-orchestration-executors-slurm-script-_render_ray_head_addresses)                                         | Export every ray service's head address, and its worker hosts, from the nodes Slurm gave it.                  |
| [`_render_ray_service`](#nemo_gym-orchestration-executors-slurm-script-_render_ray_service)                                                       | A ray service's srun steps: the head beside the driver, then one step per spanned pool's workers.             |
| [`_render_service_command`](#nemo_gym-orchestration-executors-slurm-script-_render_service_command)                                               | -                                                                                                             |
| [`_render_watched_driver`](#nemo_gym-orchestration-executors-slurm-script-_render_watched_driver)                                                 | The driver step, ended with TERM if a service it depends on exits first.                                      |
| [`_resolve_env`](#nemo_gym-orchestration-executors-slurm-script-_resolve_env)                                                                     | Return an 'env K=V ...' prefix string (trailing space) scoped to a single command, or '' if empty.            |
| [`_service_nodelist`](#nemo_gym-orchestration-executors-slurm-script-_service_nodelist)                                                           | What a service's --nodelist names, as the shell variable it expands.                                          |
| [`_service_nodes`](#nemo_gym-orchestration-executors-slurm-script-_service_nodes)                                                                 | How many nodes this service actually runs on.                                                                 |
| [`_service_pre_command`](#nemo_gym-orchestration-executors-slurm-script-_service_pre_command)                                                     | The service's own pre_command, with any exports its role requires in front.                                   |
| [`_srun_nodes`](#nemo_gym-orchestration-executors-slurm-script-_srun_nodes)                                                                       | -                                                                                                             |
| [`_srun_ntasks`](#nemo_gym-orchestration-executors-slurm-script-_srun_ntasks)                                                                     | -                                                                                                             |
| [`_strip_headless_incompatible_flags`](#nemo_gym-orchestration-executors-slurm-script-_strip_headless_incompatible_flags)                         | -                                                                                                             |
| [`_validate_env_key`](#nemo_gym-orchestration-executors-slurm-script-_validate_env_key)                                                           | -                                                                                                             |
| [`_vllm_api_node`](#nemo_gym-orchestration-executors-slurm-script-_vllm_api_node)                                                                 | The allocation node whose vLLM API server a `localhost` endpoint would reach, or None when the                |
| [`_vllm_base_flags`](#nemo_gym-orchestration-executors-slurm-script-_vllm_base_flags)                                                             | -                                                                                                             |
| [`_vllm_spans_multiple_nodes`](#nemo_gym-orchestration-executors-slurm-script-_vllm_spans_multiple_nodes)                                         | -                                                                                                             |
| [`_with_default_capture_dir`](#nemo_gym-orchestration-executors-slurm-script-_with_default_capture_dir)                                           | Enable capture for submitted evaluations unless explicitly disabled.                                          |
| [`_with_resume_flag`](#nemo_gym-orchestration-executors-slurm-script-_with_resume_flag)                                                           | Auto-enable `gym eval run`'s own cache-based resume when the auto-resume chain is on.                         |
| [`_with_vllm_traces`](#nemo_gym-orchestration-executors-slurm-script-_with_vllm_traces)                                                           | The service with vLLM request tracing switched on, when `otel.vllm_traces` asks for it.                       |
| [`build_sbatch_script`](#nemo_gym-orchestration-executors-slurm-script-build_sbatch_script)                                                       | -                                                                                                             |
| [`log_path`](#nemo_gym-orchestration-executors-slurm-script-log_path)                                                                             | The log of srun step `step` in job `job_id`, relative to the job directory.                                   |
| [`planned_log_sources`](#nemo_gym-orchestration-executors-slurm-script-planned_log_sources)                                                       | The logs `build_sbatch_script` makes a job write: the driver's, then one per service step.                    |
| [`pool_nodes_var`](#nemo_gym-orchestration-executors-slurm-script-pool_nodes_var)                                                                 | The env var build_sbatch_script exports with a node pool's comma-separated hosts.                             |
| [`ray_head_address_var`](#nemo_gym-orchestration-executors-slurm-script-ray_head_address_var)                                                     | The env var build_sbatch_script exports with a ray service's head host:port.                                  |
| [`ray_workers_var`](#nemo_gym-orchestration-executors-slurm-script-ray_workers_var)                                                               | The env var build_sbatch_script exports with a ray service's worker hosts in one pool.                        |

### Data

[`COLLECTOR_LOCAL_DIR_MARKER`](#nemo_gym-orchestration-executors-slurm-script-COLLECTOR_LOCAL_DIR_MARKER)

[`DRIVER_STEP`](#nemo_gym-orchestration-executors-slurm-script-DRIVER_STEP)

[`JOB_ID_PLACEHOLDER`](#nemo_gym-orchestration-executors-slurm-script-JOB_ID_PLACEHOLDER)

[`LOG_DIR`](#nemo_gym-orchestration-executors-slurm-script-LOG_DIR)

[`NODE_LIST_PRELUDE`](#nemo_gym-orchestration-executors-slurm-script-NODE_LIST_PRELUDE)

[`_BUILDERS`](#nemo_gym-orchestration-executors-slurm-script-_BUILDERS)

[`_HEADLESS_INCOMPATIBLE_FLAG`](#nemo_gym-orchestration-executors-slurm-script-_HEADLESS_INCOMPATIBLE_FLAG)

[`_NODE_ARRAY`](#nemo_gym-orchestration-executors-slurm-script-_NODE_ARRAY)

[`_RAY_HEAD_ONLY_FLAG`](#nemo_gym-orchestration-executors-slurm-script-_RAY_HEAD_ONLY_FLAG)

[`_RAY_SERVE_GATEWAY_SOURCE_PATH`](#nemo_gym-orchestration-executors-slurm-script-_RAY_SERVE_GATEWAY_SOURCE_PATH)

[`_RUNTIME_JOB_ID`](#nemo_gym-orchestration-executors-slurm-script-_RUNTIME_JOB_ID)

[`_SCRIPT_TEMPLATE`](#nemo_gym-orchestration-executors-slurm-script-_SCRIPT_TEMPLATE)

[`_VALID_ENV_KEY`](#nemo_gym-orchestration-executors-slurm-script-_VALID_ENV_KEY)

[`_WATCHED_DRIVER_TEMPLATE`](#nemo_gym-orchestration-executors-slurm-script-_WATCHED_DRIVER_TEMPLATE)

### API

```python
nemo_gym.orchestration.executors.slurm.script._build_nodes(
    service: nemo_gym.orchestration.api.ServiceConfig,
    compute: nemo_gym.orchestration.api.SlurmComputeConfig,
    total_nodes: int
) -> int
```

Nodes the service command is written for; a per-node server is a single-node command.

```python
nemo_gym.orchestration.executors.slurm.script._build_ray_command(
    service: nemo_gym.orchestration.api.RayServiceConfig,
    pool: str | None = None,
    address_var: str | None = None,
    worker: bool = False
) -> str
```

`ray start` for one node of a ray service: the head, or with `worker` a node joining it.

`address_var` names the env var holding the head's host:port (see ray_head_address_var).

```python
nemo_gym.orchestration.executors.slurm.script._build_router_command(
    router: nemo_gym.orchestration.api.VllmPDServiceConfig,
    offsets: dict[str, tuple[int, int]]
) -> str
```

The vllm-router invocation fronting a prefill/decode pair.

Both tiers are addressed at their pool's head node, which is where each tier's
API rank runs (the remaining ranks in a tier are headless).

```python
nemo_gym.orchestration.executors.slurm.script._build_service_command(
    service: nemo_gym.orchestration.api.ServiceConfig,
    total_nodes: int,
    gpus_per_node_values: list[int],
    offsets: dict[str, tuple[int, int]] | None = None
) -> str
```

```python
nemo_gym.orchestration.executors.slurm.script._build_vllm_command(
    service: nemo_gym.orchestration.api.VllmServiceConfig
) -> str
```

```python
nemo_gym.orchestration.executors.slurm.script._build_vllm_multi_instance_multi_node_command(
    service: nemo_gym.orchestration.api.VllmServiceConfig,
    total_nodes: int,
    head: str = '"$HEAD_NODE_IP"'
) -> str
```

```python
nemo_gym.orchestration.executors.slurm.script._build_vllm_ray_command(
    service: nemo_gym.orchestration.api.VllmServiceConfig,
    total_nodes: int,
    head: str | None = None
) -> str
```

```python
nemo_gym.orchestration.executors.slurm.script._build_vllm_ray_serve_command(
    service: nemo_gym.orchestration.api.VllmServiceConfig,
    total_nodes: int,
    gpus_per_node_values: list[int]
) -> str
```

```python
nemo_gym.orchestration.executors.slurm.script._build_vllm_single_instance_multi_node_command(
    service: nemo_gym.orchestration.api.VllmServiceConfig,
    total_nodes: int
) -> str
```

```python
nemo_gym.orchestration.executors.slurm.script._collector_command(
    config: nemo_gym.orchestration.api.SubmitConfig,
    remote_bench_dir: pathlib.Path
) -> str
```

Run a node-local copy of the collector binary, then append its node-local jsonl into `<job dir>/otel/`.

The copy-out runs in the collector's own step, the one place that sees its node-local directory: the
collector may run on another node than the batch script, or in a container.

```python
nemo_gym.orchestration.executors.slurm.script._command_env(
    benchmark: nemo_gym.orchestration.api.BenchmarkRunConfig,
    remote_bench_dir: pathlib.Path
) -> dict[str, str]
```

Environment a `command` benchmark gets in place of `gym eval run` arguments.

Values are already literal here (driver.env prefixes were resolved at
validation time), so they are passed through as-is.

```python
nemo_gym.orchestration.executors.slurm.script._driver_node(
    config: nemo_gym.orchestration.api.SubmitConfig,
    compute: nemo_gym.orchestration.api.SlurmComputeConfig
) -> int
```

The node the driver runs on in a multi-node job: the policy's first node.

The driver reaches the policy on localhost. A multi-node policy serves its API
from node 0, and a pinned one from its pool's first node.

```python
nemo_gym.orchestration.executors.slurm.script._escape_for_double_quoted_bash(
    text: str
) -> str
```

Escape text for safe embedding inside a double-quoted bash string ("...").

```python
nemo_gym.orchestration.executors.slurm.script._health_check_host(
    service: nemo_gym.orchestration.api.ServiceConfig,
    config: nemo_gym.orchestration.api.SubmitConfig,
    compute: nemo_gym.orchestration.api.SlurmComputeConfig,
    driver_node: int | None,
    total_nodes: int
) -> str
```

Where this service answers its health probe: the node it runs on, as \_service_nodelist places it.

```python
nemo_gym.orchestration.executors.slurm.script._kv_transfer_flag(
    service: nemo_gym.orchestration.api.VllmServiceConfig
) -> str
```

The --kv-transfer-config for one tier of a prefill/decode pair.

Single-quoted because it is JSON: the value travels through the same
single-quoted `bash -c` block as everything else, and the escaping helper
handles it there.

```python
nemo_gym.orchestration.executors.slurm.script._nixl_pre_command(
    service: nemo_gym.orchestration.api.VllmServiceConfig
) -> str
```

Exports NIXL needs before a tier's `vllm serve` starts.

The side-channel host is the node's own hostname and can only be known on the
node, so it is a shell statement rather than an `env` entry.

```python
nemo_gym.orchestration.executors.slurm.script._node_totals(
    compute: nemo_gym.orchestration.api.SlurmComputeConfig
) -> tuple[int, int]
```

```python
nemo_gym.orchestration.executors.slurm.script._places_services(
    config: nemo_gym.orchestration.api.SubmitConfig,
    is_multi_node: bool
) -> bool
```

Whether the script declares the allocation's host array to place services with.

```python
nemo_gym.orchestration.executors.slurm.script._pool_directive(
    pools: dict[str, nemo_gym.orchestration.api.NodePool],
    attribute: str
) -> typing.Any
```

The one value every pool agrees on for `attribute`.

A plain sbatch job takes a single --partition/--ntasks-per-node/--gpus-per-node
for the whole allocation, so pools that disagree cannot both be honoured. Slurm
would silently apply whichever directive came last; say so instead.

```python
nemo_gym.orchestration.executors.slurm.script._pool_head(
    offset: int
) -> str
```

Shell expansion for the first hostname of the pool starting at `offset`.

```python
nemo_gym.orchestration.executors.slurm.script._pool_of(
    compute: nemo_gym.orchestration.api.SlurmComputeConfig,
    node: int
) -> str | None
```

```python
nemo_gym.orchestration.executors.slurm.script._pool_offsets(
    compute: nemo_gym.orchestration.api.SlurmComputeConfig
) -> dict[str, tuple[int, int]]
```

Each pool's (first node index, node count) within the allocation.

Pools are laid out contiguously in declaration order, which is the order the
single #SBATCH --nodes total is built from, so pool i owns the nodes after
every pool before it.

```python
nemo_gym.orchestration.executors.slurm.script._probe_host(
    node: int
) -> str
```

How the batch script, which runs on the allocation's first node, reaches `node`.

```python
nemo_gym.orchestration.executors.slurm.script._ray_steps(
    name: str,
    service: nemo_gym.orchestration.api.RayServiceConfig,
    config: nemo_gym.orchestration.api.SubmitConfig,
    compute: nemo_gym.orchestration.api.SlurmComputeConfig
) -> list[str]
```

The srun step names of a ray service, as \_render_ray_service names them.

```python
nemo_gym.orchestration.executors.slurm.script._ray_worker_ranges(
    service: nemo_gym.orchestration.api.RayServiceConfig,
    compute: nemo_gym.orchestration.api.SlurmComputeConfig,
    head_node: int
) -> dict[str, tuple[int, int]]
```

Each spanned pool's worker nodes as (first index, count): every node but the head.

The head is the driver's node, which is always the first node of some pool.

```python
nemo_gym.orchestration.executors.slurm.script._render_collector_health_check(
    config: nemo_gym.orchestration.api.SubmitConfig,
    driver_node: int | None = None
) -> str
```

```python
nemo_gym.orchestration.executors.slurm.script._render_collector_service(
    config: nemo_gym.orchestration.api.SubmitConfig,
    remote_bench_dir: pathlib.Path,
    is_multi_node: bool,
    driver_node: int | None = None,
    account: str | None = None
) -> str
```

The collector's srun step. Started before the model services so the scrape covers their
startup.

Runs on exactly one node, the batch host. That is the first node of the allocation, which is
also where the Ray prelude puts the head of a multi-node vLLM service and where the driver runs,
so `localhost:<port>` reaches the API server and the OTLP endpoints from the same node. In a
container the job directory is mounted for the config and the copied `otel/*.jsonl`; on the
node it is simply there, so no mounts or workdir are passed.

```python
nemo_gym.orchestration.executors.slurm.script._render_collector_shutdown(
    config: nemo_gym.orchestration.api.SubmitConfig,
    remote_bench_dir: pathlib.Path,
    driver_node: int | None = None,
    before_exit: str = ''
) -> str
```

Run after the driver: one more scrape interval so the final counters are seen, then a
graceful stop, keeping the driver's exit code. `before_exit` runs last, just before the exit.

The TERM goes to the collector process itself, matched by its unique `--config` path. Sent to
`srun` instead, TERM makes Slurm kill the step outright and INT is treated as a console
interrupt; neither reaches the collector, so its final batch and the copy of its jsonl would be lost.

```python
nemo_gym.orchestration.executors.slurm.script._render_directives(
    compute: nemo_gym.orchestration.api.SlurmComputeConfig,
    remote_bench_dir: pathlib.Path,
    benchmark_name: str
) -> str
```

```python
nemo_gym.orchestration.executors.slurm.script._render_lustre_md_stats_start(
    compute: nemo_gym.orchestration.api.SlurmComputeConfig,
    remote_bench_dir: pathlib.Path,
    total_nodes: int
) -> str
```

Define the `lustre_md_stats` snapshot and take the `start` one, before any service starts.

A multi-node job reads every node in one host step (no container: lctl is a host tool); one node reads in place.

```python
nemo_gym.orchestration.executors.slurm.script._render_pool_directives(
    pools: dict[str, nemo_gym.orchestration.api.NodePool]
) -> list[str]
```

One set of directives for the whole allocation, not one per pool.

Pools divide an allocation between services (see \_pool_offsets); they are not
separate Slurm requests. Emitting --nodes per pool made every pool but the last
a no-op, so a two-pool job asked for one pool's nodes while the rest of the
executor sized itself on the sum.

```python
nemo_gym.orchestration.executors.slurm.script._render_pool_nodes(
    config: nemo_gym.orchestration.api.SubmitConfig,
    compute: nemo_gym.orchestration.api.SlurmComputeConfig,
    is_multi_node: bool
) -> str
```

Declare the allocation's hosts, and export each node pool's, for --nodelist.

```python
nemo_gym.orchestration.executors.slurm.script._render_ray_head_addresses(
    config: nemo_gym.orchestration.api.SubmitConfig,
    compute: nemo_gym.orchestration.api.SlurmComputeConfig
) -> str
```

Export every ray service's head address, and its worker hosts, from the nodes Slurm gave it.

The head runs beside the driver. Workers and the driver join it without knowing,
when the config is written, which host the job will land on.

```python
nemo_gym.orchestration.executors.slurm.script._render_ray_service(
    name: str,
    service: nemo_gym.orchestration.api.RayServiceConfig,
    config: nemo_gym.orchestration.api.SubmitConfig,
    compute: nemo_gym.orchestration.api.SlurmComputeConfig,
    driver_node: int | None
) -> str
```

A ray service's srun steps: the head beside the driver, then one step per spanned pool's workers.

Every step uses the service's one container, env and mounts.

```python
nemo_gym.orchestration.executors.slurm.script._render_service_command(
    name: str,
    container: str | None,
    command: str,
    env: dict[str, str] | None = None,
    mounts: list[str] | None = None,
    nodes: int | None = None,
    ntasks: int | None = None,
    pre_command: str = '',
    workdir: str | None = None,
    single_node: bool = False,
    nodelist: str | None = None,
    end_step_on_task_exit: bool = True,
    account: str | None = None
) -> str
```

```python
nemo_gym.orchestration.executors.slurm.script._render_watched_driver(
    driver: str,
    services: list[str]
) -> str
```

The driver step, ended with TERM if a service it depends on exits first.

A dead service would otherwise leave the driver retrying it until the walltime. The job exits
non-zero instead, so the auto-resume chain reruns it with fresh services; TERM is what the
driver gets at the time limit, so its in-flight rollouts leave no row and run again on resume.
That applies while the evaluation runs. When the driver has exports it creates the marker named by
`GYM_EVAL_SUCCEEDED_MARKER` as soon as the evaluation succeeds, and from then on a lost service no
longer ends the driver: the exports do not use it, and a requeue would reload the model for nothing.
The driver's PID is kept in a lowercase variable, which no service's uppercase \<NAME>\_PID can overwrite.

```python
nemo_gym.orchestration.executors.slurm.script._resolve_env(
    env: dict[str, str]
) -> str
```

Return an 'env K=V ...' prefix string (trailing space) scoped to a single command, or '' if empty.

A `runtime:VAR` value (see resolve_env_dict in api.py) is emitted as an unquoted `K=$VAR`
shell reference instead of a literal, so it's resolved from the job's own environment when
the command actually runs on the compute node, rather than baked in at script-generation time.

```python
nemo_gym.orchestration.executors.slurm.script._service_nodelist(
    service: nemo_gym.orchestration.api.ServiceConfig,
    driver_node: int | None,
    total_nodes: int
) -> str | None
```

What a service's --nodelist names, as the shell variable it expands.

A pinned service names its pool. In a multi-node job an unpinned service that
fits on one node joins the driver, which reaches it on localhost; otherwise
Slurm could start it on any node.

```python
nemo_gym.orchestration.executors.slurm.script._service_nodes(
    service: nemo_gym.orchestration.api.ServiceConfig,
    compute: nemo_gym.orchestration.api.SlurmComputeConfig,
    total_nodes: int
) -> int
```

How many nodes this service actually runs on.

A service pinned to a pool sees only that pool, so a single-node pool inside a
ten-node job is a single-node deployment and must not be built as a multi-node
Ray one.

```python
nemo_gym.orchestration.executors.slurm.script._service_pre_command(
    service: nemo_gym.orchestration.api.ServiceConfig
) -> str
```

The service's own pre_command, with any exports its role requires in front.

```python
nemo_gym.orchestration.executors.slurm.script._srun_nodes(
    service: nemo_gym.orchestration.api.ServiceConfig,
    compute: nemo_gym.orchestration.api.SlurmComputeConfig,
    total_nodes: int
) -> int | None
```

```python
nemo_gym.orchestration.executors.slurm.script._srun_ntasks(
    service: nemo_gym.orchestration.api.ServiceConfig,
    compute: nemo_gym.orchestration.api.SlurmComputeConfig,
    total_nodes: int,
    total_ntasks: int
) -> int | None
```

```python
nemo_gym.orchestration.executors.slurm.script._strip_headless_incompatible_flags(
    cmd: str
) -> str
```

```python
nemo_gym.orchestration.executors.slurm.script._validate_env_key(
    key: str
) -> None
```

```python
nemo_gym.orchestration.executors.slurm.script._vllm_api_node(
    service: nemo_gym.orchestration.api.VllmServiceConfig,
    compute: nemo_gym.orchestration.api.SlurmComputeConfig,
    driver_node: int | None,
    total_nodes: int
) -> int | None
```

The allocation node whose vLLM API server a `localhost` endpoint would reach, or None when the
service runs an API on more than one node (one server per pool node, Ray Serve replicas across nodes).

Mirrors the placement the command builders and `_service_nodelist` produce: a pinned service
answers on its pool's first node, an unpinned multi-node service on node 0, and an unpinned
single-node service beside the driver.

```python
nemo_gym.orchestration.executors.slurm.script._vllm_base_flags(
    service: nemo_gym.orchestration.api.VllmServiceConfig
) -> str
```

```python
nemo_gym.orchestration.executors.slurm.script._vllm_spans_multiple_nodes(
    service: nemo_gym.orchestration.api.ServiceConfig,
    total_nodes: int
) -> bool
```

```python
nemo_gym.orchestration.executors.slurm.script._with_default_capture_dir(
    run: dict[str, typing.Any],
    remote_bench_dir: pathlib.Path
) -> dict[str, typing.Any]
```

Enable capture for submitted evaluations unless explicitly disabled.

Auto-derive model_call_capture_dir from this benchmark's own real output
directory when observability is on and the caller didn't set one.

Hydra interpolation resolves before remote_bench_dir exists (it's computed
here, in build_sbatch_script, well after SubmitConfig validation), so
there's no way for a YAML value to reference it -- this has to happen in
Python, once the real path is known. An explicit model_call_capture_dir in
run always wins over this default.

```python
nemo_gym.orchestration.executors.slurm.script._with_resume_flag(
    run: dict[str, typing.Any],
    resumable: bool
) -> dict[str, typing.Any]
```

Auto-enable `gym eval run`'s own cache-based resume when the auto-resume chain is on.

A no-op on the chain's cold (first) run -- rollout_collection.py falls back to a fresh run
when the cache files don't exist yet -- and a no-op if the caller already set the key.

```python
nemo_gym.orchestration.executors.slurm.script._with_vllm_traces(
    name: str,
    service: nemo_gym.orchestration.api.ServiceConfig,
    config: nemo_gym.orchestration.api.SubmitConfig,
    compute: nemo_gym.orchestration.api.SlurmComputeConfig,
    driver_node: int | None
) -> nemo_gym.orchestration.api.ServiceConfig
```

The service with vLLM request tracing switched on, when `otel.vllm_traces` asks for it.

The flag and env go through the service's own `extra_args` and `env`, so every command builder
(plain, multi-node Ray, data-parallel head, Ray Serve replicas, prefill/decode tiers) carries
them without knowing about tracing. A service's explicit env wins over the defaults here.

vLLM exports to `localhost`, so only a service whose single API server runs on the collector's
node is traced. A service answering on another node, or on several nodes, is left alone rather
than pointed at a collector it cannot reach.

```python
nemo_gym.orchestration.executors.slurm.script.build_sbatch_script(
    config: nemo_gym.orchestration.api.SubmitConfig,
    benchmark_name: str,
    benchmark: nemo_gym.orchestration.api.BenchmarkRunConfig,
    compute: nemo_gym.orchestration.api.SlurmComputeConfig,
    remote_bench_dir: pathlib.Path
) -> str
```

```python
nemo_gym.orchestration.executors.slurm.script.log_path(
    step: str,
    job_id: str
) -> str
```

The log of srun step `step` in job `job_id`, relative to the job directory.

A resumed job keeps the job directory and gets a new job id, so it keeps the previous job's logs.

```python
nemo_gym.orchestration.executors.slurm.script.planned_log_sources(
    config: nemo_gym.orchestration.api.SubmitConfig,
    compute: nemo_gym.orchestration.api.SlurmComputeConfig
) -> list[nemo_gym.orchestration.jobs.LogSource]
```

The logs `build_sbatch_script` makes a job write: the driver's, then one per service step.

Steps are named as the script names them: the telemetry collector when it is on, a ray service's
head and each of its worker steps, and every other service, including a vllm_pd service's tiers
and router, by its name in `deployed_services`. Locations keep the job id placeholder.

```python
nemo_gym.orchestration.executors.slurm.script.pool_nodes_var(
    pool: str
) -> str
```

The env var build_sbatch_script exports with a node pool's comma-separated hosts.

```python
nemo_gym.orchestration.executors.slurm.script.ray_head_address_var(
    ray_service: str
) -> str
```

The env var build_sbatch_script exports with a ray service's head host:port.

```python
nemo_gym.orchestration.executors.slurm.script.ray_workers_var(
    ray_service: str,
    pool: str
) -> str
```

The env var build_sbatch_script exports with a ray service's worker hosts in one pool.

```python
nemo_gym.orchestration.executors.slurm.script.COLLECTOR_LOCAL_DIR_MARKER = f'export {COLLECTOR_LOCAL_DIR_ENV}='
```

```python
nemo_gym.orchestration.executors.slurm.script.DRIVER_STEP = 'driver'
```

```python
nemo_gym.orchestration.executors.slurm.script.JOB_ID_PLACEHOLDER = '{job_id}'
```

```python
nemo_gym.orchestration.executors.slurm.script.LOG_DIR = 'logs'
```

```python
nemo_gym.orchestration.executors.slurm.script.NODE_LIST_PRELUDE = 'gym_nodes=($(scontrol show hostnames "$SLURM_JOB_NODELIST"))'
```

```python
nemo_gym.orchestration.executors.slurm.script._BUILDERS = {VllmServiceConfig: _build_vllm_command, VllmNodeServiceConfig: _build_vllm_comm...
```

```python
nemo_gym.orchestration.executors.slurm.script._HEADLESS_INCOMPATIBLE_FLAG = re.compile('\\s--(?:api-server-count|otlp-traces-endpoint)(?:[= ]\\S+)?')
```

```python
nemo_gym.orchestration.executors.slurm.script._NODE_ARRAY = NODE_LIST_PRELUDE
```

```python
nemo_gym.orchestration.executors.slurm.script._RAY_HEAD_ONLY_FLAG = re.compile('\\s--(?:port|redis-shard-ports|include-dashboard)(?:[= ]\\S+)?')
```

```python
nemo_gym.orchestration.executors.slurm.script._RAY_SERVE_GATEWAY_SOURCE_PATH = Path(__file__).resolve().parents[2] / 'ray_serve_gateway.py'
```

```python
nemo_gym.orchestration.executors.slurm.script._RUNTIME_JOB_ID = '$SLURM_JOB_ID'
```

```python
nemo_gym.orchestration.executors.slurm.script._SCRIPT_TEMPLATE = '#!/bin/bash\n{directives}\n\n{resume_prologue}\n\n{ray_prelude}\n\n{service_com...
```

```python
nemo_gym.orchestration.executors.slurm.script._VALID_ENV_KEY = re.compile('^[A-Za-z_][A-Za-z0-9_]*$')
```

```python
nemo_gym.orchestration.executors.slurm.script._WATCHED_DRIVER_TEMPLATE = '{driver} &\n_gym_driver_pid=$!\n_gym_service_exited() {{\n    # A successful ru...
```