> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/gym/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/gym/_mcp/server.

# nemo_gym.orchestration.executors.slurm_script

## Module Contents

### Functions

| Name                                                                                                                                              | Description                                                                                         |
| ------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------- |
| [`_build_nodes`](#nemo_gym-orchestration-executors-slurm_script-_build_nodes)                                                                     | Nodes the service command is written for; a per-node server is a single-node command.               |
| [`_build_ray_command`](#nemo_gym-orchestration-executors-slurm_script-_build_ray_command)                                                         | `ray start` for one node of a ray service: the head, or with `worker` a node joining it.            |
| [`_build_router_command`](#nemo_gym-orchestration-executors-slurm_script-_build_router_command)                                                   | The vllm-router invocation fronting a prefill/decode pair.                                          |
| [`_build_service_command`](#nemo_gym-orchestration-executors-slurm_script-_build_service_command)                                                 | -                                                                                                   |
| [`_build_vllm_command`](#nemo_gym-orchestration-executors-slurm_script-_build_vllm_command)                                                       | -                                                                                                   |
| [`_build_vllm_multi_instance_multi_node_command`](#nemo_gym-orchestration-executors-slurm_script-_build_vllm_multi_instance_multi_node_command)   | -                                                                                                   |
| [`_build_vllm_ray_command`](#nemo_gym-orchestration-executors-slurm_script-_build_vllm_ray_command)                                               | -                                                                                                   |
| [`_build_vllm_ray_serve_command`](#nemo_gym-orchestration-executors-slurm_script-_build_vllm_ray_serve_command)                                   | -                                                                                                   |
| [`_build_vllm_single_instance_multi_node_command`](#nemo_gym-orchestration-executors-slurm_script-_build_vllm_single_instance_multi_node_command) | -                                                                                                   |
| [`_command_env`](#nemo_gym-orchestration-executors-slurm_script-_command_env)                                                                     | Environment a `command` benchmark gets in place of `gym eval run` arguments.                        |
| [`_driver_node`](#nemo_gym-orchestration-executors-slurm_script-_driver_node)                                                                     | The node the driver runs on in a multi-node job: the policy's first node.                           |
| [`_escape_for_double_quoted_bash`](#nemo_gym-orchestration-executors-slurm_script-_escape_for_double_quoted_bash)                                 | Escape text for safe embedding inside a double-quoted bash string ("...").                          |
| [`_health_check_host`](#nemo_gym-orchestration-executors-slurm_script-_health_check_host)                                                         | Where this service answers its health probe: the node it runs on, as \_service\_nodelist places it. |
| [`_kv_transfer_flag`](#nemo_gym-orchestration-executors-slurm_script-_kv_transfer_flag)                                                           | The --kv-transfer-config for one tier of a prefill/decode pair.                                     |
| [`_nixl_pre_command`](#nemo_gym-orchestration-executors-slurm_script-_nixl_pre_command)                                                           | Exports NIXL needs before a tier's `vllm serve` starts.                                             |
| [`_node_totals`](#nemo_gym-orchestration-executors-slurm_script-_node_totals)                                                                     | -                                                                                                   |
| [`_places_services`](#nemo_gym-orchestration-executors-slurm_script-_places_services)                                                             | Whether the script declares the allocation's host array to place services with.                     |
| [`_pool_directive`](#nemo_gym-orchestration-executors-slurm_script-_pool_directive)                                                               | The one value every pool agrees on for `attribute`.                                                 |
| [`_pool_head`](#nemo_gym-orchestration-executors-slurm_script-_pool_head)                                                                         | Shell expansion for the first hostname of the pool starting at `offset`.                            |
| [`_pool_of`](#nemo_gym-orchestration-executors-slurm_script-_pool_of)                                                                             | -                                                                                                   |
| [`_pool_offsets`](#nemo_gym-orchestration-executors-slurm_script-_pool_offsets)                                                                   | Each pool's (first node index, node count) within the allocation.                                   |
| [`_probe_host`](#nemo_gym-orchestration-executors-slurm_script-_probe_host)                                                                       | How the batch script, which runs on the allocation's first node, reaches `node`.                    |
| [`_ray_worker_ranges`](#nemo_gym-orchestration-executors-slurm_script-_ray_worker_ranges)                                                         | Each spanned pool's worker nodes as (first index, count): every node but the head.                  |
| [`_render_collector_health_check`](#nemo_gym-orchestration-executors-slurm_script-_render_collector_health_check)                                 | -                                                                                                   |
| [`_render_collector_service`](#nemo_gym-orchestration-executors-slurm_script-_render_collector_service)                                           | The collector's srun step. Started before the model services so the scrape covers their             |
| [`_render_collector_shutdown`](#nemo_gym-orchestration-executors-slurm_script-_render_collector_shutdown)                                         | Run after the driver: one more scrape interval so the final counters are seen, then a               |
| [`_render_directives`](#nemo_gym-orchestration-executors-slurm_script-_render_directives)                                                         | -                                                                                                   |
| [`_render_pool_directives`](#nemo_gym-orchestration-executors-slurm_script-_render_pool_directives)                                               | One set of directives for the whole allocation, not one per pool.                                   |
| [`_render_pool_nodes`](#nemo_gym-orchestration-executors-slurm_script-_render_pool_nodes)                                                         | Declare the allocation's hosts, and export each node pool's, for --nodelist.                        |
| [`_render_ray_head_addresses`](#nemo_gym-orchestration-executors-slurm_script-_render_ray_head_addresses)                                         | Export every ray service's head address, and its worker hosts, from the nodes Slurm gave it.        |
| [`_render_ray_service`](#nemo_gym-orchestration-executors-slurm_script-_render_ray_service)                                                       | A ray service's srun steps: the head beside the driver, then one step per spanned pool's workers.   |
| [`_render_service_command`](#nemo_gym-orchestration-executors-slurm_script-_render_service_command)                                               | -                                                                                                   |
| [`_resolve_env`](#nemo_gym-orchestration-executors-slurm_script-_resolve_env)                                                                     | Return an 'env K=V ...' prefix string (trailing space) scoped to a single command, or '' if empty.  |
| [`_service_nodelist`](#nemo_gym-orchestration-executors-slurm_script-_service_nodelist)                                                           | What a service's --nodelist names, as the shell variable it expands.                                |
| [`_service_nodes`](#nemo_gym-orchestration-executors-slurm_script-_service_nodes)                                                                 | How many nodes this service actually runs on.                                                       |
| [`_service_pre_command`](#nemo_gym-orchestration-executors-slurm_script-_service_pre_command)                                                     | The service's own pre\_command, with any exports its role requires in front.                        |
| [`_srun_nodes`](#nemo_gym-orchestration-executors-slurm_script-_srun_nodes)                                                                       | -                                                                                                   |
| [`_srun_ntasks`](#nemo_gym-orchestration-executors-slurm_script-_srun_ntasks)                                                                     | -                                                                                                   |
| [`_strip_headless_incompatible_flags`](#nemo_gym-orchestration-executors-slurm_script-_strip_headless_incompatible_flags)                         | -                                                                                                   |
| [`_validate_env_key`](#nemo_gym-orchestration-executors-slurm_script-_validate_env_key)                                                           | -                                                                                                   |
| [`_vllm_base_flags`](#nemo_gym-orchestration-executors-slurm_script-_vllm_base_flags)                                                             | -                                                                                                   |
| [`_vllm_spans_multiple_nodes`](#nemo_gym-orchestration-executors-slurm_script-_vllm_spans_multiple_nodes)                                         | -                                                                                                   |
| [`_with_default_capture_dir`](#nemo_gym-orchestration-executors-slurm_script-_with_default_capture_dir)                                           | Enable capture for submitted evaluations unless explicitly disabled.                                |
| [`_with_resume_flag`](#nemo_gym-orchestration-executors-slurm_script-_with_resume_flag)                                                           | Auto-enable `gym eval run`'s own cache-based resume when the auto-resume chain is on.               |
| [`build_sbatch_script`](#nemo_gym-orchestration-executors-slurm_script-build_sbatch_script)                                                       | -                                                                                                   |
| [`pool_nodes_var`](#nemo_gym-orchestration-executors-slurm_script-pool_nodes_var)                                                                 | The env var build\_sbatch\_script exports with a node pool's comma-separated hosts.                 |
| [`ray_head_address_var`](#nemo_gym-orchestration-executors-slurm_script-ray_head_address_var)                                                     | The env var build\_sbatch\_script exports with a ray service's head host:port.                      |
| [`ray_workers_var`](#nemo_gym-orchestration-executors-slurm_script-ray_workers_var)                                                               | The env var build\_sbatch\_script exports with a ray service's worker hosts in one pool.            |

### Data

[`NODE_LIST_PRELUDE`](#nemo_gym-orchestration-executors-slurm_script-NODE_LIST_PRELUDE)

[`_BUILDERS`](#nemo_gym-orchestration-executors-slurm_script-_BUILDERS)

[`_HEADLESS_INCOMPATIBLE_FLAG`](#nemo_gym-orchestration-executors-slurm_script-_HEADLESS_INCOMPATIBLE_FLAG)

[`_NODE_ARRAY`](#nemo_gym-orchestration-executors-slurm_script-_NODE_ARRAY)

[`_RAY_HEAD_ONLY_FLAG`](#nemo_gym-orchestration-executors-slurm_script-_RAY_HEAD_ONLY_FLAG)

[`_RAY_SERVE_GATEWAY_SOURCE_PATH`](#nemo_gym-orchestration-executors-slurm_script-_RAY_SERVE_GATEWAY_SOURCE_PATH)

[`_SCRIPT_TEMPLATE`](#nemo_gym-orchestration-executors-slurm_script-_SCRIPT_TEMPLATE)

[`_VALID_ENV_KEY`](#nemo_gym-orchestration-executors-slurm_script-_VALID_ENV_KEY)

### API

```python
nemo_gym.orchestration.executors.slurm_script._build_nodes(
    service: nemo_gym.orchestration.api.ServiceConfig,
    compute: nemo_gym.orchestration.api.SlurmComputeConfig,
    total_nodes: int
) -> int
```

Nodes the service command is written for; a per-node server is a single-node command.

```python
nemo_gym.orchestration.executors.slurm_script._build_ray_command(
    service: nemo_gym.orchestration.api.RayServiceConfig,
    pool: str | None = None,
    address_var: str | None = None,
    worker: bool = False
) -> str
```

`ray start` for one node of a ray service: the head, or with `worker` a node joining it.

`address_var` names the env var holding the head's host:port (see ray\_head\_address\_var).

```python
nemo_gym.orchestration.executors.slurm_script._build_router_command(
    router: nemo_gym.orchestration.api.VllmPDServiceConfig,
    offsets: dict[str, tuple[int, int]]
) -> str
```

The vllm-router invocation fronting a prefill/decode pair.

Both tiers are addressed at their pool's head node, which is where each tier's
API rank runs (the remaining ranks in a tier are headless).

```python
nemo_gym.orchestration.executors.slurm_script._build_service_command(
    service: nemo_gym.orchestration.api.ServiceConfig,
    total_nodes: int,
    gpus_per_node_values: list[int],
    offsets: dict[str, tuple[int, int]] | None = None
) -> str
```

```python
nemo_gym.orchestration.executors.slurm_script._build_vllm_command(
    service: nemo_gym.orchestration.api.VllmServiceConfig
) -> str
```

```python
nemo_gym.orchestration.executors.slurm_script._build_vllm_multi_instance_multi_node_command(
    service: nemo_gym.orchestration.api.VllmServiceConfig,
    total_nodes: int,
    head: str = '"$HEAD_NODE_IP"'
) -> str
```

```python
nemo_gym.orchestration.executors.slurm_script._build_vllm_ray_command(
    service: nemo_gym.orchestration.api.VllmServiceConfig,
    total_nodes: int,
    head: str | None = None
) -> str
```

```python
nemo_gym.orchestration.executors.slurm_script._build_vllm_ray_serve_command(
    service: nemo_gym.orchestration.api.VllmServiceConfig,
    total_nodes: int,
    gpus_per_node_values: list[int]
) -> str
```

```python
nemo_gym.orchestration.executors.slurm_script._build_vllm_single_instance_multi_node_command(
    service: nemo_gym.orchestration.api.VllmServiceConfig,
    total_nodes: int
) -> str
```

```python
nemo_gym.orchestration.executors.slurm_script._command_env(
    benchmark: nemo_gym.orchestration.api.BenchmarkRunConfig,
    remote_bench_dir: pathlib.Path
) -> dict[str, str]
```

Environment a `command` benchmark gets in place of `gym eval run` arguments.

Values are already literal here (driver.env prefixes were resolved at
validation time), so they are passed through as-is.

```python
nemo_gym.orchestration.executors.slurm_script._driver_node(
    config: nemo_gym.orchestration.api.SubmitConfig,
    compute: nemo_gym.orchestration.api.SlurmComputeConfig
) -> int
```

The node the driver runs on in a multi-node job: the policy's first node.

The driver reaches the policy on localhost. A multi-node policy serves its API
from node 0, and a pinned one from its pool's first node.

```python
nemo_gym.orchestration.executors.slurm_script._escape_for_double_quoted_bash(
    text: str
) -> str
```

Escape text for safe embedding inside a double-quoted bash string ("...").

```python
nemo_gym.orchestration.executors.slurm_script._health_check_host(
    service: nemo_gym.orchestration.api.ServiceConfig,
    config: nemo_gym.orchestration.api.SubmitConfig,
    compute: nemo_gym.orchestration.api.SlurmComputeConfig,
    driver_node: int | None,
    total_nodes: int
) -> str
```

Where this service answers its health probe: the node it runs on, as \_service\_nodelist places it.

```python
nemo_gym.orchestration.executors.slurm_script._kv_transfer_flag(
    service: nemo_gym.orchestration.api.VllmServiceConfig
) -> str
```

The --kv-transfer-config for one tier of a prefill/decode pair.

Single-quoted because it is JSON: the value travels through the same
single-quoted `bash -c` block as everything else, and the escaping helper
handles it there.

```python
nemo_gym.orchestration.executors.slurm_script._nixl_pre_command(
    service: nemo_gym.orchestration.api.VllmServiceConfig
) -> str
```

Exports NIXL needs before a tier's `vllm serve` starts.

The side-channel host is the node's own hostname and can only be known on the
node, so it is a shell statement rather than an `env` entry.

```python
nemo_gym.orchestration.executors.slurm_script._node_totals(
    compute: nemo_gym.orchestration.api.SlurmComputeConfig
) -> tuple[int, int]
```

```python
nemo_gym.orchestration.executors.slurm_script._places_services(
    config: nemo_gym.orchestration.api.SubmitConfig,
    is_multi_node: bool
) -> bool
```

Whether the script declares the allocation's host array to place services with.

```python
nemo_gym.orchestration.executors.slurm_script._pool_directive(
    pools: dict[str, nemo_gym.orchestration.api.NodePool],
    attribute: str
) -> typing.Any
```

The one value every pool agrees on for `attribute`.

A plain sbatch job takes a single --partition/--ntasks-per-node/--gpus-per-node
for the whole allocation, so pools that disagree cannot both be honoured. Slurm
would silently apply whichever directive came last; say so instead.

```python
nemo_gym.orchestration.executors.slurm_script._pool_head(
    offset: int
) -> str
```

Shell expansion for the first hostname of the pool starting at `offset`.

```python
nemo_gym.orchestration.executors.slurm_script._pool_of(
    compute: nemo_gym.orchestration.api.SlurmComputeConfig,
    node: int
) -> str | None
```

```python
nemo_gym.orchestration.executors.slurm_script._pool_offsets(
    compute: nemo_gym.orchestration.api.SlurmComputeConfig
) -> dict[str, tuple[int, int]]
```

Each pool's (first node index, node count) within the allocation.

Pools are laid out contiguously in declaration order, which is the order the
single #SBATCH --nodes total is built from, so pool i owns the nodes after
every pool before it.

```python
nemo_gym.orchestration.executors.slurm_script._probe_host(
    node: int
) -> str
```

How the batch script, which runs on the allocation's first node, reaches `node`.

```python
nemo_gym.orchestration.executors.slurm_script._ray_worker_ranges(
    service: nemo_gym.orchestration.api.RayServiceConfig,
    compute: nemo_gym.orchestration.api.SlurmComputeConfig,
    head_node: int
) -> dict[str, tuple[int, int]]
```

Each spanned pool's worker nodes as (first index, count): every node but the head.

The head is the driver's node, which is always the first node of some pool.

```python
nemo_gym.orchestration.executors.slurm_script._render_collector_health_check(
    config: nemo_gym.orchestration.api.SubmitConfig,
    driver_node: int | None = None
) -> str
```

```python
nemo_gym.orchestration.executors.slurm_script._render_collector_service(
    config: nemo_gym.orchestration.api.SubmitConfig,
    remote_bench_dir: pathlib.Path,
    is_multi_node: bool,
    driver_node: int | None = None
) -> str
```

The collector's srun step. Started before the model services so the scrape covers their
startup.

Runs on exactly one node, the batch host. That is the first node of the allocation, which is
also where the Ray prelude puts the head of a multi-node vLLM service and where the driver runs,
so `localhost:<port>` reaches the API server and the OTLP endpoints from the same node. In a
container the job directory is mounted for the config and the local `otel/*.jsonl` output; on
the node it is simply there, so no mounts or workdir are passed.

```python
nemo_gym.orchestration.executors.slurm_script._render_collector_shutdown(
    config: nemo_gym.orchestration.api.SubmitConfig,
    remote_bench_dir: pathlib.Path
) -> str
```

Run after the driver: one more scrape interval so the final counters are seen, then a
graceful stop, keeping the driver's exit code.

The TERM goes to the collector process itself, matched by its unique `--config` path. Sent to
`srun` instead, TERM makes Slurm kill the step outright and INT is treated as a console
interrupt; neither reaches the collector, so its final batch would be lost.

```python
nemo_gym.orchestration.executors.slurm_script._render_directives(
    compute: nemo_gym.orchestration.api.SlurmComputeConfig,
    remote_bench_dir: pathlib.Path,
    benchmark_name: str
) -> str
```

```python
nemo_gym.orchestration.executors.slurm_script._render_pool_directives(
    pools: dict[str, nemo_gym.orchestration.api.NodePool]
) -> list[str]
```

One set of directives for the whole allocation, not one per pool.

Pools divide an allocation between services (see \_pool\_offsets); they are not
separate Slurm requests. Emitting --nodes per pool made every pool but the last
a no-op, so a two-pool job asked for one pool's nodes while the rest of the
executor sized itself on the sum.

```python
nemo_gym.orchestration.executors.slurm_script._render_pool_nodes(
    config: nemo_gym.orchestration.api.SubmitConfig,
    compute: nemo_gym.orchestration.api.SlurmComputeConfig,
    is_multi_node: bool
) -> str
```

Declare the allocation's hosts, and export each node pool's, for --nodelist.

```python
nemo_gym.orchestration.executors.slurm_script._render_ray_head_addresses(
    config: nemo_gym.orchestration.api.SubmitConfig,
    compute: nemo_gym.orchestration.api.SlurmComputeConfig
) -> str
```

Export every ray service's head address, and its worker hosts, from the nodes Slurm gave it.

The head runs beside the driver. Workers and the driver join it without knowing,
when the config is written, which host the job will land on.

```python
nemo_gym.orchestration.executors.slurm_script._render_ray_service(
    name: str,
    service: nemo_gym.orchestration.api.RayServiceConfig,
    config: nemo_gym.orchestration.api.SubmitConfig,
    compute: nemo_gym.orchestration.api.SlurmComputeConfig,
    driver_node: int | None
) -> str
```

A ray service's srun steps: the head beside the driver, then one step per spanned pool's workers.

Every step uses the service's one container, env and mounts.

```python
nemo_gym.orchestration.executors.slurm_script._render_service_command(
    name: str,
    container: str | None,
    command: str,
    env: dict[str, str] | None = None,
    mounts: list[str] | None = None,
    nodes: int | None = None,
    ntasks: int | None = None,
    pre_command: str = '',
    workdir: str | None = None,
    single_node: bool = False,
    nodelist: str | None = None
) -> str
```

```python
nemo_gym.orchestration.executors.slurm_script._resolve_env(
    env: dict[str, str]
) -> str
```

Return an 'env K=V ...' prefix string (trailing space) scoped to a single command, or '' if empty.

A `runtime:VAR` value (see resolve\_env\_dict in api.py) is emitted as an unquoted `K=$VAR`
shell reference instead of a literal, so it's resolved from the job's own environment when
the command actually runs on the compute node, rather than baked in at script-generation time.

```python
nemo_gym.orchestration.executors.slurm_script._service_nodelist(
    service: nemo_gym.orchestration.api.ServiceConfig,
    driver_node: int | None,
    total_nodes: int
) -> str | None
```

What a service's --nodelist names, as the shell variable it expands.

A pinned service names its pool. In a multi-node job an unpinned service that
fits on one node joins the driver, which reaches it on localhost; otherwise
Slurm could start it on any node.

```python
nemo_gym.orchestration.executors.slurm_script._service_nodes(
    service: nemo_gym.orchestration.api.ServiceConfig,
    compute: nemo_gym.orchestration.api.SlurmComputeConfig,
    total_nodes: int
) -> int
```

How many nodes this service actually runs on.

A service pinned to a pool sees only that pool, so a single-node pool inside a
ten-node job is a single-node deployment and must not be built as a multi-node
Ray one.

```python
nemo_gym.orchestration.executors.slurm_script._service_pre_command(
    service: nemo_gym.orchestration.api.ServiceConfig
) -> str
```

The service's own pre\_command, with any exports its role requires in front.

```python
nemo_gym.orchestration.executors.slurm_script._srun_nodes(
    service: nemo_gym.orchestration.api.ServiceConfig,
    compute: nemo_gym.orchestration.api.SlurmComputeConfig,
    total_nodes: int
) -> int | None
```

```python
nemo_gym.orchestration.executors.slurm_script._srun_ntasks(
    service: nemo_gym.orchestration.api.ServiceConfig,
    compute: nemo_gym.orchestration.api.SlurmComputeConfig,
    total_nodes: int,
    total_ntasks: int
) -> int | None
```

```python
nemo_gym.orchestration.executors.slurm_script._strip_headless_incompatible_flags(
    cmd: str
) -> str
```

```python
nemo_gym.orchestration.executors.slurm_script._validate_env_key(
    key: str
) -> None
```

```python
nemo_gym.orchestration.executors.slurm_script._vllm_base_flags(
    service: nemo_gym.orchestration.api.VllmServiceConfig
) -> str
```

```python
nemo_gym.orchestration.executors.slurm_script._vllm_spans_multiple_nodes(
    service: nemo_gym.orchestration.api.ServiceConfig,
    total_nodes: int
) -> bool
```

```python
nemo_gym.orchestration.executors.slurm_script._with_default_capture_dir(
    run: dict[str, typing.Any],
    remote_bench_dir: pathlib.Path
) -> dict[str, typing.Any]
```

Enable capture for submitted evaluations unless explicitly disabled.

Auto-derive model\_call\_capture\_dir from this benchmark's own real output
directory when observability is on and the caller didn't set one.

Hydra interpolation resolves before remote\_bench\_dir exists (it's computed
here, in build\_sbatch\_script, well after SubmitConfig validation), so
there's no way for a YAML value to reference it -- this has to happen in
Python, once the real path is known. An explicit model\_call\_capture\_dir in
run always wins over this default.

```python
nemo_gym.orchestration.executors.slurm_script._with_resume_flag(
    run: dict[str, typing.Any],
    resumable: bool
) -> dict[str, typing.Any]
```

Auto-enable `gym eval run`'s own cache-based resume when the auto-resume chain is on.

A no-op on the chain's cold (first) run -- rollout\_collection.py falls back to a fresh run
when the cache files don't exist yet -- and a no-op if the caller already set the key.

```python
nemo_gym.orchestration.executors.slurm_script.build_sbatch_script(
    config: nemo_gym.orchestration.api.SubmitConfig,
    benchmark_name: str,
    benchmark: nemo_gym.orchestration.api.BenchmarkRunConfig,
    compute: nemo_gym.orchestration.api.SlurmComputeConfig,
    remote_bench_dir: pathlib.Path
) -> str
```

```python
nemo_gym.orchestration.executors.slurm_script.pool_nodes_var(
    pool: str
) -> str
```

The env var build\_sbatch\_script exports with a node pool's comma-separated hosts.

```python
nemo_gym.orchestration.executors.slurm_script.ray_head_address_var(
    ray_service: str
) -> str
```

The env var build\_sbatch\_script exports with a ray service's head host:port.

```python
nemo_gym.orchestration.executors.slurm_script.ray_workers_var(
    ray_service: str,
    pool: str
) -> str
```

The env var build\_sbatch\_script exports with a ray service's worker hosts in one pool.

```python
nemo_gym.orchestration.executors.slurm_script.NODE_LIST_PRELUDE = 'gym_nodes=($(scontrol show hostnames "$SLURM_JOB_NODELIST"))'
```

```python
nemo_gym.orchestration.executors.slurm_script._BUILDERS = {VllmServiceConfig: _build_vllm_command, VllmPDTierConfig: _build_vllm_command, ...
```

```python
nemo_gym.orchestration.executors.slurm_script._HEADLESS_INCOMPATIBLE_FLAG = re.compile('\\s--api-server-count(?:[= ]\\S+)?')
```

```python
nemo_gym.orchestration.executors.slurm_script._NODE_ARRAY = NODE_LIST_PRELUDE
```

```python
nemo_gym.orchestration.executors.slurm_script._RAY_HEAD_ONLY_FLAG = re.compile('\\s--(?:port|redis-shard-ports|include-dashboard)(?:[= ]\\S+)?')
```

```python
nemo_gym.orchestration.executors.slurm_script._RAY_SERVE_GATEWAY_SOURCE_PATH = Path(__file__).resolve().parent.parent / 'ray_serve_gateway.py'
```

```python
nemo_gym.orchestration.executors.slurm_script._SCRIPT_TEMPLATE = '#!/bin/bash\n{directives}\n\n{resume_prologue}\n\n{ray_prelude}\n\n{service_com...
```

```python
nemo_gym.orchestration.executors.slurm_script._VALID_ENV_KEY = re.compile('^[A-Za-z_][A-Za-z0-9_]*$')
```