SLURM Deployment

Distribute evaluations across an HPC cluster using SLURM.

Quick start

1
# slurm_eval.yaml
2
services:
3
  model:
4
    type: vllm
5
    model: nvidia/Llama-3.1-70B-Instruct
6
    protocol: chat_completions
7
    tensor_parallel_size: 4
8
    port: 8000
9
    node_pool: compute
10
11
benchmarks:
12
  - name: gsm8k
13
    repeats: 8
14
    solver:
15
      type: simple
16
      service: model
17
18
cluster:
19
  type: slurm
20
  walltime: "02:00:00"
21
  node_pools:
22
    compute:
23
      partition: batch
24
      nodes: 1
25
      ntasks_per_node: 1
26
      gres: "gpu:4"

$
nel eval run slurm_eval.yaml

This generates an sbatch script, deploys vLLM on the allocated GPU node, and runs the benchmark.

Sharded evaluation

Split a large benchmark across multiple independent SLURM jobs:

1
# slurm_sharded.yaml
2
services:
3
  model:
4
    type: vllm
5
    model: nvidia/Llama-3.1-70B-Instruct
6
    protocol: chat_completions
7
    tensor_parallel_size: 4
8
    port: 8000
9
    node_pool: compute
10
11
benchmarks:
12
  - name: gsm8k
13
    repeats: 8
14
    solver:
15
      type: simple
16
      service: model
17
18
cluster:
19
  type: slurm
20
  walltime: "02:00:00"
21
  shards: 16
22
  node_pools:
23
    compute:
24
      partition: batch
25
      nodes: 1
26
      ntasks_per_node: 1
27
      gres: "gpu:4"

$
nel eval run slurm_sharded.yaml

This submits 16 independent shard jobs under shard_0/, shard_1/, and so on. Each job evaluates a disjoint slice of the dataset. After all tasks complete, merge results:

$
nel eval merge ./eval_results

Generated scripts

nel eval run with a SLURM cluster config generates an sbatch script in --output-dir:

nel_eval.sbatch

For sharded runs, each shard directory gets its own script:

$
./eval_results/shard_0/nel_eval.sbatch
$
./eval_results/shard_1/nel_eval.sbatch
$
...

Each script exports fixed shard coordinates, for example NEL_SHARD_IDX=0 and NEL_TOTAL_SHARDS=16, and writes into that shard’s output directory.

Resume after failure

If a benchmark fails within a multi-benchmark SLURM suite, re-submit with --resume to skip already-completed benchmarks:

$
nel eval run slurm_eval.yaml --resume

ai-dynamo deployment

Deploy NVIDIA ai-dynamo (sglang engine) as a managed service. Dynamo is multi-process by design — NEL launches the NATS broker, etcd, the dynamo.frontend HTTP server, and the dynamo.sglang worker(s) in one sbatch job. The OpenAI-compatible endpoint is the frontend on svc.port.

Aggregated (one worker):

1
services:
2
  glm:
3
    type: dynamo
4
    model: /lustre/path/to/GLM-5.1-FP8
5
    served_model_name: glm
6
    protocol: chat_completions
7
    port: 8000
8
    tensor_parallel_size: 8

For multi-node TP the worker rendezvouses via sglang’s torch.distributed (--nnodes/--node-rank/--dist-init-addr); set num_nodes: 2 and tensor_parallel_size: 16 and NEL gates the NATS/etcd/frontend bootstrap on rank 0.

Disaggregated (separate prefill + decode workers, KV transfer over NIxl/UCX-CUDA):

1
services:
2
  glm:
3
    type: dynamo
4
    model: /lustre/path/to/GLM-5.1-FP8
5
    served_model_name: glm
6
    protocol: chat_completions
7
    port: 8000
8
    prefill:
9
      tensor_parallel_size: 8
10
      num_nodes: 1
11
      node_pool: prefill_pool
12
      extra_env:
13
        UCX_TLS: "rc_x,rc,cuda_copy,cuda_ipc"
14
        UCX_NET_DEVICES: "mlx5_0:1"
15
    decode:
16
      tensor_parallel_size: 8
17
      num_nodes: 1
18
      node_pool: decode_pool
19
      extra_env:
20
        UCX_TLS: "rc_x,rc,cuda_copy,cuda_ipc"
21
        UCX_NET_DEVICES: "mlx5_0:1"
22
23
cluster:
24
  type: slurm
25
  node_pools:
26
    prefill_pool:
27
      partition: batch
28
      nodes: 1
29
      gres: "gpu:8"
30
    decode_pool:
31
      partition: batch
32
      nodes: 1
33
      gres: "gpu:8"

Mode is implicit: presence of prefill AND decode switches to disaggregated; their absence (default) is aggregated. Setting only one of the pair is a validation error.

Default image: nvcr.io/nvidia/ai-dynamo/sglang-runtime:1.1.1-cuda13 from NGC. Override per service with image: or globally via containers.toml.

Heterogeneous jobs

For configurations that require separate GPU and CPU nodes (e.g., model on GPU + sandboxes on CPU), define multiple node pools:

1
cluster:
2
  type: slurm
3
  walltime: "04:00:00"
4
  node_pools:
5
    gpu:
6
      partition: gpu
7
      nodes: 1
8
      gres: "gpu:4"
9
    sandbox:
10
      partition: cpu
11
      nodes: 2
12
      slots_per_node: 4

Services and sandboxes reference pools via node_pool: gpu or node_pool: sandbox.

Note: shards is incompatible with heterogeneous jobs (multiple node pools).

Manual workflow

$
# 1. Generate scripts without submitting
$
nel eval run slurm_eval.yaml --dry-run
$
$
# 2. Review
$
cat ./eval_results/nel_eval.sbatch
$
$
# 3. Submit eval
$
EVAL_JOB=$(sbatch ./eval_results/nel_eval.sbatch | awk '{print $NF}')
$
$
# 4. For sharded runs, merge after all tasks complete
$
nel eval merge ./eval_results

Serve on SLURM

Long-running environment server for Gym training:

$
nel serve -b gsm8k --gym-compat --port 9090

Wrap in an sbatch script for SLURM submission.

Cluster config options

For remote SLURM execution (hostname set), output.dir must be an absolute path visible on the remote login and compute nodes. Relative output paths are rejected because scripts, logs, and shard outputs must be copied and mounted predictably.

Node pool options