Multi-Node Ray on Slurm

SlurmRayClient is a drop-in replacement for RayClient that bootstraps a multi-node Ray cluster under SLURM. The head node starts ray start --head, writes the assigned GCS port to a shared file, and waits for workers to join. Worker nodes block on the port file and call ray start --block. Only the head returns from start(), so your pipeline code runs on the head while workers stay attached to the cluster.

When to Use It

Use SlurmRayClient when:

• You’re submitting Ray-backed pipelines through sbatch and want to span more than one node.
• You want a single-binary entry point: the same script works on a 1-node debug job and a 2-node production job without code changes.

For single-node SLURM jobs, the regular RayClient is sufficient.

Prerequisites

• SLURM cluster: a working SLURM cluster with at least one GPU node.
• Shared filesystem: a filesystem visible from every allocated node (NFS, Lustre, GPFS) — required for the Ray port broadcast file.
• Pyxis + enroot (recommended): for the container-based submit script. If unavailable, use the bare-metal uv script.
• uv (alternative): for bare-metal execution. Install via astral.sh/uv.
• Source checkout: NeMo Curator source code on a shared filesystem path readable from every node.

Drop-in Replacement

Replace RayClient with SlurmRayClient. No other changes are required:

1
from nemo_curator.core.client import SlurmRayClient
2
3
client = SlurmRayClient()
4
client.start()
5
6
# ... pipeline code ...
7
8
client.stop()

The head/worker split is determined automatically from SLURM_NODEID. Workers stay inside SlurmRayClient.start() for the duration of the job; the head runs your pipeline.

Required Environment

Submit Scripts

The repository ships two reference submit scripts at tutorials/slurm/:

$
#!/bin/bash
$
#SBATCH --job-name=curator-slurm-demo
$
#SBATCH --nodes=2
$
#SBATCH --ntasks-per-node=1
$
#SBATCH --cpus-per-task=16
$
#SBATCH --gpus-per-node=2
$
#SBATCH --time=00:10:00
$
#SBATCH --output=logs/slurm_demo_%j.log
$
$
set -euo pipefail
$
$
# Path on shared filesystem, visible from every node
$
CURATOR_DIR="${CURATOR_DIR:-$(cd "$(dirname "$0")/../.." && pwd)}"
$
CONTAINER_IMAGE="nvcr.io/nvidia/nemo-curator:{{ container_version }}"
$
$
# Shared filesystem path for Ray's port broadcast
>
export RAY_PORT_BROADCAST_DIR="${CURATOR_DIR}/logs"
>
export RAY_TMPDIR="/tmp/ray_${SLURM_JOB_ID}"
>
>
mkdir -p logs
>
>
# Run the pipeline inside the container on every node
>
srun \
>
    --ntasks-per-node=1 \
>
    --container-image="${CONTAINER_IMAGE}" \
>
    --container-mounts="${CURATOR_DIR}:${CURATOR_DIR}" \
>
    bash -c "
>
cd '${CURATOR_DIR}'
>
export RAY_TMPDIR=/tmp/ray_\${SLURM_JOB_ID}
>
export RAY_PORT_BROADCAST_DIR='${CURATOR_DIR}/logs'
>
python '${CURATOR_DIR}/tutorials/slurm/pipeline.py' --slurm --num-tasks 80
>
"

$
sbatch --nodes=2 --gpus-per-node=8 tutorials/slurm/submit_container.sh

Verified Configurations

End-to-end tested on H100 nodes using the nvcr.io/nvidia/nemo-curator:26.02 container at the time SlurmRayClient was introduced. The reference workload is a word-count + GPU-tagging pipeline shipped in the tutorials/slurm/ directory; it works unchanged on subsequent container versions, including {{ container_version }}.

Sample output from the 2-node, 8-GPU run:

Tasks processed by 2 distinct node(s):
  pool0-00786: 8 GPU(s): NVIDIA H100 80GB HBM3, 81559 MiB; ...
  pool0-00795: 8 GPU(s): NVIDIA H100 80GB HBM3, 81559 MiB; ...

Monitoring and Logs

Check job status:

$
squeue -u "$USER"

View logs (the bundled scripts write to logs/slurm_demo_<jobid>.log):

$
tail -f logs/slurm_demo_<jobid>.log

The first line of each node’s log identifies its role:

[head-node-name] SLURM_NODEID=0 ray=2.54.0 ...
[worker-node-name] SLURM_NODEID=1 joining head at <ip:port>

Troubleshooting

Performance Considerations

• GPU Memory: ensure all nodes have homogeneous GPU memory; mixed-memory clusters cause Ray to schedule conservatively against the smallest GPU.
• Network: Ray’s object store and shuffle traffic flow over the cluster’s primary network. Use InfiniBand or 100 GbE+ for large-scale jobs.
• Shared FS bandwidth: stage worker reads/writes through the shared filesystem. For S3-backed pipelines, prefer S3 over the shared FS to reduce contention.
• Time limits: SLURM kills jobs at the wall-clock limit. The bundled scripts use --time=00:10:00 for the demo; raise it to match your real workload.

Related Topics

• Deploy Image Curation on Slurm — full image curation pipeline using SLURM and Pyxis.
• Container Environments — image versions and SLURM-specific environment variables.
• Execution Backends — Ray Data, Xenna, and Ray actor pool executor selection.