Job Launchers

NeMo AutoModel provides several ways to launch training. The right choice depends on your hardware and environment.

Which Launcher Should I Use?

I Have 1–2 GPUs on My Workstation

Use the interactive launcher. No scheduler or cluster software is needed:

$
automodel examples/llm_finetune/llama3_2/llama3_2_1b_squad.yaml

See the Run on Your Local Workstation guide.

I Have Access to a Slurm Cluster

From the repository root, copy the reference slurm.sub, edit the copy for your cluster and recipe, and submit it directly with sbatch:

$
cp slurm.sub my_cluster.sub
$
# Edit CONFIG, the #SBATCH directives, container image, and mount paths.
$
sbatch my_cluster.sub

The submission script launches torchrun across the Slurm allocation and runs the recipe selected by its CONFIG variable. Do not add a slurm: section to the recipe YAML; the automodel CLI does not dispatch Slurm jobs.

See the Run on a Cluster guide.

I Want Managed Job Submission (Slurm, Kubernetes, Docker)

Add a nemo_run: section to your YAML config. NeMo-Run loads a pre-configured executor for your compute target and submits the job:

$
automodel config_with_nemo_run.yaml

See the NeMo-Run guide.

I Want to Train on the Cloud

Add a skypilot: section to your YAML config. SkyPilot provisions VMs on any major cloud. AutoModel submits with unmanaged sky.launch, so spot instances can be preempted and this path does not recover them automatically. Managed recovery requires SkyPilot’s jobs API, which this launcher does not use:

$
automodel config_with_skypilot.yaml

See the SkyPilot guide.

I Want to Train on Kubernetes with SkyPilot

Use the same skypilot: launcher, but set cloud: kubernetes. This is a good fit when your team already has a GPU-backed Kubernetes cluster and you want SkyPilot to handle job submission and multi-node orchestration:

$
automodel examples/llm_finetune/llama3_2/llama3_2_1b_squad_skypilot_kubernetes.yaml

See the SkyPilot + Kubernetes tutorial.

All Launchers Use the Same Config

Every launch path shares the same YAML recipe format. The automodel CLI recognizes nemo_run: and skypilot: as optional launcher sections; without either section, training runs interactively on the current machine. Slurm is a separate submission path: configure the repository-root slurm.sub, point its CONFIG variable at the recipe YAML, and submit the script with sbatch.