Manage Jobs

This section describes how to configure jobs in NeMo Platform. The Jobs service is responsible for scheduling batch jobs, collecting telemetry and managing job results.

Execution Profiles

You can configure jobs to run on different hardware by specifying an execution profile. This allows you to define different compute resources and job execution backends for different job types and use cases.

An execution profile is defined by the following attributes:

• A profile name (profile)
• A compute provider (e.g. cpu or gpu)
• An execution backend (e.g. docker, kubernetes_job, volcano_job)

By default, the NeMo Platform defines a default CPU and default GPU provider to launch CPU and GPU bound jobs.

You can configure multiple execution profiles to suit the shape of your compute environment. For example, if you have a compute environment with heterogeneous infrastructure (e.g., two types of GPU hardware such as A100 and H200), you can define a list of execution profiles as follows:

1
jobs:
2
 executors:
3
 # Allow any CPU-bound job to run anywhere
4
 - provider: cpu
5
 profile: default
6
 backend: kubernetes_job
7
 config: {...}
8
 
9
 # Run on A100 hardware
10
 - provider: gpu
11
 profile: a100-pool
12
 backend: kubernetes_job
13
 config:
14
 # use the appropriate pod scheduling for a100-pool
15
 node_selector:
16
 node-pool-name: a100-pool
17
18
 # Run on H200 hardware
19
 - provider: gpu
20
 profile: h200-pool
21
 backend: kubernetes_job
22
 config:
23
 # use the appropriate pod scheduling for h200-pool
24
 node_selector:
25
 node-pool-name: h200-pool

For full configuration details, see the platform configuration reference.

Default Execution Profiles

The NeMo Platform defines a default execution profile for each execution backend depending on the platform’s control plane. The default execution profile is used when no specific execution profile is specified for a job.

• Default CPU Execution Profile (cpu): The default execution profile for CPU based jobs.
• Default GPU Execution Profile (gpu): The default execution profile for GPU based jobs.

You may configure the default execution profiles by updating the executor_defaults section of the jobs section of the platform configuration. The structure of the executor_defaults matches the configuration of the execution backend configuration.

1
jobs:
2
 executor_defaults:
3
 # Override the default Docker execution profile configuration
4
 docker:
5
 storage:
6
 volume_name: nemo-jobs-storage
7
8
 # Override the default Kubernetes execution profile configuration
9
 kubernetes_job:
10
 tolerations:
11
 - key: nvidia.com/gpu
12
 operator: Exists
13
 effect: NoSchedule
14
 node_selector:
15
 kubernetes.io/arch: amd64
16
17
 # Override the default Volcano execution profile configuration
18
 volcano_job:
19
 storage:
20
 pvc_name: nemo-jobs-storage
21
 tolerations:
22
 - key: nvidia.com/gpu
23
 operator: Exists
24
 effect: NoSchedule
25
 node_selector:
26
 kubernetes.io/arch: amd64

Execution Backends

Execution backends are the containerized job execution systems that NeMo Platform jobs are scheduled to run on. Each execution backend is responsible for launching and managing the job containers, and the Jobs service communicates with the execution backend to schedule and manage jobs.

NeMo Platform currently supports the following execution backends:

• Docker (docker)
• Kubernetes Jobs (kubernetes_job)
• Volcano Jobs (volcano_job)

Docker

The NeMo Platform supports Docker as an execution backend for CPU and GPU based jobs. When the shared GPU pool is configured (see below), Docker will use only the configured GPU devices and will not over-schedule jobs if there are not currently enough GPUs available.

Note: The NeMo Platform supports Docker-based job execution by default when running the NeMo Platform in quickstart mode, which requires no configuration.

1
jobs:
2
 executors:
3
 # Define the default CPU provider
4
 - provider: cpu
5
 profile: default
6
 backend: docker
7
 config:
8
 storage:
9
 volume_name: nemo-platform_jobs_storage
10
11
 # Define the default GPU provider
12
 - provider: gpu
13
 profile: default
14
 backend: docker
15
 config:
16
 storage:
17
 volume_name: nemo-platform_jobs_storage

Kubernetes Jobs

The NeMo Platform supports Kubernetes Jobs as an execution backend for CPU and GPU based jobs.

1
jobs:
2
 executors:
3
 # Define the default CPU provider
4
 - profile: default
5
 backend: kubernetes_job
6
 provider: cpu
7
 config:
8
 # Storage is the kubernetes_job storage configuration
9
 storage:
10
 # Define the name of a persistent volume claim that will be used by launched jobs
11
 pvc_name: nemo-core-jobs-storage
12
13
 # Define the default GPU provider
14
 - profile: default
15
 backend: kubernetes_job
16
 provider: gpu
17
 config:
18
 storage:
19
 pvc_name: nemo-core-jobs-storage
20
 # You can configure custom labels and annotations on Kubernetes Jobs and their pods. This may be useful within environments that require adding integration with service meshes or similar cluster-level integrations.
21
 job_metadata:
22
 labels:
23
 my-custom-label: "value"
24
 annotations:
25
 example.com/annotation: "value"
26
 pod_metadata:
27
 labels:
28
 sidecar.istio.io/inject: "false"
29
 annotations:
30
 example.com/annotation: "value"
31
 # You can configure typical Kubernetes pod scheduling behavior via node selectors, tolerations, and node/pod affinities.
32
 node_selector:
33
 kubernetes.io/arch: amd64
34
 tolerations:
35
 - key: nvidia.com/gpu
36
 operator: Exists
37
 effect: NoSchedule
38
 affinity:
39
 nodeAffinity:
40
 requiredDuringSchedulingIgnoredDuringExecution:
41
 nodeSelectorTerms:
42
 - matchExpressions:
43
 - key: highmem
44
 operator: In
45
 values:
46
 - "true"

KAI Scheduler

KAI Scheduler is a Kubernetes scheduler for AI/ML workloads. You can direct Kubernetes Jobs to use KAI Scheduler by setting scheduler_name in the execution profile config, and by adding the required queue labels to the job and pod metadata.

1
jobs:
2
 executors:
3
 - profile: default
4
 backend: kubernetes_job
5
 provider: gpu
6
 config:
7
 # Direct pods to the KAI scheduler
8
 scheduler_name: kai-scheduler
9
 # Required labels for KAI Scheduler queue assignment
10
 job_metadata:
11
 labels:
12
 kai.scheduler/queue-name: default
13
 pod_metadata:
14
 labels:
15
 kai.scheduler/queue-name: default
16
 storage:
17
 pvc_name: nemo-core-jobs-storage

See the KAI Scheduler documentation for more information on how to configure jobs with KAI Scheduler.

Volcano Jobs

The NeMo Platform supports Volcano Jobs as an execution backend for launching distributed GPU jobs.

Volcano Jobs are configured using the same configuration as Kubernetes Jobs, but with the following additional configuration:

• queue: The Volcano queue to submit the job to.
• scheduler_name: The Volcano scheduler to use for the job.
• plugins: The Volcano plugins to use for the job.
• max_retry: The maximum number of retries for the job.
• enable_multi_node_networking: Enable multi-node networking injection. Sets annotations to trigger Kyverno policy mutations. This is only available if the platform is configured to use multi-node networking (see Multi-Node Networking).

1
jobs:
2
 executors:
3
 - profile: default
4
 backend: volcano_job
5
 provider: gpu_distributed
6
 config:
7
 queue: default
8
 scheduler_name: volcano
9
 plugins:
10
 pytorch: ["--master=leader", "--worker=worker", "--port=23456"]
11
 max_retry: 0
12
 # Additional configuration for the Volcano Job, shared with Kubernetes Jobs
13
 ...