Manage Jobs

This section describes how to configure jobs in NeMo Helix. The Jobs service is responsible for scheduling batch jobs, collecting telemetry and managing job results.

Viewing Job Logs

Use nemo jobs get-logs to inspect paginated job logs:

nemo jobs get-logs <job-name> --workspace default --limit 100

Use nemo jobs tail when you only need the newest lines from a long or running job:

nemo jobs tail <job-name> --workspace default -n 500

Both commands support --attempt-id, --step-id, and --task-id to narrow the log output.

Execution Profiles

You can configure jobs to run on different hardware by specifying an execution profile. This allows you to define different compute resources and job execution backends for different job types and use cases.

An execution profile is defined by the following attributes:

• A profile name (profile)
• A compute provider (e.g. cpu or gpu)
• An execution backend (e.g. subprocess, docker, kubernetes_job, volcano_job)

By default, the NeMo Helix defines a default CPU and default GPU provider to launch CPU and GPU bound jobs.

You can configure multiple execution profiles to suit the shape of your compute environment. For example, if you have a compute environment with heterogeneous infrastructure (e.g., two types of GPU hardware such as A100 and H200), you can define a list of execution profiles as follows:

jobs:
  executors:
    # Allow any CPU-bound job to run anywhere
    - provider: cpu
      profile: default
      backend: kubernetes_job
      config: {}
    # Run on A100 hardware
    - provider: gpu
      profile: a100-pool
      backend: kubernetes_job
      config:
        # Use the appropriate pod scheduling for a100-pool.
        node_selector:
          node-pool-name: a100-pool
    # Run on H200 hardware
    - provider: gpu
      profile: h200-pool
      backend: kubernetes_job
      config:
        # Use the appropriate pod scheduling for h200-pool.
        node_selector:
          node-pool-name: h200-pool

For full configuration details, see the platform configuration reference.

Default Execution Profiles

Each execution profile is addressed by its (provider, profile) pair. When a job does not request a named profile, it uses profile: default for its provider. The built-in profiles depend on the platform runtime.

• CPU provider: provider: cpu, profile: default.
• GPU provider: provider: gpu, profile: default.
• Local host subprocess provider: provider: subprocess, profile: default. It is enabled by default for Docker and no-control-plane runtimes, and disabled by default for Kubernetes.

You may configure the default execution profiles by updating the executor_defaults section of the jobs section of the platform configuration. The structure of the executor_defaults matches the configuration of the execution backend configuration.

jobs:
  executor_defaults:
    # Override the default Docker execution profile configuration.
    docker:
      storage:
        volume_name: nemo-jobs-storage
    # Override the default host subprocess execution profile configuration.
    subprocess:
      working_directory: /tmp/nhx-subprocess-jobs
      cleanup_completed_jobs_immediately: false
    # Override the default Kubernetes execution profile configuration.
    kubernetes_job:
      tolerations:
        - key: nvidia.com/gpu
          operator: Exists
          effect: NoSchedule
      node_selector:
        kubernetes.io/arch: amd64
    # Override the default Volcano execution profile configuration.
    volcano_job:
      storage:
        pvc_name: nemo-jobs-storage
      tolerations:
        - key: nvidia.com/gpu
          operator: Exists
          effect: NoSchedule
      node_selector:
        kubernetes.io/arch: amd64

Execution Backends

Execution backends launch and manage job tasks. Docker, Kubernetes, and Volcano run tasks in containers; the subprocess backend runs a command directly on the control plane host.

NeMo Helix currently supports the following execution backends:

• Docker (docker)
• Kubernetes Jobs (kubernetes_job)
• Volcano Jobs (volcano_job)
• Subprocess (subprocess)

For when the control plane itself needs Docker (deployments preflight) versus when jobs use subprocess, see Local Docker and Executors. Inspect registered profiles with nemo jobs list-execution-profiles -f json.

Subprocess

The subprocess backend runs job commands directly on the control plane host. It is useful for trusted local development workloads, but it does not provide container isolation and should not run untrusted code.

With platform.runtime: docker or platform.runtime: none, NeMo Helix registers subprocess/default unless it is explicitly disabled. Kubernetes deployments do not enable it by default. CPU container steps that request a configured subprocess profile are translated to this host backend only when container.entrypoint or container.command provides a command; otherwise job creation returns HTTP 422. Some local profiles (for example auditor and GPU/customizer) still use Docker even when subprocess/default is present.

jobs:
  executors:
    - provider: subprocess
      profile: default
      backend: subprocess
      config:
        working_directory: /tmp/nhx-subprocess-jobs
        cleanup_completed_jobs_immediately: false

Docker

The NeMo Helix supports Docker as an execution backend for CPU and GPU based jobs. When the shared GPU pool is configured (see below), Docker will use only the configured GPU devices and will not over-schedule jobs if there are not currently enough GPUs available.

With platform.runtime: docker, NeMo Helix registers Docker-backed cpu/default and gpu/default profiles when Docker is available. Docker and none runtimes also register subprocess/default unless it is explicitly disabled; Kubernetes does not. CPU container steps that request a configured subprocess profile are translated to it only when container.entrypoint or container.command provides a command; otherwise job creation returns HTTP 422. Configure or select an explicit image-backed profile when a workload must run in a container. See Local Docker and Executors for how this relates to control plane startup preflight (deployments) versus job backends.

jobs:
  executors:
    # Define the default CPU provider.
    - provider: cpu
      profile: default
      backend: docker
      config:
        storage:
          volume_name: nemo-helix_jobs_storage
    # Define the default GPU provider.
    - provider: gpu
      profile: default
      backend: docker
      config:
        storage:
          volume_name: nemo-helix_jobs_storage

Kubernetes Jobs

The NeMo Helix supports Kubernetes Jobs as an execution backend for CPU and GPU based jobs.

jobs:
  executors:
    # Define the default CPU provider.
    - profile: default
      backend: kubernetes_job
      provider: cpu
      config:
        # Storage is the kubernetes_job storage configuration.
        storage:
          # Persistent volume claim used by launched jobs.
          pvc_name: nemo-core-jobs-storage
    # Define the default GPU provider.
    - profile: default
      backend: kubernetes_job
      provider: gpu
      config:
        storage:
          pvc_name: nemo-core-jobs-storage
        # Add integration metadata to Kubernetes Jobs and their pods.
        job_metadata:
          labels:
            my-custom-label: "value"
          annotations:
            example.com/annotation: "value"
        pod_metadata:
          labels:
            sidecar.istio.io/inject: "false"
          annotations:
            example.com/annotation: "value"
        # Configure standard Kubernetes pod scheduling behavior.
        node_selector:
          kubernetes.io/arch: amd64
        tolerations:
          - key: nvidia.com/gpu
            operator: Exists
            effect: NoSchedule
        affinity:
          nodeAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              nodeSelectorTerms:
                - matchExpressions:
                    - key: highmem
                      operator: In
                      values:
                        - "true"

KAI Scheduler

KAI Scheduler is a Kubernetes scheduler for AI/ML workloads. You can direct Kubernetes Jobs to use KAI Scheduler by setting scheduler_name in the execution profile config, and by adding the required queue labels to the job and pod metadata.

jobs:
  executors:
    - profile: default
      backend: kubernetes_job
      provider: gpu
      config:
        # Direct pods to the KAI scheduler.
        scheduler_name: kai-scheduler
        # Required labels for KAI Scheduler queue assignment.
        job_metadata:
          labels:
            kai.scheduler/queue-name: default
        pod_metadata:
          labels:
            kai.scheduler/queue-name: default
        storage:
          pvc_name: nemo-core-jobs-storage

See the KAI Scheduler documentation for more information on how to configure jobs with KAI Scheduler.

Volcano Jobs

The NeMo Helix supports Volcano Jobs as an execution backend for launching distributed GPU jobs. Install Volcano as a cluster engine; see Volcano.

Volcano Jobs are configured using the same configuration as Kubernetes Jobs, but with the following additional configuration:

• queue: The Volcano queue to submit the job to.
• scheduler_name: The Volcano scheduler to use for the job.
• plugins: The Volcano plugins to use for the job.
• max_retry: The maximum number of retries for the job.
• enable_multi_node_networking: Enable multi-node networking injection. Sets annotations to trigger Kyverno policy mutations. This is only available if the platform is configured to use multi-node networking (see Multi-Node Networking).

jobs:
  executors:
    - profile: default
      backend: volcano_job
      provider: gpu_distributed
      config:
        queue: default
        scheduler_name: volcano
        plugins:
          pytorch: ["--master=leader", "--worker=worker", "--port=23456"]
        max_retry: 0
        # Add any other supported Kubernetes Job configuration here.