> This page is for version 0.7.0.
> For other versions, use one of these documentation indexes:
> - Latest (default): https://docs.nvidia.com/nemo-helix/latest/llms.txt
> - 0.8.0: https://docs.nvidia.com/nemo-helix/v0.8.0/llms.txt
> - 0.7.1: https://docs.nvidia.com/nemo-helix/v0.7.1/llms.txt
> - 0.7.0: https://docs.nvidia.com/nemo-helix/v0.7.0/llms.txt
> - 0.6.0: https://docs.nvidia.com/nemo-helix/v0.6.0/llms.txt
> - 0.5.1: https://docs.nvidia.com/nemo-helix/v0.5.1/llms.txt
> - 0.5.0: https://docs.nvidia.com/nemo-helix/v0.5.0/llms.txt
> - 0.4.0: https://docs.nvidia.com/nemo-helix/v0.4.0/llms.txt
> - 0.3.0: https://docs.nvidia.com/nemo-helix/v0.3.0/llms.txt
> - 0.2.0: https://docs.nvidia.com/nemo-helix/v0.2.0/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo-helix/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo-helix/_mcp/server.

# Manage Jobs

This section describes how to configure jobs in NeMo Helix. The Jobs service is responsible for scheduling batch jobs, collecting telemetry and managing job results.

## Viewing Job Logs

Use `nemo jobs get-logs` to inspect paginated job logs:

```bash
nemo jobs get-logs <job-name> --workspace default --limit 100
```

Use `nemo jobs tail` when you only need the newest lines from a long or running job:

```bash
nemo jobs tail <job-name> --workspace default -n 500
```

Both commands support `--attempt-id`, `--step-id`, and `--task-id` to narrow the log output.

## Execution Profiles

You can configure jobs to run on different hardware by specifying an execution profile. This allows you to define different compute resources and job execution backends for different job types and use cases.

An execution profile is defined by the following attributes:

* A profile name (`profile`)
* A compute provider (e.g. `cpu` or `gpu`)
* An [execution backend](#execution-backends) (e.g. `subprocess`, `docker`, `kubernetes_job`, `volcano_job`)

By default, the NeMo Helix defines a default CPU and default GPU provider to launch CPU and GPU bound jobs.

You can configure multiple execution profiles to suit the shape of your compute environment. For example, if you have a compute environment with heterogeneous infrastructure (e.g., two types of GPU hardware such as A100 and H200), you can define a list of execution profiles as follows:

```yaml
jobs:
  executors:
    # Allow any CPU-bound job to run anywhere
    - provider: cpu
      profile: default
      backend: kubernetes_job
      config: {}

    # Run on A100 hardware
    - provider: gpu
      profile: a100-pool
      backend: kubernetes_job
      config:
        # Use the appropriate pod scheduling for a100-pool.
        node_selector:
          node-pool-name: a100-pool

    # Run on H200 hardware
    - provider: gpu
      profile: h200-pool
      backend: kubernetes_job
      config:
        # Use the appropriate pod scheduling for h200-pool.
        node_selector:
          node-pool-name: h200-pool
```

For full configuration details, see the [platform configuration reference](/documentation/kubernetes-deployment/config-reference).

### Default Execution Profiles

Each execution profile is addressed by its `(provider, profile)` pair. When a job does not request a named profile, it uses `profile: default` for its provider. The built-in profiles depend on the platform runtime.

* CPU provider: `provider: cpu`, `profile: default`.
* GPU provider: `provider: gpu`, `profile: default`.
* Local host subprocess provider: `provider: subprocess`, `profile: default`. It is enabled by default for Docker and no-control-plane runtimes, and disabled by default for Kubernetes.

You may configure the default execution profiles by updating the `executor_defaults` section of the `jobs` section of the [platform configuration](/documentation/kubernetes-deployment/config-reference). The structure of the `executor_defaults` matches the configuration of the execution backend configuration.

```yaml
jobs:
  executor_defaults:
    # Override the default Docker execution profile configuration.
    docker:
      storage:
        volume_name: nemo-jobs-storage

    # Override the default host subprocess execution profile configuration.
    subprocess:
      working_directory: /tmp/nhx-subprocess-jobs
      cleanup_completed_jobs_immediately: false

    # Override the default Kubernetes execution profile configuration.
    kubernetes_job:
      tolerations:
        - key: nvidia.com/gpu
          operator: Exists
          effect: NoSchedule
      node_selector:
        kubernetes.io/arch: amd64

    # Override the default Volcano execution profile configuration.
    volcano_job:
      storage:
        pvc_name: nemo-jobs-storage
      tolerations:
        - key: nvidia.com/gpu
          operator: Exists
          effect: NoSchedule
      node_selector:
        kubernetes.io/arch: amd64
```

## Execution Backends

Execution backends launch and manage job tasks. Docker, Kubernetes, and Volcano run tasks in containers; the subprocess backend runs a command directly on the control plane host.

NeMo Helix currently supports the following execution backends:

* Docker (`docker`)
* Kubernetes Jobs (`kubernetes_job`)
* Volcano Jobs (`volcano_job`)
* Subprocess (`subprocess`)

For when the **control plane** itself needs Docker (deployments preflight) versus when jobs use subprocess, see [Local Docker and Executors](/documentation/get-started/local-docker-and-executors). Inspect registered profiles with `nemo jobs list-execution-profiles -f json`.

### Subprocess

The subprocess backend runs job commands directly on the control plane host. It is useful for trusted local development workloads, but it does not provide container isolation and should not run untrusted code.

With `platform.runtime: docker` or `platform.runtime: none`, NeMo Helix registers `subprocess/default` unless it is explicitly disabled. Kubernetes deployments do not enable it by default. CPU container steps that request a configured subprocess profile are translated to this host backend only when `container.entrypoint` or `container.command` provides a command; otherwise job creation returns HTTP 422. Some local profiles (for example auditor and GPU/customizer) still use Docker even when `subprocess/default` is present.

```yaml
jobs:
  executors:
    - provider: subprocess
      profile: default
      backend: subprocess
      config:
        working_directory: /tmp/nhx-subprocess-jobs
        cleanup_completed_jobs_immediately: false
```

### Docker

The NeMo Helix supports Docker as an execution backend for CPU and GPU based jobs. When the shared GPU pool is configured (see below), Docker will use only the configured GPU devices and will not over-schedule jobs if there are not currently enough GPUs available.

With `platform.runtime: docker`, NeMo Helix registers Docker-backed `cpu/default` and `gpu/default` profiles when Docker is available. Docker and `none` runtimes also register `subprocess/default` unless it is explicitly disabled; Kubernetes does not. CPU container steps that request a configured subprocess profile are translated to it only when `container.entrypoint` or `container.command` provides a command; otherwise job creation returns HTTP 422. Configure or select an explicit image-backed profile when a workload must run in a container. See [Local Docker and Executors](/documentation/get-started/local-docker-and-executors) for how this relates to control plane startup preflight (deployments) versus job backends.

> **Tip**
>
> If you are running GPU jobs with Docker, see [GPU Configuration](/documentation/get-started) for information on configuring the shared GPU pool. This configuration is shared between the jobs and models services to prevent GPU resource conflicts.

```yaml
jobs:
  executors:
    # Define the default CPU provider.
    - provider: cpu
      profile: default
      backend: docker
      config:
        storage:
          volume_name: nemo-helix_jobs_storage

    # Define the default GPU provider.
    - provider: gpu
      profile: default
      backend: docker
      config:
        storage:
          volume_name: nemo-helix_jobs_storage
```

### Kubernetes Jobs

The NeMo Helix supports Kubernetes Jobs as an execution backend for CPU and GPU based jobs.

```yaml
jobs:
  executors:
    # Define the default CPU provider.
    - profile: default
      backend: kubernetes_job
      provider: cpu
      config:
        # Storage is the kubernetes_job storage configuration.
        storage:
          # Persistent volume claim used by launched jobs.
          pvc_name: nemo-core-jobs-storage

    # Define the default GPU provider.
    - profile: default
      backend: kubernetes_job
      provider: gpu
      config:
        storage:
          pvc_name: nemo-core-jobs-storage
        # Add integration metadata to Kubernetes Jobs and their pods.
        job_metadata:
          labels:
            my-custom-label: "value"
          annotations:
            example.com/annotation: "value"
        pod_metadata:
          labels:
            sidecar.istio.io/inject: "false"
          annotations:
            example.com/annotation: "value"
        # Configure standard Kubernetes pod scheduling behavior.
        node_selector:
          kubernetes.io/arch: amd64
        tolerations:
          - key: nvidia.com/gpu
            operator: Exists
            effect: NoSchedule
        affinity:
          nodeAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              nodeSelectorTerms:
                - matchExpressions:
                    - key: highmem
                      operator: In
                      values:
                        - "true"
```

#### KAI Scheduler

[KAI Scheduler](https://github.com/NVIDIA/KAI-Scheduler) is a Kubernetes scheduler for AI/ML workloads. You can direct Kubernetes Jobs to use KAI Scheduler by setting `scheduler_name` in the execution profile config, and by adding the required queue labels to the job and pod metadata.

```yaml
jobs:
  executors:
    - profile: default
      backend: kubernetes_job
      provider: gpu
      config:
        # Direct pods to the KAI scheduler.
        scheduler_name: kai-scheduler
        # Required labels for KAI Scheduler queue assignment.
        job_metadata:
          labels:
            kai.scheduler/queue-name: default
        pod_metadata:
          labels:
            kai.scheduler/queue-name: default
        storage:
          pvc_name: nemo-core-jobs-storage
```

See the [KAI Scheduler documentation](https://github.com/kai-scheduler/KAI-Scheduler/tree/main/docs) for more information on how to configure jobs with KAI Scheduler.

### Volcano Jobs

The NeMo Helix supports Volcano Jobs as an execution backend for launching distributed GPU jobs. Install Volcano as a cluster engine; see [Volcano](/documentation/kubernetes-deployment/setup/helm/volcano).

Volcano Jobs are configured using the same configuration as Kubernetes Jobs, but with the following additional configuration:

* `queue`: The Volcano queue to submit the job to.
* `scheduler_name`: The Volcano scheduler to use for the job.
* `plugins`: The Volcano plugins to use for the job.
* `max_retry`: The maximum number of retries for the job.
* `enable_multi_node_networking`: Enable multi-node networking injection. Sets annotations to trigger Kyverno policy mutations. This is only available if the platform is configured to use multi-node networking (see [Multi-Node Networking](/documentation/kubernetes-deployment/setup/helm/multinode-networking)).

```yaml
jobs:
  executors:
    - profile: default
      backend: volcano_job
      provider: gpu_distributed
      config:
        queue: default
        scheduler_name: volcano
        plugins:
          pytorch: ["--master=leader", "--worker=worker", "--port=23456"]
        max_retry: 0
        # Add any other supported Kubernetes Job configuration here.
```