Manage Jobs
This section describes how to configure jobs in NeMo Helix. The Jobs service is responsible for scheduling batch jobs, collecting telemetry and managing job results.
Creation and deletion lifecycle
A Job is visible as creation_pending while its resources are being prepared. It becomes created only after setup is durable. Execution and normal lifecycle mutations wait for that publication. A valid created Job can have no current Attempt; get, list, and status still return it, while attempt-dependent operations report that no Attempt exists.
Deletion first records deletion_pending on an eligible Job. HTTP 204 acknowledges that request; the Jobs controller finishes resource cleanup and removes the Job afterward. Poll the existing get operation using the returned Job ID until it returns not found when you need to wait for completion. A service error or timeout does not mean the Job has disappeared. Workspace deletion performs this wait before advancing to Files cleanup.
The controller retries failed cleanup and retains the Job until its known owned resources are settled. Caller-supplied output locations remain borrowed. Do not select another Job’s auto-created artifact Fileset as an output location: that usage has no sharing protection. An unknown empty allocation can remain after an allocation response is lost, and can block a later create with the same name.
Deleting a legacy Job without an allocation receipt removes the Job and its child records but retains the unverified Fileset for reconciliation. That Fileset can still block creating another Job with the same name until it is reconciled.
jobs.storage.creation_timeout_seconds sets the grace before unfinished creation becomes eligible for automatic cleanup; its default is 900 seconds. Accepted deletion is eligible for cleanup immediately, including for recently created Jobs. An explicit deletion request can also end unfinished creation before this timeout. Run exactly one active Jobs cleanup controller. The supplied Helm deployment uses one replica and a Recreate rollout; custom startup, restart, and failover procedures must stop the old controller before starting its replacement.
Continuing SDK handles and watches retain the Job ID they received, so ordinary deletion and recreation under the same name does not move them to the replacement. Existing HTTP addressing still tries an ID and then a name. A literal Job name equal to a disappeared ID remains ambiguous at that public boundary.
Listing Jobs by Type
Every plugin that submits jobs has a list-jobs command that lists only that plugin’s jobs, newest first. Use --type to filter to specific job types; the command’s help names them.
nemo jobs list --type does the same from the core jobs command. Pass a plugin to list all its job types, or <plugin>.<job> for one job type. Repeat --type to list several:
When several job types are listed, the results are merged into one list. --page-size caps the merged list and --all-pages fetches every job; --page works only when exactly one job type is listed. Each job in the output has a type field naming its job type.
Viewing Job Attempts
Each job run creates an attempt. The job response includes attempt_id for the current attempt. List the attempt history or retrieve the current attempt without first looking up its ID:
Step and task APIs use attempt selectors to keep execution updates attached to the run that produced them. Log, watch, and tail commands also accept an attempt ID or current when you need data from a specific run.
Results remain job-owned, and result names are unique across the whole job. New result records include the concrete attempt ID that produced them. Use nemo jobs results list <job-name> --attempt-id <attempt-id> (or --attempt-id current) to filter the canonical result list. nemo jobs results get and nemo jobs results download identify a result directly by its job-wide unique name and do not need an attempt selector.
Updating Jobs
Use nemo jobs update to change mutable metadata or the job definition used by future attempts:
Definition updates must provide both spec and platform_spec; the Jobs Factory recompiles plugin job specs before forwarding the paired fields. To clear a job description, use --input-data '{"description": null}'.
Viewing Job Logs
Use nemo jobs get-logs to inspect paginated job logs:
Use nemo jobs tail when you only need the newest lines from a long or running job:
Both commands support --attempt-id, --step-id, and --task-id to narrow the log output. The attempt selector accepts an attempt ID or current.
Concurrency Limits and Queueing
The Jobs service bounds how much work runs at once. Before it starts a job step, the scheduler checks the number of jobs currently running against three limits. A step starts only when it is under all of them:
On Kubernetes, the Helm chart sets these through core.jobs.maxActiveJobsPerUser, core.jobs.maxActiveJobsPerWorkspace, and core.jobs.maxActiveJobs (the executor limit for Kubernetes and Volcano profiles).
Each running job counts as one unit, no matter how many resources or tasks it uses. A job runs its steps one at a time, so the scheduler counts the job’s current step: it counts toward the limits while it is pending, active, cancelling, or pausing. Queued, paused, and finished steps do not count. The scheduler counts current work each time it runs, so the limits do not depend on past usage.
What happens at a limit
A step that does not fit is not failed. It moves to queued, and the reason, such as executor cpu/default is at its limit (8/8 active jobs), appears in the step’s status_details.message. The scheduler retries queued steps on every pass (every schedule_interval_seconds, 5 by default) and starts them as running jobs finish.
When many steps are waiting and capacity is limited, the scheduler takes one step per user in turn, so a user with a large backlog does not block other users. Each user’s own steps still start in the order they were created.
If the scheduler cannot read the current load, it queues the steps for that pass and tries again on the next one.
A per-workspace override is not available yet. Every workspace uses max_active_jobs_per_workspace. A limit per job group is also not available yet.
Execution Profiles
You can configure jobs to run on different hardware by specifying an execution profile. This allows you to define different compute resources and job execution backends for different job types and use cases.
An execution profile is defined by the following attributes:
- A profile name (
profile) - A compute provider (e.g.
cpuorgpu) - An execution backend (e.g.
subprocess,docker,kubernetes_job,volcano_job)
By default, NeMo Helix defines a default CPU and default GPU provider to launch CPU and GPU bound jobs.
You can configure multiple execution profiles to suit the shape of your compute environment. For example, if you have a compute environment with heterogeneous infrastructure (e.g., two types of GPU hardware such as A100 and H200), you can define a list of execution profiles as follows:
For full configuration details, see the platform configuration reference.
Default Execution Profiles
Each execution profile is addressed by its (provider, profile) pair. When a job does not request a named profile, it uses profile: default for its provider. The built-in profiles depend on the platform runtime.
- CPU provider:
provider: cpu,profile: default. - GPU provider:
provider: gpu,profile: default. - Local host subprocess provider:
provider: subprocess,profile: default. It is enabled by default for Docker and no-control-plane runtimes, and disabled by default for Kubernetes.
You may configure the default execution profiles by updating the executor_defaults section of the jobs section of the platform configuration. The structure of the executor_defaults matches the configuration of the execution backend configuration.
Execution Backends
Execution backends launch and manage job tasks. Docker, Kubernetes, and Volcano run tasks in containers; the subprocess backend runs a command directly on the control plane host.
NeMo Helix currently supports the following execution backends:
- Docker (
docker) - Kubernetes Jobs (
kubernetes_job) - Volcano Jobs (
volcano_job) - Subprocess (
subprocess)
For when the control plane itself needs Docker (deployments preflight) versus when jobs use subprocess, see Local Docker and Executors. Inspect registered profiles with nemo jobs list-execution-profiles -f json.
Subprocess
The subprocess backend runs job commands directly on the control plane host. It is useful for trusted local development workloads, but it does not provide container isolation and should not run untrusted code.
With platform.runtime: docker or platform.runtime: none, NeMo Helix registers subprocess/default unless it is explicitly disabled. Kubernetes deployments do not enable it by default. CPU container steps that request a configured subprocess profile are translated to this host backend only when container.entrypoint or container.command provides a command; otherwise job creation returns HTTP 422. Some local profiles (for example auditor and GPU/customizer) still use Docker even when subprocess/default is present.
Docker
NeMo Helix supports Docker as an execution backend for CPU and GPU based jobs. When the shared GPU pool is configured (see below), Docker will use only the configured GPU devices and will not over-schedule jobs if there are not currently enough GPUs available.
With platform.runtime: docker, NeMo Helix registers Docker-backed cpu/default and gpu/default profiles when Docker is available. Docker and none runtimes also register subprocess/default unless it is explicitly disabled; Kubernetes does not. CPU container steps that request a configured subprocess profile are translated to it only when container.entrypoint or container.command provides a command; otherwise job creation returns HTTP 422. Configure or select an explicit image-backed profile when a workload must run in a container. See Local Docker and Executors for how this relates to control plane startup preflight (deployments) versus job backends.
If you are running GPU jobs with Docker, see GPU Configuration for information on configuring the shared GPU pool. This configuration is shared between the jobs and models services to prevent GPU resource conflicts.
Kubernetes Jobs
NeMo Helix supports Kubernetes Jobs as an execution backend for CPU and GPU based jobs.
KAI Scheduler
KAI Scheduler is a Kubernetes scheduler for AI/ML workloads. You can direct Kubernetes Jobs to use KAI Scheduler by setting scheduler_name in the execution profile config, and by adding the required queue labels to the job and pod metadata.
See the KAI Scheduler documentation for more information on how to configure jobs with KAI Scheduler.
Volcano Jobs
NeMo Helix supports Volcano Jobs as an execution backend for launching distributed GPU jobs. Install Volcano as a cluster engine; see Volcano.
Volcano Jobs are configured using the same configuration as Kubernetes Jobs, but with the following additional configuration:
queue: The Volcano queue to submit the job to.scheduler_name: The Volcano scheduler to use for the job.plugins: The Volcano plugins to use for the job.max_retry: The maximum number of retries for the job.enable_multi_node_networking: Enable multi-node networking injection. Sets annotations to trigger Kyverno policy mutations. This is only available if the platform is configured to use multi-node networking (see Multi-Node Networking).