> This page is for version Latest (default).
> For other versions, use one of these documentation indexes:
> - Latest (default): https://docs.nvidia.com/nemo-helix/latest/llms.txt
> - 0.9: https://docs.nvidia.com/nemo-helix/v0.9/llms.txt
> - 0.8: https://docs.nvidia.com/nemo-helix/v0.8/llms.txt
> - 0.7: https://docs.nvidia.com/nemo-helix/v0.7/llms.txt
> - 0.6: https://docs.nvidia.com/nemo-helix/v0.6/llms.txt
> - 0.5: https://docs.nvidia.com/nemo-helix/v0.5/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo-helix/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo-helix/_mcp/server.

# Pause, Resume, and Rerun a Job

## Prerequisites

Before you can pause, resume, or rerun a customization job, make sure that you have configured a CLI context that points at your NeMo Helix deployment. Refer to [Connect with CLI Contexts](/nemo-helix/documentation/reference/cli-reference/connect-to-deployments).

Customization jobs run on the platform's Jobs service, so you pause, resume, and rerun them through that service with the job's name and workspace, the same way for every backend. Submitting a job returns its name. To find it later, list the workspace's customization jobs in the status you want to act on, for example the jobs that ended in `error`:

```bash
nemo jobs list --workspace default --filter.source customization --filter.status error
```

---

## Pause a Job

Pausing stops a job without losing its progress. A job can be paused while its current step supports pausing, which depends on the training backend. Otherwise, for example while the model is still downloading, or when the backend's training step does not support pausing, the request returns `409` and names the step. Training on every backend supports pausing: `automodel`, `unsloth`, and `rl` for both DPO and GRPO.

```bash
nemo jobs pause automodel-a1b2c3d4e5f6 --workspace default
```

The training step finishes the step it is on, saves a checkpoint, and exits, which frees its GPUs. The job's status moves from `pausing` to `paused`. Its storage, including the checkpoint, is kept so that the job can resume later.

If training doesn't save its checkpoint and exit in time, the pause fails and the job ends in `error`. The limit is 30 minutes on `automodel` and `unsloth`, and one hour on `rl`, where one step includes generating rollouts. You can then rerun the job from its newest checkpoint. Operators can change each limit with the backend's `training_pause_deadline_seconds` platform setting, for example `unsloth.training_pause_deadline_seconds`.

If training finishes before it acts on the pause, the job completes as usual.

## Resume a Paused Job

```bash
nemo jobs resume automodel-a1b2c3d4e5f6 --workspace default
```

Resume restarts the paused step in the same job. Training continues from the checkpoint it saved when it paused, and the job's metrics continue from where they stopped. If the job logs to W\&B or MLflow, it continues the same run.

If the training image changed while the job was paused, for example because its tag now points to a newer build, the training step's status details include an `image_digest_warning` that names both images. Training continues, but it may behave differently or fail to load its checkpoint. A rerun compares the image the same way.

By default, a paused job keeps its storage for 7 days. To keep it for a different period, up to the platform maximum of 30 days, set the job's pause window in seconds:

```bash
nemo jobs control automodel-a1b2c3d4e5f6 --workspace default --pause-ttl-seconds 1209600
```

The window is measured from when the job paused, so changing it does not restart it. When the window ends, the job's storage is deleted and the job is cancelled. It can't be resumed after that.

## Rerun a Failed Job

```bash
nemo jobs rerun automodel-a1b2c3d4e5f6 --workspace default
```

Rerun starts a new attempt of a job that ended in `error`. Training continues from the newest checkpoint the failed attempt saved, so only the steps after that checkpoint run again. The failed attempt is kept as a record, and if the job logs to W\&B or MLflow, the rerun continues the same run.

A rerun uses the same configuration as the failed attempt. If the configuration caused the failure, for example by running out of GPU memory, the rerun fails the same way, so create a new job with a different configuration instead. When the failed attempt reported a cause like this, the rerun's status details include a `rerun_warning`.

By default, a failed job keeps its storage for 4 hours. After that, rerun returns `409`.

## When Each Action Is Allowed

| Job status                     | Pause                                                      | Resume                                     | Rerun             |
| ------------------------------ | ---------------------------------------------------------- | ------------------------------------------ | ----------------- |
| `created`, `pending`, `active` | Yes, if the current step supports pausing, otherwise `409` | Has no effect on `active`, otherwise `409` | `409`             |
| `pausing`                      | Has no effect                                              | `409`                                      | `409`             |
| `paused`                       | Has no effect                                              | Yes                                        | `409`: use resume |
| `error`                        | `409`                                                      | `409`: use rerun                           | Yes               |
| `completed`, `cancelled`       | `409`                                                      | `409`                                      | `409`             |

The default storage periods are platform settings that each deployment can change.

## Use the SDK

```python
from nemo_helix_plugin.client.client import NemoClient
from nemo_helix_plugin.jobs.types import JobControl

client = NemoClient.from_config()
job_name = "automodel-a1b2c3d4e5f6"
workspace = "default"

# Save a checkpoint and stop, freeing the job's GPUs.
client.jobs.pause_job(name=job_name, workspace=workspace)
# Keep the paused job's storage for 14 days instead of the default 7, so it can still resume after a week.
client.jobs.update_job_control(name=job_name, workspace=workspace, body=JobControl(pause_ttl_seconds=1209600))
# Continue training from the checkpoint the pause saved.
client.jobs.resume_job(name=job_name, workspace=workspace)

# For a job that ended in error, start a new attempt from the newest checkpoint.
client.jobs.rerun_job(job=job_name, workspace=workspace)
```

> **Note**
>
> These examples use the `default` workspace. If your current CLI context uses a different workspace, run `nemo config set --workspace default` first. To check, run `nemo config view`.