> This page is for version Main (default).
> For other versions, use one of these documentation indexes:
> - Main (default): https://docs.nvidia.com/nemo/gym/main/llms.txt
> - 0.7.0: https://docs.nvidia.com/nemo/gym/v0.7.0/llms.txt
> - 0.6.0: https://docs.nvidia.com/nemo/gym/v0.6.0/llms.txt
> - 0.5.1: https://docs.nvidia.com/nemo/gym/v0.5.1/llms.txt
> - 0.5.0: https://docs.nvidia.com/nemo/gym/v0.5.0/llms.txt
> - 0.4.0: https://docs.nvidia.com/nemo/gym/v0.4.0/llms.txt
> - 0.3.0: https://docs.nvidia.com/nemo/gym/v0.3.0/llms.txt
> - 0.2.1: https://docs.nvidia.com/nemo/gym/v0.2.1/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/gym/llms.txt.

# Job Management

> List submitted runs, see what is happening to them, find where their parts are, stop them, read their logs, and queue them again.

`gym eval submit` queues one job per benchmark and leaves a [job record](./job-records) behind. The `gym job` commands read that record, so you can look at a run without going to the scheduler yourself.

| Command                         | Asks the scheduler?         | What it does                                                          |
| ------------------------------- | --------------------------- | --------------------------------------------------------------------- |
| `gym job list`                  | No                          | Lists the submissions in this machine's job index, newest first.      |
| `gym job status [ID ...]`       | Yes                         | Shows what is happening to each benchmark of one or more submissions. |
| `gym job info ID [BENCHMARK]`   | Yes, for job ids            | Describes one submission and where its parts are.                     |
| `gym job cancel ID BENCHMARK`   | Yes                         | Stops a benchmark of a submission, or every benchmark with `--all`.   |
| `gym job logs ID BENCHMARK`     | Yes, for the current job id | Prints one of a benchmark's logs.                                     |
| `gym job resubmit ID BENCHMARK` | Yes                         | Queues a benchmark of a submission again, in place.                   |

`gym job status` is the only command that reports live state. `gym job list` never touches the network, so it is instant and works offline. `gym job --help` lists every command, and `gym eval submit` ends by printing the submission's id and the command to check on it.

## List your submissions

```bash
gym job list [--limit N] [--json]
```

```text
ID                               SUBMITTED             CLUSTER    EXECUTOR  BENCHMARKS
gym-job-20261029T093012Z-4f1a9c  2026-10-29 09:30:12Z  cluster-a  slurm     gpqa, ruler, ifbench
gym-job-20261028T154501Z-77be02  2026-10-28 15:45:01Z  cluster-a  slurm     ruler
```

The index is per machine and per user. A submission made from another machine does not appear here.

## See what is happening

```bash
gym job status [ID ...] [--benchmark NAME[,NAME...]] [--limit N | --all] [--json]
```

With ids, `status` shows each submission as a block: a header with its cluster and rolled-up state, then one row per benchmark. With no ids it shows the same blocks for the newest submissions in the index (10 by default, or `--all`).

```text
gym-job-20261029T093012Z-4f1a9c  cluster-a  RUNNING (1 RUNNING, 2 SUCCEEDED)
  BENCHMARK  STATE      JOB ID                     DETAIL
  gpqa       SUCCEEDED  8123401                    COMPLETED
  ruler      RUNNING    8123455 (record: 8123402)  RUNNING
  ifbench    SUCCEEDED  8123403                    COMPLETED

gym-job-20261028T154501Z-77be02  cluster-a  FAILED (1 FAILED)
  BENCHMARK  STATE   JOB ID   DETAIL
  ruler      FAILED  8119870  FAILED
             Chain stopped at the walltime cap with rollouts missing; resubmit to continue.
             logs: gym job logs gym-job-20261028T154501Z-77be02 ruler --tail 100
```

The job id shown is the one the scheduler reports. When it differs from the id in the record (a resumed run, or a job started by hand), the recorded id is shown beside it.

A submission has one state, rolled up from its benchmarks: running wins, then queued, failed, cancelled, and only a submission whose benchmarks all succeeded is `SUCCEEDED`. The counts beside it show a mixed result.

`--benchmark` limits every submission to the named benchmarks. With ids it is strict: a submission that lacks one fails the whole call, before anything is asked, and the error lists each offender with the benchmarks it has. With no ids it selects: only submissions that have all of the named benchmarks are shown.

`--json` prints one object per submission, with the record and every benchmark's status.

### States

| State       | Meaning                                                                                                                                   |
| ----------- | ----------------------------------------------------------------------------------------------------------------------------------------- |
| `QUEUED`    | Waiting for resources.                                                                                                                    |
| `RUNNING`   | Running.                                                                                                                                  |
| `SUCCEEDED` | The run finished and exited with status 0.                                                                                                |
| `FAILED`    | The scheduler reports a failure, the run exited non-zero, or a chain of resumed jobs stopped before the run finished. The row says which. |
| `CANCELLED` | Cancelled in the scheduler.                                                                                                               |
| `UNKNOWN`   | The scheduler could not be reached, or no longer remembers the job.                                                                       |

The state comes from two places. The scheduler says what happened to the job. The job also writes files into its directory, and they win over the scheduler for a job that has finished: a chain that stopped early (`.gym_chain_incomplete`, which holds the reason shown) makes it `FAILED`, and the exit code of the run (`eval.exit`) decides success or failure. That is how a result survives the scheduler forgetting the job. A `CANCELLED` job is never overridden by a file left by an earlier attempt.

Nothing is stored: `status` reads and never writes the record.

### Many submissions at once

The cost of `gym job status` grows with the number of hosts, not submissions. Submissions are grouped by the host they are reached through, the hosts are asked at the same time, and each is asked once. If a host cannot be reached, its jobs are `UNKNOWN` with the reason, and the others still print.

On Slurm, one script per host runs a single `squeue` for the whole batch, a single `sacct` over every job id any of them has had, and reads the state files of each job directory. Jobs are found by the directory they run in, not by the recorded id, so a job started by hand or resubmitted from another machine is still seen. `sacct` has to be available for jobs that have left the queue; without it a finished job is `UNKNOWN` unless its files say how it ended.

## Find where things are

```bash
gym job info ID [BENCHMARK] [--config] [--json]
```

```text
Submission  gym-job-20261029T093012Z-4f1a9c
Executor    slurm, cluster-a
Submitted   2026-10-29 09:30:12Z by tgrzegorzek
Gym         0.4.0 (commit c155b77c8)
Run dir     /lustre/runs/gym-job-20261029T093012Z-4f1a9c

ruler
  Job dir   /lustre/runs/gym-job-20261029T093012Z-4f1a9c/ruler
  Jobs      8123402, 8123455
  Logs      SOURCE  INSTANCE  LOCATION
            driver  -         logs/driver-8123455.log
            vllm    -         logs/vllm-8123455.log

Config      not shown; use --config to print the resolved config
```

* **Jobs** lists every job id seen for the benchmark, oldest first: the whole chain of a resumed run.
* **Logs** lists the logs the job was set up to write, planned when the run was submitted (see `log_sources` in [Job Records](./job-records)), with the newest job's id filled in. A planned log is not proof that the file exists. A submission made before log sources were recorded shows none.
* **Config** is the resolved config the run was submitted with, read from the run directory. It is shown only with `--config`, because it can hold literal values a user typed. When the run directory cannot be read, the reason is shown instead.

## Stop a run

```bash
gym job cancel ID BENCHMARK
gym job cancel ID --all [--confirm ID] [--json]
```

`cancel` asks the scheduler what is running, leaves a benchmark that has already finished alone, and stops everything else of that benchmark. On Slurm that includes a queued follow-up of an auto-resume chain: cancelling only the running job would let the follow-up start, so the pending jobs are cancelled first and then the running one. It then asks again and prints what it sees. The scheduler can take a moment to report a stopped job, so the state printed may still read `RUNNING`. Nothing is written to the record.

```text
ruler: cancel requested; still RUNNING (the scheduler can take a moment to report CANCELLED)
```

`--all` stops every live benchmark of the submission. Cancelling a whole submission by accident costs more than typing its id, so it lists the live jobs and asks you to type the submission's id. `--confirm ID` supplies it for scripts; there is no `--yes`. Without a terminal and without `--confirm` it fails and cancels nothing, and a wrong id cancels nothing. A benchmark the scheduler refuses to stop (another user's job, say) is reported on its own, the others still proceed, and the exit code is 1.

## Read a log

```bash
gym job logs ID BENCHMARK [--source NAME] [--instance ID] [--tail N]
```

`logs` prints the driver's log by default. `--source` names a service instead; `gym job info` lists what a benchmark has. For a resumed run it reads the log of the job that is running now, not the first one. `--tail N` prints only the last N lines, which is worth using on a large log: the whole file is read in one call.

* A source that does not exist, or an instance given for a source that is not split by instance, fails and lists what does exist.
* A planned log that has no file means the job has not reached the step that writes it.
* A submission made before log sources were recorded says so and points at the job's `logs/` directory on the cluster.
* Bytes in a log that are not UTF-8 are replaced, not an error. There is no `--follow` yet.

## Queue a benchmark again

```bash
gym job resubmit ID BENCHMARK
```

`resubmit` queues the benchmark's `job.sh` again from its job directory and prints the new job id. It refuses while a job of that benchmark is still queued or running in that directory, so two jobs never write one run. The record is updated with the new job id.

`gym job resubmit` is the plain resubmit. It decides nothing from the run's saved evidence, and it resets the budgets. A benchmark that uses the built-in `gym eval run` runner resumes from its saved rollouts and gives every eligible failed rollout a fresh attempt budget; when an auto-resume chain is configured, the queued job also starts with a fresh retry count and walltime budget. A benchmark with a custom `command` runs that command again, so the command has to preserve or resume its own outputs.

> **Note**
>
> `gym eval resubmit` runs the same command and adds recovery: `--recover` decides from the run's saved evidence what to run again and keeps the run's budgets, and `--inspect` shows that decision without queuing anything. `gym job resubmit` has neither, so use `gym eval resubmit --recover` for a run that has failed work.