> This page is for version 0.2.1.
> For other versions, use one of these documentation indexes:
> - Main (default): https://docs.nvidia.com/nemo/gym/main/llms.txt
> - 0.7.0: https://docs.nvidia.com/nemo/gym/v0.7.0/llms.txt
> - 0.6.0: https://docs.nvidia.com/nemo/gym/v0.6.0/llms.txt
> - 0.5.1: https://docs.nvidia.com/nemo/gym/v0.5.1/llms.txt
> - 0.5.0: https://docs.nvidia.com/nemo/gym/v0.5.0/llms.txt
> - 0.4.0: https://docs.nvidia.com/nemo/gym/v0.4.0/llms.txt
> - 0.3.0: https://docs.nvidia.com/nemo/gym/v0.3.0/llms.txt
> - 0.2.1: https://docs.nvidia.com/nemo/gym/v0.2.1/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/gym/llms.txt.

# nemo_gym.eval.recovery.markers

The chain markers: the strings the auto-resume chain scripts write into a job directory and the recovery
evidence reader parses back.

They are the whole contract between the chain (`nemo_gym.orchestration.executors.slurm.resume_script`, which renders
the scripts) and the recovery planner (`nemo_gym.eval.recovery`), so they live below both: the chain imports
them from here, and the planner never imports the chain.

## Module Contents

### Functions

| Name                                                                         | Description                                                                                              |
| ---------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------- |
| [`record_failure_kind`](#nemo_gym-eval-recovery-markers-record_failure_kind) | Leave the registered kind `error` carries in the failure file the driver step names, for the next guard. |

### Data

[`ACCOUNTED_WALLTIME_PREFIX`](#nemo_gym-eval-recovery-markers-ACCOUNTED_WALLTIME_PREFIX)

[`ACCOUNTING_UNKNOWN_PREFIX`](#nemo_gym-eval-recovery-markers-ACCOUNTING_UNKNOWN_PREFIX)

[`ACCUMULATED_WALLTIME_NAME`](#nemo_gym-eval-recovery-markers-ACCUMULATED_WALLTIME_NAME)

[`CANCELLED_FOLLOWUP_MARKER`](#nemo_gym-eval-recovery-markers-CANCELLED_FOLLOWUP_MARKER)

[`CHAIN_ACCOUNTING_ENV`](#nemo_gym-eval-recovery-markers-CHAIN_ACCOUNTING_ENV)

[`CONTINUATION_REASON_PREFIX`](#nemo_gym-eval-recovery-markers-CONTINUATION_REASON_PREFIX)

[`DRIVER_FAILURE_ENV`](#nemo_gym-eval-recovery-markers-DRIVER_FAILURE_ENV)

[`FOLLOW_UP_SUBMISSION_REFUSED`](#nemo_gym-eval-recovery-markers-FOLLOW_UP_SUBMISSION_REFUSED)

[`GUARDED_DRIVER_FAILURE_KINDS`](#nemo_gym-eval-recovery-markers-GUARDED_DRIVER_FAILURE_KINDS)

[`NO_ELIGIBLE_RETRY_REASON`](#nemo_gym-eval-recovery-markers-NO_ELIGIBLE_RETRY_REASON)

[`PROVISIONAL_MARKER_PREFIX`](#nemo_gym-eval-recovery-markers-PROVISIONAL_MARKER_PREFIX)

[`REQUEUE_RECONCILE_STATE`](#nemo_gym-eval-recovery-markers-REQUEUE_RECONCILE_STATE)

[`REQUEUE_UNKNOWN_REASON`](#nemo_gym-eval-recovery-markers-REQUEUE_UNKNOWN_REASON)

[`RETRY_EXTENSION_NAME`](#nemo_gym-eval-recovery-markers-RETRY_EXTENSION_NAME)

[`RETRY_FILE_NAME`](#nemo_gym-eval-recovery-markers-RETRY_FILE_NAME)

[`logger`](#nemo_gym-eval-recovery-markers-logger)

### API

```python
nemo_gym.eval.recovery.markers.record_failure_kind(
    error: BaseException
) -> None
```

Leave the registered kind `error` carries in the failure file the driver step names, for the next guard.

The driver step of a resumable job names the file in `DRIVER_FAILURE_ENV`; elsewhere this does nothing.

```python
nemo_gym.eval.recovery.markers.ACCOUNTED_WALLTIME_PREFIX = '__GYM_ACCOUNTED_WALLTIME:'
```

```python
nemo_gym.eval.recovery.markers.ACCOUNTING_UNKNOWN_PREFIX = 'GYM_ACCOUNTING_UNKNOWN:'
```

```python
nemo_gym.eval.recovery.markers.ACCUMULATED_WALLTIME_NAME = '.gym_accumulated_walltime'
```

```python
nemo_gym.eval.recovery.markers.CANCELLED_FOLLOWUP_MARKER = 'was cancelled before it started'
```

```python
nemo_gym.eval.recovery.markers.CHAIN_ACCOUNTING_ENV = 'GYM_CHAIN_ACCOUNTING'
```

```python
nemo_gym.eval.recovery.markers.CONTINUATION_REASON_PREFIX = '__GYM_CONTINUATION_REASON:'
```

```python
nemo_gym.eval.recovery.markers.DRIVER_FAILURE_ENV = 'GYM_DRIVER_FAILURE_FILE'
```

```python
nemo_gym.eval.recovery.markers.FOLLOW_UP_SUBMISSION_REFUSED = 'follow_up_submission_refused'
```

```python
nemo_gym.eval.recovery.markers.GUARDED_DRIVER_FAILURE_KINDS = (SERVER_SPINUP_TIMEOUT, DRIVER_INSTALL_FAILED, COLLECTION_RESUME_REFUSED, PROVID...
```

```python
nemo_gym.eval.recovery.markers.NO_ELIGIBLE_RETRY_REASON = 'No tasks are eligible for retry: the job ended its chain itself, with nothing q...
```

```python
nemo_gym.eval.recovery.markers.PROVISIONAL_MARKER_PREFIX = 'Continuation '
```

```python
nemo_gym.eval.recovery.markers.REQUEUE_RECONCILE_STATE = 'REQUEUED'
```

```python
nemo_gym.eval.recovery.markers.REQUEUE_UNKNOWN_REASON = 'Requeued allocation elapsed time is unknown; recover only after complete termin...
```

```python
nemo_gym.eval.recovery.markers.RETRY_EXTENSION_NAME = '.gym_infra_retry_extension'
```

```python
nemo_gym.eval.recovery.markers.RETRY_FILE_NAME = '.gym_infra_retries'
```

```python
nemo_gym.eval.recovery.markers.logger = logging.getLogger(__name__)
```