> This page is for version Main (default).
> For other versions, use one of these documentation indexes:
> - Main (default): https://docs.nvidia.com/nemo/gym/main/llms.txt
> - 0.7.0: https://docs.nvidia.com/nemo/gym/v0.7.0/llms.txt
> - 0.6.0: https://docs.nvidia.com/nemo/gym/v0.6.0/llms.txt
> - 0.5.1: https://docs.nvidia.com/nemo/gym/v0.5.1/llms.txt
> - 0.5.0: https://docs.nvidia.com/nemo/gym/v0.5.0/llms.txt
> - 0.4.0: https://docs.nvidia.com/nemo/gym/v0.4.0/llms.txt
> - 0.3.0: https://docs.nvidia.com/nemo/gym/v0.3.0/llms.txt
> - 0.2.1: https://docs.nvidia.com/nemo/gym/v0.2.1/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/gym/llms.txt.

# nemo_gym.agents.utils.run_with_checkpoint_pauses

Run one supervised harness launch to its end, pausing the harness in place at every checkpoint.

## Module Contents

### Classes

| Name                                                                                       | Description                                                                                                  |
| ------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------ |
| [`LaunchFinished`](#nemo_gym-agents-utils-run_with_checkpoint_pauses-LaunchFinished)       | The launch ended on its own, possibly after pauses in place; `result` is what `launch` returned.             |
| [`LaunchStopped`](#nemo_gym-agents-utils-run_with_checkpoint_pauses-LaunchStopped)         | A checkpoint stopped the launch because pausing or continuing the harness failed, and the activation parked. |
| [`ParkingActivation`](#nemo_gym-agents-utils-run_with_checkpoint_pauses-ParkingActivation) | The agent checkpoint participant's handle for one running activation.                                        |

### Functions

| Name                                                                                                         | Description                                                                                            |
| ------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------ |
| [`_continue`](#nemo_gym-agents-utils-run_with_checkpoint_pauses-_continue)                                   | -                                                                                                      |
| [`_ended_by_stop`](#nemo_gym-agents-utils-run_with_checkpoint_pauses-_ended_by_stop)                         | Whether a launch this module stopped ended by that stop, not on its own.                               |
| [`_pause`](#nemo_gym-agents-utils-run_with_checkpoint_pauses-_pause)                                         | Pause the harness, repeating a request that failed in transport; an answer of the supervisor is final. |
| [`_retrieve_exception`](#nemo_gym-agents-utils-run_with_checkpoint_pauses-_retrieve_exception)               | Read a finished task's exception, so asyncio does not log it as never retrieved.                       |
| [`_retry_delay`](#nemo_gym-agents-utils-run_with_checkpoint_pauses-_retry_delay)                             | -                                                                                                      |
| [`_stop`](#nemo_gym-agents-utils-run_with_checkpoint_pauses-_stop)                                           | Stop the harness and wait for the launch to collect.                                                   |
| [`_wait_for_staging`](#nemo_gym-agents-utils-run_with_checkpoint_pauses-_wait_for_staging)                   | Return once the launch has ended and its staging no longer runs; warn every `stop_timeout` until then. |
| [`run_with_checkpoint_pauses`](#nemo_gym-agents-utils-run_with_checkpoint_pauses-run_with_checkpoint_pauses) | Run `launch` (the adapter's `session.execute(...)`), pausing its harness in place at every checkpoint. |

### Data

[`CONTINUE_ATTEMPTS`](#nemo_gym-agents-utils-run_with_checkpoint_pauses-CONTINUE_ATTEMPTS)

[`LOG`](#nemo_gym-agents-utils-run_with_checkpoint_pauses-LOG)

[`PAUSE_ATTEMPTS`](#nemo_gym-agents-utils-run_with_checkpoint_pauses-PAUSE_ATTEMPTS)

[`RETRY_DELAY_S`](#nemo_gym-agents-utils-run_with_checkpoint_pauses-RETRY_DELAY_S)

### API

```python
class nemo_gym.agents.utils.run_with_checkpoint_pauses.LaunchFinished(
    result: nemo_gym.agents.utils.run_with_checkpoint_pauses.LaunchFinished[T]
)
```

Dataclass

The launch ended on its own, possibly after pauses in place; `result` is what `launch` returned.

**`result`** `T`

---

```python
class nemo_gym.agents.utils.run_with_checkpoint_pauses.LaunchStopped(
    receipt: nemo_gym.agents.utils.supervisor_client.CleanupReceipt | None,
    error: BaseException | None = None
)
```

Dataclass

A checkpoint stopped the launch because pausing or continuing the harness failed, and the activation parked.

The harness did not finish: launch it again from the state it saved. `receipt` is the supervisor's cleanup
receipt; `error` is what the stopped launch raised while it collected its artifacts, if anything.

**`error`** `BaseException | None = None`

---

**`receipt`** `CleanupReceipt | None`

---

```python
class nemo_gym.agents.utils.run_with_checkpoint_pauses.ParkingActivation()
```

Protocol

The agent checkpoint participant's handle for one running activation.

Implemented by `nemo_gym.server_runtime._checkpoint.agent.Activation`.

**`park_requested`** `bool`

---

```python
nemo_gym.agents.utils.run_with_checkpoint_pauses.ParkingActivation.boundary(
    snapshot: collections.abc.Callable[[], dict[str, pydantic.JsonValue]]
) -> None
```

async

Park here until the checkpoint releases its participants; return at once when none asks.

```python
nemo_gym.agents.utils.run_with_checkpoint_pauses.ParkingActivation.queued() -> contextlib.AbstractAsyncContextManager[None]
```

```python
nemo_gym.agents.utils.run_with_checkpoint_pauses.ParkingActivation.wait_park_requested() -> None
```

async

Return once a checkpoint asks this activation to park; raise once its session is retired or closed.

```python
nemo_gym.agents.utils.run_with_checkpoint_pauses._continue(
    session: nemo_gym.agents.utils.sandbox_session.SandboxSession[typing.Any]
) -> bool
```

async

```python
nemo_gym.agents.utils.run_with_checkpoint_pauses._ended_by_stop(
    receipt: nemo_gym.agents.utils.supervisor_client.CleanupReceipt | None
) -> bool
```

Whether a launch this module stopped ended by that stop, not on its own.

The supervisor records the stop it served. A supervisor that died while stopping left no receipt; the one the
provider's proof stands in for (`receipt_missing`) has no `stopped`, and the harness did not finish either.

```python
nemo_gym.agents.utils.run_with_checkpoint_pauses._pause(
    session: nemo_gym.agents.utils.sandbox_session.SandboxSession[typing.Any]
) -> nemo_gym.agents.utils.sandbox_session.HarnessPauseResult
```

async

Pause the harness, repeating a request that failed in transport; an answer of the supervisor is final.

A request that timed out is not repeated: the provider had no capacity for it, or the supervisor did not
answer within the pause timeout, and neither clears in a few seconds.

```python
nemo_gym.agents.utils.run_with_checkpoint_pauses._retrieve_exception(
    task: asyncio.Future[typing.Any]
) -> None
```

Read a finished task's exception, so asyncio does not log it as never retrieved.

```python
nemo_gym.agents.utils.run_with_checkpoint_pauses._retry_delay(
    attempt: int
) -> None
```

async

```python
nemo_gym.agents.utils.run_with_checkpoint_pauses._stop(
    session: nemo_gym.agents.utils.sandbox_session.SandboxSession[typing.Any],
    launched: asyncio.Future[nemo_gym.agents.utils.run_with_checkpoint_pauses._stop[T]],
    stop_timeout: float
) -> nemo_gym.agents.utils.run_with_checkpoint_pauses.LaunchFinished[nemo_gym.agents.utils.run_with_checkpoint_pauses._stop[T]] | nemo_gym.agents.utils.run_with_checkpoint_pauses.LaunchStopped
```

async

Stop the harness and wait for the launch to collect.

A harness that ended on its own first is `LaunchFinished`. A launch that had not started its harness is cancelled, and an error of a launch no stop ended propagates.

```python
nemo_gym.agents.utils.run_with_checkpoint_pauses._wait_for_staging(
    session: nemo_gym.agents.utils.sandbox_session.SandboxSession[typing.Any],
    launched: asyncio.Future[typing.Any],
    stop_timeout: float
) -> None
```

async

Return once the launch has ended and its staging no longer runs; warn every `stop_timeout` until then.

Not bounded: while the caller waits here its activation is unfinished, so a checkpoint waits for the session
and fails at its own deadline instead of taking a snapshot beside the staging's writes. A retire or a close of
the session cancels the wait.

```python
nemo_gym.agents.utils.run_with_checkpoint_pauses.run_with_checkpoint_pauses(
    session: nemo_gym.agents.utils.sandbox_session.SandboxSession[typing.Any],
    activation: nemo_gym.agents.utils.run_with_checkpoint_pauses.ParkingActivation,
    launch: collections.abc.Callable[[], collections.abc.Awaitable[nemo_gym.agents.utils.run_with_checkpoint_pauses.run_with_checkpoint_pauses[T]]],
    on_parked: collections.abc.Callable[[], dict[str, pydantic.JsonValue]],
    stop_timeout: float
) -> nemo_gym.agents.utils.run_with_checkpoint_pauses.LaunchFinished[nemo_gym.agents.utils.run_with_checkpoint_pauses.run_with_checkpoint_pauses[T]] | nemo_gym.agents.utils.run_with_checkpoint_pauses.LaunchStopped
```

async

Run `launch` (the adapter's `session.execute(...)`), pausing its harness in place at every checkpoint.

When a checkpoint asks the activation to park, the harness is paused in place and the activation parks at
`on_parked`, the adapter's boundary snapshot, until the checkpoint releases it; then the harness is continued
and the launch keeps running for later checkpoints. A pause stops the harness's processes, not its clocks: a
tool that runs across the pause sees the paused time, so its own timeout can fire when the harness continues.
Model calls wait at the model server and are not affected.

A pause or continue request that fails in transport is repeated after a delay, with the same sequence number,
so the supervisor serves it once. If the pause is still not confirmed, or the continue is refused by the
supervisor or fails `CONTINUE_ATTEMPTS` times, the harness is stopped: the supervisor sends TERM and, after the cleanup grace
period, KILL to what is left, so a harness that ignores TERM costs the whole grace period. The activation
parks after a pause failure, and `LaunchStopped` tells the adapter to launch again. A supervisor that died
during this stop, leaving a receipt the provider had to prove (`receipt_missing`), counts as stopped too. A
launch that ends on its own, also as a park arrives, returns `LaunchFinished`. An error of `launch`
propagates, and a launch that is still staging and does not end when cancelled raises: the activation must not
park while anything of it can still write to the sandbox. Nor may it fail meanwhile, since a failed activation
no longer holds up the checkpoint: any error is raised only once the launch has ended and its staging no longer
runs, however long that takes. Until then the checkpoint waits for the session, up to its own deadline.

Cancelled, for example when its session is retired or closed, it stops the harness, shielded and bounded by
`stop_timeout`, before it re-raises.

```python
nemo_gym.agents.utils.run_with_checkpoint_pauses.CONTINUE_ATTEMPTS = 3
```

```python
nemo_gym.agents.utils.run_with_checkpoint_pauses.LOG = logging.getLogger(__name__)
```

```python
nemo_gym.agents.utils.run_with_checkpoint_pauses.PAUSE_ATTEMPTS = 4
```

```python
nemo_gym.agents.utils.run_with_checkpoint_pauses.RETRY_DELAY_S = 1.0
```