Skip to navigation

nemo_gym.agents.utils.run_with_checkpoint_pauses

View as MarkdownOpen in Claude

Run one supervised harness launch to its end, pausing the harness in place at every checkpoint.

Module Contents

Classes

NameDescription
LaunchFinishedThe launch ended on its own, possibly after pauses in place; result is what launch returned.
LaunchStoppedA checkpoint stopped the launch because pausing or continuing the harness failed, and the activation parked.
ParkingActivationThe agent checkpoint participant’s handle for one running activation.

Functions

NameDescription
_continue-
_ended_by_stopWhether a launch this module stopped ended by that stop, not on its own.
_pausePause the harness, repeating a request that failed in transport; an answer of the supervisor is final.
_retrieve_exceptionRead a finished task’s exception, so asyncio does not log it as never retrieved.
_retry_delay-
_stopStop the harness and wait for the launch to collect.
_wait_for_stagingReturn once the launch has ended and its staging no longer runs; warn every stop_timeout until then.
run_with_checkpoint_pausesRun launch (the adapter’s session.execute(...)), pausing its harness in place at every checkpoint.

Data

CONTINUE_ATTEMPTS

LOG

PAUSE_ATTEMPTS

RETRY_DELAY_S

API

Dataclass

The launch ended on its own, possibly after pauses in place; result is what launch returned.

result
T
class nemo_gym.agents.utils.run_with_checkpoint_pauses.LaunchStopped(
error: BaseException | None = None
)
Dataclass

A checkpoint stopped the launch because pausing or continuing the harness failed, and the activation parked.

The harness did not finish: launch it again from the state it saved. receipt is the supervisor’s cleanup receipt; error is what the stopped launch raised while it collected its artifacts, if anything.

error
BaseException | None = None
receipt
CleanupReceipt | None
class nemo_gym.agents.utils.run_with_checkpoint_pauses.ParkingActivation()
Protocol

The agent checkpoint participant’s handle for one running activation.

Implemented by nemo_gym.server_runtime._checkpoint.agent.Activation.

park_requested
bool
nemo_gym.agents.utils.run_with_checkpoint_pauses.ParkingActivation.boundary(
snapshot: collections.abc.Callable[[], dict[str, pydantic.JsonValue]]
) -> None
async

Park here until the checkpoint releases its participants; return at once when none asks.

nemo_gym.agents.utils.run_with_checkpoint_pauses.ParkingActivation.queued() -> contextlib.AbstractAsyncContextManager[None]
nemo_gym.agents.utils.run_with_checkpoint_pauses.ParkingActivation.wait_park_requested() -> None
async

Return once a checkpoint asks this activation to park; raise once its session is retired or closed.

nemo_gym.agents.utils.run_with_checkpoint_pauses._continue(
) -> bool
async
nemo_gym.agents.utils.run_with_checkpoint_pauses._ended_by_stop(
) -> bool

Whether a launch this module stopped ended by that stop, not on its own.

The supervisor records the stop it served. A supervisor that died while stopping left no receipt; the one the provider’s proof stands in for (receipt_missing) has no stopped, and the harness did not finish either.

nemo_gym.agents.utils.run_with_checkpoint_pauses._pause(
async

Pause the harness, repeating a request that failed in transport; an answer of the supervisor is final.

A request that timed out is not repeated: the provider had no capacity for it, or the supervisor did not answer within the pause timeout, and neither clears in a few seconds.

nemo_gym.agents.utils.run_with_checkpoint_pauses._retrieve_exception(
task: asyncio.Future[typing.Any]
) -> None

Read a finished task’s exception, so asyncio does not log it as never retrieved.

nemo_gym.agents.utils.run_with_checkpoint_pauses._retry_delay(
attempt: int
) -> None
async
async

Stop the harness and wait for the launch to collect.

A harness that ended on its own first is LaunchFinished. A launch that had not started its harness is cancelled, and an error of a launch no stop ended propagates.

nemo_gym.agents.utils.run_with_checkpoint_pauses._wait_for_staging(
launched: asyncio.Future[typing.Any],
stop_timeout: float
) -> None
async

Return once the launch has ended and its staging no longer runs; warn every stop_timeout until then.

Not bounded: while the caller waits here its activation is unfinished, so a checkpoint waits for the session and fails at its own deadline instead of taking a snapshot beside the staging’s writes. A retire or a close of the session cancels the wait.

async

Run launch (the adapter’s session.execute(...)), pausing its harness in place at every checkpoint.

When a checkpoint asks the activation to park, the harness is paused in place and the activation parks at on_parked, the adapter’s boundary snapshot, until the checkpoint releases it; then the harness is continued and the launch keeps running for later checkpoints. A pause stops the harness’s processes, not its clocks: a tool that runs across the pause sees the paused time, so its own timeout can fire when the harness continues. Model calls wait at the model server and are not affected.

A pause or continue request that fails in transport is repeated after a delay, with the same sequence number, so the supervisor serves it once. If the pause is still not confirmed, or the continue is refused by the supervisor or fails CONTINUE_ATTEMPTS times, the harness is stopped: the supervisor sends TERM and, after the cleanup grace period, KILL to what is left, so a harness that ignores TERM costs the whole grace period. The activation parks after a pause failure, and LaunchStopped tells the adapter to launch again. A supervisor that died during this stop, leaving a receipt the provider had to prove (receipt_missing), counts as stopped too. A launch that ends on its own, also as a park arrives, returns LaunchFinished. An error of launch propagates, and a launch that is still staging and does not end when cancelled raises: the activation must not park while anything of it can still write to the sandbox. Nor may it fail meanwhile, since a failed activation no longer holds up the checkpoint: any error is raised only once the launch has ended and its staging no longer runs, however long that takes. Until then the checkpoint waits for the session, up to its own deadline.

Cancelled, for example when its session is retired or closed, it stops the harness, shielded and bounded by stop_timeout, before it re-raises.

nemo_gym.agents.utils.run_with_checkpoint_pauses.CONTINUE_ATTEMPTS = 3
nemo_gym.agents.utils.run_with_checkpoint_pauses.LOG = logging.getLogger(__name__)
nemo_gym.agents.utils.run_with_checkpoint_pauses.PAUSE_ATTEMPTS = 4
nemo_gym.agents.utils.run_with_checkpoint_pauses.RETRY_DELAY_S = 1.0