nemo_gym.agents.utils.run_with_checkpoint_pauses
nemo_gym.agents.utils.run_with_checkpoint_pauses
Run one supervised harness launch to its end, pausing the harness in place at every checkpoint.
Module Contents
Classes
Functions
Data
API
The launch ended on its own, possibly after pauses in place; result is what launch returned.
A checkpoint stopped the launch because pausing or continuing the harness failed, and the activation parked.
The harness did not finish: launch it again from the state it saved. receipt is the supervisor’s cleanup
receipt; error is what the stopped launch raised while it collected its artifacts, if anything.
The agent checkpoint participant’s handle for one running activation.
Implemented by nemo_gym.server_runtime._checkpoint.agent.Activation.
Park here until the checkpoint releases its participants; return at once when none asks.
Return once a checkpoint asks this activation to park; raise once its session is retired or closed.
Whether a launch this module stopped ended by that stop, not on its own.
The supervisor records the stop it served. A supervisor that died while stopping left no receipt; the one the
provider’s proof stands in for (receipt_missing) has no stopped, and the harness did not finish either.
Pause the harness, repeating a request that failed in transport; an answer of the supervisor is final.
A request that timed out is not repeated: the provider had no capacity for it, or the supervisor did not answer within the pause timeout, and neither clears in a few seconds.
Read a finished task’s exception, so asyncio does not log it as never retrieved.
Stop the harness and wait for the launch to collect.
A harness that ended on its own first is LaunchFinished. A launch that had not started its harness is cancelled, and an error of a launch no stop ended propagates.
Return once the launch has ended and its staging no longer runs; warn every stop_timeout until then.
Not bounded: while the caller waits here its activation is unfinished, so a checkpoint waits for the session and fails at its own deadline instead of taking a snapshot beside the staging’s writes. A retire or a close of the session cancels the wait.
Run launch (the adapter’s session.execute(...)), pausing its harness in place at every checkpoint.
When a checkpoint asks the activation to park, the harness is paused in place and the activation parks at
on_parked, the adapter’s boundary snapshot, until the checkpoint releases it; then the harness is continued
and the launch keeps running for later checkpoints. A pause stops the harness’s processes, not its clocks: a
tool that runs across the pause sees the paused time, so its own timeout can fire when the harness continues.
Model calls wait at the model server and are not affected.
A pause or continue request that fails in transport is repeated after a delay, with the same sequence number,
so the supervisor serves it once. If the pause is still not confirmed, or the continue is refused by the
supervisor or fails CONTINUE_ATTEMPTS times, the harness is stopped: the supervisor sends TERM and, after the cleanup grace
period, KILL to what is left, so a harness that ignores TERM costs the whole grace period. The activation
parks after a pause failure, and LaunchStopped tells the adapter to launch again. A supervisor that died
during this stop, leaving a receipt the provider had to prove (receipt_missing), counts as stopped too. A
launch that ends on its own, also as a park arrives, returns LaunchFinished. An error of launch
propagates, and a launch that is still staging and does not end when cancelled raises: the activation must not
park while anything of it can still write to the sandbox. Nor may it fail meanwhile, since a failed activation
no longer holds up the checkpoint: any error is raised only once the launch has ended and its staging no longer
runs, however long that takes. Until then the checkpoint waits for the session, up to its own deadline.
Cancelled, for example when its session is retired or closed, it stops the harness, shielded and bounded by
stop_timeout, before it re-raises.