Make a Harness Checkpointable
A partial-rollout checkpoint stops a training run at a consistent cut and lets it continue its unfinished rollouts
later. An agent that runs its harness inside a sandbox it borrows from the resources server takes part by pausing the
harness while the checkpoint is taken, and by resuming the harness’s saved conversation after a crash. OpenCode
(responses_api_agents/opencode_agent) is the reference adapter.
The resources server that lends the sandbox snapshots it; see Make a Resources Server Checkpointable. For the adapter contract this builds on, see Add or Update a Harness.
Capability tiers
An episode is as checkpointable as its weakest server: any seed reply with x-ng-checkpoint-restart: 1 makes it a
restart, so any pairing degrades safely.
Pause in place
Adapters built on SandboxSession (from nemo_gym.agents) run their harness under the bash process
supervisor. Construct the session with pausable=True and run each launch with run_with_checkpoint_pauses:
When a checkpoint opens, Activation.wait_park_requested() returns, run_with_checkpoint_pauses asks the supervisor
to pause (SIGSTOP to every process it reaches, detached services included), and parks the activation at on_parked until the
checkpoint resumes; then it continues the harness (SIGCONT) as if its model call were slow. Time paused does not count
against the harness timeout; pause_allowance_s extends the provider’s command deadline to cover it. OpenCode exposes
it as checkpoint_pause_allowance_seconds (default 3600) and the session root as sandbox_session_root (default
/tmp/nemo-gym-opencode-sessions); see its README. If the pause is not confirmed, or continuing fails three times, the
harness is stopped and run_with_checkpoint_pauses returns LaunchStopped (or LaunchFinished, if the harness ended
on its own first): on LaunchStopped, relaunch it, resuming its saved conversation.
Take the agent’s own concurrency slot with activation.slot(semaphore, deadline=...): a checkpoint counts the waiting
activation as parked, and the slot is given back on every path, including a retire or close while the activation is
parked.
Resume after a crash
Per-harness policy stays in the adapter:
- State location. Keep the harness’s home and data under the session root (
SandboxSession.root_dir), on the sandbox’s snapshotted disk. Each launch gets its own supervisor directory under that root (SandboxSession.begin_activation). - Export and restore.
export_agent_sessions(session_keys, *, checkpoint_id)returns each session’s bookmark, includingSandboxSession.checkpoint_state().restore_agent_sessionsinstalls sessions without a sandbox connection. - Re-bind. After a restore the Environment Server seeds the session again under the replacement attempt, with the
forked sandbox. Override
_rebind_restored_agent_session_stateto connect it and callSandboxSession.rebind. Declarerebinds_tool_accesses = Trueif the re-bound session takes the new seed’s tool grants, so episodes with resources MCP grants can continue instead of restarting. - Resume decision. On the first launch after a stop or a restore, read the saved conversation. Resume it when it is
the task’s own and the model answered at least once; otherwise start afresh from the original prompt. OpenCode
re-posts the last user message with empty
partsthroughopencode serve(reprompt_opencode.sh), so no turn is added. - Completed activation’s reply. Keep the reply of the activation that completed in the bookmark
(
completed_reply): a caller whose checkpoint precedes it sends the activation again after a restore and gets the same reply. - Launch count. Report how many launches an activation took (OpenCode:
metadata.harness_launches), so a run can tell pauses in place from relaunches. - Restarts. Return
Truefrom_seed_restarts(body)for a session the adapter cannot capture, such as one in a sandbox it owns, or one that continues an imported conversation.
Check it
check_harness_adapter (from nemo_gym.server_runtime) runs your adapter with its real supervisor on this
machine (LocalExecProvider) and a test harness in place of the CLI. Build a HarnessFixture (also from
nemo_gym.server_runtime) and call await check_harness_adapter(fixture, tmp_path / "state"). It checks pause → park → continue, a re-sent
activation, the stop fallback and relaunch, the close of a restored session that was never re-bound, re-bind and resume
after a crash, and a close while the harness runs. responses_api_agents/opencode_agent/tests/test_checkpoint.py shows
a fixture with a fake opencode executable.