Skip to navigation

Make a Harness Checkpointable

View as MarkdownOpen in Claude

A partial-rollout checkpoint stops a training run at a consistent cut and lets it continue its unfinished rollouts later. An agent that runs its harness inside a sandbox it borrows from the resources server takes part by pausing the harness while the checkpoint is taken, and by resuming the harness’s saved conversation after a crash. OpenCode (responses_api_agents/opencode_agent) is the reference adapter.

The resources server that lends the sandbox snapshots it; see Make a Resources Server Checkpointable. For the adapter contract this builds on, see Add or Update a Harness.

Capability tiers

TierHarnessEpisode after a crash
0restart_only (the default)Starts over from its input
1Paused in place at commits; after a crash, resumed with one extra “continue” turnContinues, with a turn the task never had
2Paused in place; resumed exactly, with no extra turnContinues

An episode is as checkpointable as its weakest server: any seed reply with x-ng-checkpoint-restart: 1 makes it a restart, so any pairing degrades safely.

Pause in place

Adapters built on SandboxSession (from nemo_gym.agents) run their harness under the bash process supervisor. Construct the session with pausable=True and run each launch with run_with_checkpoint_pauses:

from nemo_gym.agents import LaunchFinished, LaunchStopped, run_with_checkpoint_pauses
async with self.checkpoint_participant.activation(session_key, episode_id, index=index) as activation:
outcome = await run_with_checkpoint_pauses(
session,
activation,
launch=lambda: session.execute(..., pause_allowance_s=pause_allowance),
on_parked=lambda: {"launch": launch, "conversation": conversation_id},
stop_timeout=close_timeout,
)

When a checkpoint opens, Activation.wait_park_requested() returns, run_with_checkpoint_pauses asks the supervisor to pause (SIGSTOP to every process it reaches, detached services included), and parks the activation at on_parked until the checkpoint resumes; then it continues the harness (SIGCONT) as if its model call were slow. Time paused does not count against the harness timeout; pause_allowance_s extends the provider’s command deadline to cover it. OpenCode exposes it as checkpoint_pause_allowance_seconds (default 3600) and the session root as sandbox_session_root (default /tmp/nemo-gym-opencode-sessions); see its README. If the pause is not confirmed, or continuing fails three times, the harness is stopped and run_with_checkpoint_pauses returns LaunchStopped (or LaunchFinished, if the harness ended on its own first): on LaunchStopped, relaunch it, resuming its saved conversation.

Take the agent’s own concurrency slot with activation.slot(semaphore, deadline=...): a checkpoint counts the waiting activation as parked, and the slot is given back on every path, including a retire or close while the activation is parked.

Resume after a crash

Per-harness policy stays in the adapter:

  • State location. Keep the harness’s home and data under the session root (SandboxSession.root_dir), on the sandbox’s snapshotted disk. Each launch gets its own supervisor directory under that root (SandboxSession.begin_activation).
  • Export and restore. export_agent_sessions(session_keys, *, checkpoint_id) returns each session’s bookmark, including SandboxSession.checkpoint_state(). restore_agent_sessions installs sessions without a sandbox connection.
  • Re-bind. After a restore the Environment Server seeds the session again under the replacement attempt, with the forked sandbox. Override _rebind_restored_agent_session_state to connect it and call SandboxSession.rebind. Declare rebinds_tool_accesses = True if the re-bound session takes the new seed’s tool grants, so episodes with resources MCP grants can continue instead of restarting.
  • Resume decision. On the first launch after a stop or a restore, read the saved conversation. Resume it when it is the task’s own and the model answered at least once; otherwise start afresh from the original prompt. OpenCode re-posts the last user message with empty parts through opencode serve (reprompt_opencode.sh), so no turn is added.
  • Completed activation’s reply. Keep the reply of the activation that completed in the bookmark (completed_reply): a caller whose checkpoint precedes it sends the activation again after a restore and gets the same reply.
  • Launch count. Report how many launches an activation took (OpenCode: metadata.harness_launches), so a run can tell pauses in place from relaunches.
  • Restarts. Return True from _seed_restarts(body) for a session the adapter cannot capture, such as one in a sandbox it owns, or one that continues an imported conversation.

Check it

check_harness_adapter (from nemo_gym.server_runtime) runs your adapter with its real supervisor on this machine (LocalExecProvider) and a test harness in place of the CLI. Build a HarnessFixture (also from nemo_gym.server_runtime) and call await check_harness_adapter(fixture, tmp_path / "state"). It checks pause → park → continue, a re-sent activation, the stop fallback and relaunch, the close of a restored session that was never re-bound, re-bind and resume after a crash, and a close while the harness runs. responses_api_agents/opencode_agent/tests/test_checkpoint.py shows a fixture with a fake opencode executable.