Make a Resources Server Checkpointable
A partial-rollout checkpoint lets a training run stop and later continue its unfinished rollouts instead of starting
them over. Each server takes part for the state it owns. This page is for a resources server whose episode seed starts
a task sandbox that the agent works in, as SWE-bench’s does. A server whose session state is a small JSON object
declares checkpoint_mode = "exported" and implements export_session_states and restore_session_states directly;
see resources_servers/example_session_state_mgmt. For the agent side, see
Make a Harness Checkpointable.
Build on CheckpointedSandboxSessionResourcesServer
SandboxSessionResourcesServer (resources_servers/swebench/sandbox_sessions.py) serves the episode sessions of a
server whose seed starts one task sandbox: the episode seed lends the sandbox through sandbox_access, verify captures
the agent’s patch once and stops the sandbox, and close stops it and fences a late seed.
CheckpointedSandboxSessionResourcesServer (resources_servers/swebench/sandbox_session_checkpoints.py) adds the
checkpoint half and leaves that flow as it is. A subclass:
Your seed_session and verify routes call the base’s helpers: seed_task_sandbox_session(request, body, MyTask)
for an episode seed, and verify_task_sandbox_session(request, body, verify=..., lost_response=...), whose verify
callback captures the patch inside capture_task_sandbox_patch(session_id, capture) and then calls
stop_task_sandbox_after_capture(session_id). Close is the base’s close_resources_session.
resources_servers/swebench/app.py shows all three.
sandbox_checkpoint settings:
What a checkpoint holds
A commit snapshots each session’s task sandbox through the provider and indexes the snapshot under
<checkpoint.state_dir>/<server>/snapshots/<checkpoint id>/. It exports the seed, the agent’s working directory, the
patch verify captured (if it has) and the task state. The snapshot is the restore point: a restore forks it into a new
sandbox, whatever happened to the original since. The snapshot keeps files, not processes.
- Grade a captured patch, not a live sandbox. Declare
checkpoint_verify = "replay": a checkpoint does not wait for grading, and a verify that runs again after a restore grades the saved patch instead of capturing it again. A session past its capture exports no sandbox. Declare it only if/verifyuses the task sandbox solely insidecapture_task_sandbox_patchand stops it solely withstop_task_sandbox_after_capture: both hold the lock a commit holds while it snapshots. What the capture returns must be a JSON value, because the checkpoint stores it. - In-place graders that need processes the agent started declare
checkpoint_mode = "restart_only". The seed reply says so, and the episode starts over after a crash instead of continuing in a fork that has none of those processes. - After a restore, the environment server seeds the session again under the replacement attempt. The session answers with the forked sandbox and does not set the task up again.
Requirements
- A sandbox provider that can snapshot (
SupportsSandboxSnapshot; OpenSandbox today). On any other provider the server reportsrestart_only, says so on its seed reply, and its episodes start over after a crash; commits still succeed. checkpoint.state_dir: an absolute path every restart of the run can read, outside anything the controller deletes. A server that exports its sessions refuses to start with checkpointing on and nostate_dir, or a relative one.- One
state_dirper job and its restarts: two jobs running at the same time that share one delete each other’s snapshots at their releases, so put the job’s ID in the path. Itsflockmust hold across every node that runs the server (on Lustre, mount with-o flock). - The controller calls
release(kept)after each checkpoint it saves, andrelease([])at the end of a run that will not be resumed. A job that ended for good without that final release leaves its last snapshots. Their names start withng-<state dir id>-, where the id is the content of<state_dir>/state_dir_id; on OpenSandbox, delete them as OpenSandbox Provider describes.
Checkpoint timings and outcomes are reported as Partial-Rollout Checkpoint Metrics.
Check it
check_sandbox_session_checkpoints (from resources_servers.swebench.testing.checkpoint_conformance) drives your server through its real routes
against an in-process provider with durable snapshots: seed, retry and conflicts, commit, restore into a fork, the
replacement attempt’s seed, the captured patch surviving a restore, release (including snapshots a crashed process
left), close, shutdown, and restart_only on a provider that cannot snapshot.
make_server(provider, global_config) builds your server on the provider it is given (shared_provider() must return
it) with global_config as its global configuration; episode_seed() is one ResourcesSeedSessionRequest your server
accepts.
resources_servers/swebench/tests/test_checkpoint.py is a complete example.