Skip to navigation

nemo_gym.agents.testing.checkpoint_conformance

View as MarkdownOpen in Claude

Conformance checks for harness adapters that checkpoint their agent sessions. Import them from tests.

check_harness_checkpointing drives an adapter through its real routes and checkpoint control routes, in process, with its real process supervisor, in a sandbox that is this machine. An adapter’s own tests call it with a HarnessFixture that runs a test harness in place of the adapter’s CLI.

Module Contents

Classes

NameDescription
HarnessFixtureWhat check_harness_checkpointing needs to know about one harness adapter and its test harness.
_LocalExecProviderA provider whose one “sandbox” is this machine: commands run in a local shell and files are local paths.

Functions

NameDescription
_control-
_crashEnd the agent whose tasks started after earlier was taken, as a crash of its server and sandbox would.
_kill-
_kill_sandbox_processesKill every process the provider’s commands started and every harness process, stopped ones included.
_until-
check_harness_checkpointingCheck a harness adapter’s checkpoint contract with its real supervisor; raise AssertionError on a violation.
process_stateThe state letter of process pid in /proc: R, S, T when stopped, Z for a zombie.

Data

_AUTH

_CRASH_TIMEOUT_S

_MARKER_VARIABLE

_TOKEN

API

class nemo_gym.agents.testing.checkpoint_conformance.HarnessFixture()

What check_harness_checkpointing needs to know about one harness adapter and its test harness.

The adapter runs a test harness in place of its CLI, in the sandbox of provider. The harness waits for finish() before it ends; harness_pids() lists its running processes, and started() how many launches have started, relaunches included.

provider
SandboxProvider

The sandbox provider the adapter must use: its one sandbox is this machine.

Commands run in a local shell and files are local paths, so the adapter’s real process supervisor, real signals and real files are exercised. Every agent make_agent returns must reach its sandbox through this provider: the crash check ends what the crashed agent ran through it.

nemo_gym.agents.testing.checkpoint_conformance.HarnessFixture.activation() -> dict[str, pydantic.JsonValue]

The /v1/responses body of the session’s activation.

nemo_gym.agents.testing.checkpoint_conformance.HarnessFixture.break_next_pause(
) -> None

Make the next pause of the harness fail to confirm, as a process stuck in disk I/O would.

nemo_gym.agents.testing.checkpoint_conformance.HarnessFixture.crash_recovery_assertion() -> collections.abc.Callable[[dict[str, JsonValue]], None] | None

Record the conversation a crash is about to interrupt; return the check that recovery continued it.

Called in the crash scenario only, once its checkpoint is committed and while the harness is still paused. The returned callable is called once, with the reply of the activation that recovered from the crash, and raises AssertionError unless that activation continued the conversation recorded here: the same conversation, no new user message, and progress made after the recovery. A harness that started the task from the beginning must fail it. Return None, the default, to check only that the harness was relaunched once.

nemo_gym.agents.testing.checkpoint_conformance.HarnessFixture.finish() -> None

Let every running and future launch of the test harness end.

nemo_gym.agents.testing.checkpoint_conformance.HarnessFixture.harness_pids() -> list[int]

The test harness processes that are still alive (see process_state).

nemo_gym.agents.testing.checkpoint_conformance.HarnessFixture.hold() -> None

Make launches wait again until the next finish.

nemo_gym.agents.testing.checkpoint_conformance.HarnessFixture.make_agent(
global_config: dict[str, pydantic.JsonValue]

The adapter, configured for global_config, running the test harness in the sandbox of provider.

nemo_gym.agents.testing.checkpoint_conformance.HarnessFixture.seed(
agent_session_id: str,
attempt: int
) -> dict[str, pydantic.JsonValue]

An agent seed body that borrows the local sandbox.

nemo_gym.agents.testing.checkpoint_conformance.HarnessFixture.started() -> int

How many launches of the test harness have started so far, relaunches included.

class nemo_gym.agents.testing.checkpoint_conformance._LocalExecProvider()

A provider whose one “sandbox” is this machine: commands run in a local shell and files are local paths.

For harness conformance tests, which need the real process supervisor, real signals and real files. Every handle names the same machine; connect accepts any descriptor this provider serialized. Every process a command starts inherits this provider’s marker in its environment, so processes finds what is left of them.

_generation
= 0
_marker
= uuid4().hex
commands
list[str] = []
name
= 'local-exec'
nemo_gym.agents.testing.checkpoint_conformance._LocalExecProvider._require_connected(
) -> None
nemo_gym.agents.testing.checkpoint_conformance._LocalExecProvider.aclose() -> None
async
nemo_gym.agents.testing.checkpoint_conformance._LocalExecProvider.close(
) -> None
async
nemo_gym.agents.testing.checkpoint_conformance._LocalExecProvider.connect(
descriptor: collections.abc.Mapping[str, typing.Any]
async
nemo_gym.agents.testing.checkpoint_conformance._LocalExecProvider.create(
async
nemo_gym.agents.testing.checkpoint_conformance._LocalExecProvider.disconnect_handles() -> None

Make every handle created or connected so far refuse commands and file transfers.

This is what a crashed agent server holds: it can no longer change the sandbox. Handles made afterwards work.

nemo_gym.agents.testing.checkpoint_conformance._LocalExecProvider.download_file(
source_path: str,
target_path: pathlib.Path
) -> None
async
nemo_gym.agents.testing.checkpoint_conformance._LocalExecProvider.exec(
command: str,
cwd: str | None = None,
env: dict[str, str] | None = None,
timeout_s: int | float | None = None,
user: str | int | None = None
async
nemo_gym.agents.testing.checkpoint_conformance._LocalExecProvider.processes() -> list[int]

The processes this provider’s commands started, with their descendants, that are still alive.

nemo_gym.agents.testing.checkpoint_conformance._LocalExecProvider.serialize_handle(
scope: str | None = None
) -> dict[str, pydantic.JsonValue]
async
nemo_gym.agents.testing.checkpoint_conformance._LocalExecProvider.status(
async
nemo_gym.agents.testing.checkpoint_conformance._LocalExecProvider.upload_file(
source_path: pathlib.Path,
target_path: str
) -> None
async
nemo_gym.agents.testing.checkpoint_conformance._control(
checkpoint_id: str,
extra: pydantic.JsonValue = {}
) -> dict[str, pydantic.JsonValue]
nemo_gym.agents.testing.checkpoint_conformance._crash(
earlier: set[asyncio.Task[typing.Any]]
) -> None
async

End the agent whose tasks started after earlier was taken, as a crash of its server and sandbox would.

The agent’s sandbox handles stop working first, so nothing it does while its tasks are cancelled can change the files the restored session finds. Then every process in the sandbox is killed, the harness and its supervisor among them, and every task the agent started is cancelled. Raises AssertionError if a process or a task is left.

nemo_gym.agents.testing.checkpoint_conformance._kill(
pid: int
) -> None
nemo_gym.agents.testing.checkpoint_conformance._kill_sandbox_processes(
timeout: float = 30
) -> list[int]
async

Kill every process the provider’s commands started and every harness process, stopped ones included.

Returns the processes still alive after timeout: an empty list when all are gone.

nemo_gym.agents.testing.checkpoint_conformance._until(
condition: collections.abc.Callable[[], object],
timeout: float = 120
) -> None
async
nemo_gym.agents.testing.checkpoint_conformance.check_harness_checkpointing(
state_dir: pathlib.Path
) -> None
async

Check a harness adapter’s checkpoint contract with its real supervisor; raise AssertionError on a violation.

Checked, through the adapter’s routes and checkpoint control routes:

  • a checkpoint pauses the running harness in place (every harness process stopped) and parks the activation; the commit exports the session, and the resume continues the same launch, which then finishes;
  • a re-sent activation gets the reply it already gave, without another launch;
  • a pause that is not confirmed stops the harness and parks the activation, and the resume relaunches it;
  • after a crash, a restored session closed before its replacement attempt re-binds it returns its saved evidence;
  • after a crash, the replacement attempt’s seed re-binds the restored session to the sandbox, and its re-sent activation relaunches the harness, which continues the saved conversation (HarnessFixture.crash_recovery_assertion);
  • a session closed while its activation runs stops the harness.

The crash ends the agent that took the checkpoint before the session is restored: its sandbox handles stop working, every process it started in the sandbox is killed, and its tasks are cancelled and must end. Whether the checks pass or fail, no process started through fixture.provider and no harness process is left running.

nemo_gym.agents.testing.checkpoint_conformance.process_state(
pid: int
) -> str | None

The state letter of process pid in /proc: R, S, T when stopped, Z for a zombie.

None when there is no such process. For a fixture’s harness_pids: a process whose state is None, Z or X is no longer alive.

nemo_gym.agents.testing.checkpoint_conformance._AUTH = {'Authorization': f'Bearer {_TOKEN}'}
nemo_gym.agents.testing.checkpoint_conformance._CRASH_TIMEOUT_S = 60.0
nemo_gym.agents.testing.checkpoint_conformance._MARKER_VARIABLE = 'NEMO_GYM_CONFORMANCE_SANDBOX'
nemo_gym.agents.testing.checkpoint_conformance._TOKEN = 'conformance-token'