nemo_gym.agents.testing.checkpoint_conformance
nemo_gym.agents.testing.checkpoint_conformance
Conformance checks for harness adapters that checkpoint their agent sessions. Import them from tests.
check_harness_checkpointing drives an adapter through its real routes and checkpoint control routes, in
process, with its real process supervisor, in a sandbox that is this machine. An adapter’s own tests call it with a
HarnessFixture that runs a test harness in place of the adapter’s CLI.
Module Contents
Classes
Functions
Data
API
What check_harness_checkpointing needs to know about one harness adapter and its test harness.
The adapter runs a test harness in place of its CLI, in the sandbox of provider. The harness waits for
finish() before it ends; harness_pids() lists its running processes, and started() how many launches
have started, relaunches included.
The sandbox provider the adapter must use: its one sandbox is this machine.
Commands run in a local shell and files are local paths, so the adapter’s real process supervisor, real
signals and real files are exercised. Every agent make_agent returns must reach its sandbox through this
provider: the crash check ends what the crashed agent ran through it.
The /v1/responses body of the session’s activation.
Make the next pause of the harness fail to confirm, as a process stuck in disk I/O would.
Record the conversation a crash is about to interrupt; return the check that recovery continued it.
Called in the crash scenario only, once its checkpoint is committed and while the harness is still paused.
The returned callable is called once, with the reply of the activation that recovered from the crash, and
raises AssertionError unless that activation continued the conversation recorded here: the same
conversation, no new user message, and progress made after the recovery. A harness that started the task
from the beginning must fail it. Return None, the default, to check only that the harness was relaunched
once.
Let every running and future launch of the test harness end.
The test harness processes that are still alive (see process_state).
Make launches wait again until the next finish.
The adapter, configured for global_config, running the test harness in the sandbox of provider.
An agent seed body that borrows the local sandbox.
How many launches of the test harness have started so far, relaunches included.
A provider whose one “sandbox” is this machine: commands run in a local shell and files are local paths.
For harness conformance tests, which need the real process supervisor, real signals and real files. Every
handle names the same machine; connect accepts any descriptor this provider serialized. Every process a
command starts inherits this provider’s marker in its environment, so processes finds what is left of them.
Make every handle created or connected so far refuse commands and file transfers.
This is what a crashed agent server holds: it can no longer change the sandbox. Handles made afterwards work.
The processes this provider’s commands started, with their descendants, that are still alive.
End the agent whose tasks started after earlier was taken, as a crash of its server and sandbox would.
The agent’s sandbox handles stop working first, so nothing it does while its tasks are cancelled can change the
files the restored session finds. Then every process in the sandbox is killed, the harness and its supervisor
among them, and every task the agent started is cancelled. Raises AssertionError if a process or a task is
left.
Kill every process the provider’s commands started and every harness process, stopped ones included.
Returns the processes still alive after timeout: an empty list when all are gone.
Check a harness adapter’s checkpoint contract with its real supervisor; raise AssertionError on a violation.
Checked, through the adapter’s routes and checkpoint control routes:
- a checkpoint pauses the running harness in place (every harness process stopped) and parks the activation; the commit exports the session, and the resume continues the same launch, which then finishes;
- a re-sent activation gets the reply it already gave, without another launch;
- a pause that is not confirmed stops the harness and parks the activation, and the resume relaunches it;
- after a crash, a restored session closed before its replacement attempt re-binds it returns its saved evidence;
- after a crash, the replacement attempt’s seed re-binds the restored session to the sandbox, and its re-sent
activation relaunches the harness, which continues the saved conversation
(
HarnessFixture.crash_recovery_assertion); - a session closed while its activation runs stops the harness.
The crash ends the agent that took the checkpoint before the session is restored: its sandbox handles stop
working, every process it started in the sandbox is killed, and its tasks are cancelled and must end. Whether
the checks pass or fail, no process started through fixture.provider and no harness process is left running.
The state letter of process pid in /proc: R, S, T when stopped, Z for a zombie.
None when there is no such process. For a fixture’s harness_pids: a process whose state is None,
Z or X is no longer alive.