> This page is for version Main (default).
> For other versions, use one of these documentation indexes:
> - Main (default): https://docs.nvidia.com/nemo/gym/main/llms.txt
> - 0.7.0: https://docs.nvidia.com/nemo/gym/v0.7.0/llms.txt
> - 0.6.0: https://docs.nvidia.com/nemo/gym/v0.6.0/llms.txt
> - 0.5.1: https://docs.nvidia.com/nemo/gym/v0.5.1/llms.txt
> - 0.5.0: https://docs.nvidia.com/nemo/gym/v0.5.0/llms.txt
> - 0.4.0: https://docs.nvidia.com/nemo/gym/v0.4.0/llms.txt
> - 0.3.0: https://docs.nvidia.com/nemo/gym/v0.3.0/llms.txt
> - 0.2.1: https://docs.nvidia.com/nemo/gym/v0.2.1/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/gym/llms.txt.

# nemo_gym.agents.testing.checkpoint_conformance

Conformance checks for harness adapters that checkpoint their agent sessions. Import them from tests.

`check_harness_checkpointing` drives an adapter through its real routes and checkpoint control routes, in
process, with its real process supervisor, in a sandbox that is this machine. An adapter's own tests call it with a
`HarnessFixture` that runs a test harness in place of the adapter's CLI.

## Module Contents

### Classes

| Name                                                                                       | Description                                                                                              |
| ------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------- |
| [`HarnessFixture`](#nemo_gym-agents-testing-checkpoint_conformance-HarnessFixture)         | What `check_harness_checkpointing` needs to know about one harness adapter and its test harness.         |
| [`_LocalExecProvider`](#nemo_gym-agents-testing-checkpoint_conformance-_LocalExecProvider) | A provider whose one "sandbox" is this machine: commands run in a local shell and files are local paths. |

### Functions

| Name                                                                                                         | Description                                                                                                    |
| ------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------- |
| [`_control`](#nemo_gym-agents-testing-checkpoint_conformance-_control)                                       | -                                                                                                              |
| [`_crash`](#nemo_gym-agents-testing-checkpoint_conformance-_crash)                                           | End the agent whose tasks started after `earlier` was taken, as a crash of its server and sandbox would.       |
| [`_kill`](#nemo_gym-agents-testing-checkpoint_conformance-_kill)                                             | -                                                                                                              |
| [`_kill_sandbox_processes`](#nemo_gym-agents-testing-checkpoint_conformance-_kill_sandbox_processes)         | Kill every process the provider's commands started and every harness process, stopped ones included.           |
| [`_until`](#nemo_gym-agents-testing-checkpoint_conformance-_until)                                           | -                                                                                                              |
| [`check_harness_checkpointing`](#nemo_gym-agents-testing-checkpoint_conformance-check_harness_checkpointing) | Check a harness adapter's checkpoint contract with its real supervisor; raise `AssertionError` on a violation. |
| [`process_state`](#nemo_gym-agents-testing-checkpoint_conformance-process_state)                             | The state letter of process `pid` in `/proc`: `R`, `S`, `T` when stopped, `Z` for a zombie.                    |

### Data

[`_AUTH`](#nemo_gym-agents-testing-checkpoint_conformance-_AUTH)

[`_CRASH_TIMEOUT_S`](#nemo_gym-agents-testing-checkpoint_conformance-_CRASH_TIMEOUT_S)

[`_MARKER_VARIABLE`](#nemo_gym-agents-testing-checkpoint_conformance-_MARKER_VARIABLE)

[`_TOKEN`](#nemo_gym-agents-testing-checkpoint_conformance-_TOKEN)

### API

```python
class nemo_gym.agents.testing.checkpoint_conformance.HarnessFixture()
```

What `check_harness_checkpointing` needs to know about one harness adapter and its test harness.

The adapter runs a test harness in place of its CLI, in the sandbox of `provider`. The harness waits for
`finish()` before it ends; `harness_pids()` lists its running processes, and `started()` how many launches
have started, relaunches included.

**`provider`** `SandboxProvider`

The sandbox provider the adapter must use: its one sandbox is this machine.

Commands run in a local shell and files are local paths, so the adapter's real process supervisor, real
signals and real files are exercised. Every agent `make_agent` returns must reach its sandbox through this
provider: the crash check ends what the crashed agent ran through it.

---

```python
nemo_gym.agents.testing.checkpoint_conformance.HarnessFixture.activation() -> dict[str, pydantic.JsonValue]
```

The `/v1/responses` body of the session's activation.

```python
nemo_gym.agents.testing.checkpoint_conformance.HarnessFixture.break_next_pause(
    agent: nemo_gym.agents.base_responses_api_agent.SimpleResponsesAPIAgent
) -> None
```

Make the next pause of the harness fail to confirm, as a process stuck in disk I/O would.

```python
nemo_gym.agents.testing.checkpoint_conformance.HarnessFixture.crash_recovery_assertion() -> collections.abc.Callable[[dict[str, JsonValue]], None] | None
```

Record the conversation a crash is about to interrupt; return the check that recovery continued it.

Called in the crash scenario only, once its checkpoint is committed and while the harness is still paused.
The returned callable is called once, with the reply of the activation that recovered from the crash, and
raises `AssertionError` unless that activation continued the conversation recorded here: the same
conversation, no new user message, and progress made after the recovery. A harness that started the task
from the beginning must fail it. Return `None`, the default, to check only that the harness was relaunched
once.

```python
nemo_gym.agents.testing.checkpoint_conformance.HarnessFixture.finish() -> None
```

Let every running and future launch of the test harness end.

```python
nemo_gym.agents.testing.checkpoint_conformance.HarnessFixture.harness_pids() -> list[int]
```

The test harness processes that are still alive (see `process_state`).

```python
nemo_gym.agents.testing.checkpoint_conformance.HarnessFixture.hold() -> None
```

Make launches wait again until the next `finish`.

```python
nemo_gym.agents.testing.checkpoint_conformance.HarnessFixture.make_agent(
    global_config: dict[str, pydantic.JsonValue]
) -> nemo_gym.agents.base_responses_api_agent.SimpleResponsesAPIAgent
```

The adapter, configured for `global_config`, running the test harness in the sandbox of `provider`.

```python
nemo_gym.agents.testing.checkpoint_conformance.HarnessFixture.seed(
    agent_session_id: str,
    attempt: int
) -> dict[str, pydantic.JsonValue]
```

An agent seed body that borrows the local sandbox.

```python
nemo_gym.agents.testing.checkpoint_conformance.HarnessFixture.started() -> int
```

How many launches of the test harness have started so far, relaunches included.

```python
class nemo_gym.agents.testing.checkpoint_conformance._LocalExecProvider()
```

A provider whose one "sandbox" is this machine: commands run in a local shell and files are local paths.

For harness conformance tests, which need the real process supervisor, real signals and real files. Every
handle names the same machine; `connect` accepts any descriptor this provider serialized. Every process a
command starts inherits this provider's marker in its environment, so `processes` finds what is left of them.

**`_generation`** `= 0`

---

**`_marker`** `= uuid4().hex`

---

**`commands`** `list[str] = []`

---

**`name`** `= 'local-exec'`

---

```python
nemo_gym.agents.testing.checkpoint_conformance._LocalExecProvider._require_connected(
    handle: nemo_gym.sandbox.providers.base.SandboxHandle
) -> None
```

```python
nemo_gym.agents.testing.checkpoint_conformance._LocalExecProvider.aclose() -> None
```

async

```python
nemo_gym.agents.testing.checkpoint_conformance._LocalExecProvider.close(
    handle: nemo_gym.sandbox.providers.base.SandboxHandle
) -> None
```

async

```python
nemo_gym.agents.testing.checkpoint_conformance._LocalExecProvider.connect(
    descriptor: collections.abc.Mapping[str, typing.Any]
) -> nemo_gym.sandbox.providers.base.SandboxHandle
```

async

```python
nemo_gym.agents.testing.checkpoint_conformance._LocalExecProvider.create(
    spec: nemo_gym.sandbox.providers.base.SandboxSpec
) -> nemo_gym.sandbox.providers.base.SandboxHandle
```

async

```python
nemo_gym.agents.testing.checkpoint_conformance._LocalExecProvider.disconnect_handles() -> None
```

Make every handle created or connected so far refuse commands and file transfers.

This is what a crashed agent server holds: it can no longer change the sandbox. Handles made afterwards work.

```python
nemo_gym.agents.testing.checkpoint_conformance._LocalExecProvider.download_file(
    handle: nemo_gym.sandbox.providers.base.SandboxHandle,
    source_path: str,
    target_path: pathlib.Path
) -> None
```

async

```python
nemo_gym.agents.testing.checkpoint_conformance._LocalExecProvider.exec(
    handle: nemo_gym.sandbox.providers.base.SandboxHandle,
    command: str,
    cwd: str | None = None,
    env: dict[str, str] | None = None,
    timeout_s: int | float | None = None,
    user: str | int | None = None
) -> nemo_gym.sandbox.providers.base.SandboxExecResult
```

async

```python
nemo_gym.agents.testing.checkpoint_conformance._LocalExecProvider.processes() -> list[int]
```

The processes this provider's commands started, with their descendants, that are still alive.

```python
nemo_gym.agents.testing.checkpoint_conformance._LocalExecProvider.serialize_handle(
    handle: nemo_gym.sandbox.providers.base.SandboxHandle,
    scope: str | None = None
) -> dict[str, pydantic.JsonValue]
```

async

```python
nemo_gym.agents.testing.checkpoint_conformance._LocalExecProvider.status(
    handle: nemo_gym.sandbox.providers.base.SandboxHandle
) -> nemo_gym.sandbox.providers.base.SandboxStatus
```

async

```python
nemo_gym.agents.testing.checkpoint_conformance._LocalExecProvider.upload_file(
    handle: nemo_gym.sandbox.providers.base.SandboxHandle,
    source_path: pathlib.Path,
    target_path: str
) -> None
```

async

```python
nemo_gym.agents.testing.checkpoint_conformance._control(
    checkpoint_id: str,
    extra: pydantic.JsonValue = {}
) -> dict[str, pydantic.JsonValue]
```

```python
nemo_gym.agents.testing.checkpoint_conformance._crash(
    provider: nemo_gym.agents.testing.checkpoint_conformance._LocalExecProvider,
    fixture: nemo_gym.agents.testing.checkpoint_conformance.HarnessFixture,
    earlier: set[asyncio.Task[typing.Any]]
) -> None
```

async

End the agent whose tasks started after `earlier` was taken, as a crash of its server and sandbox would.

The agent's sandbox handles stop working first, so nothing it does while its tasks are cancelled can change the
files the restored session finds. Then every process in the sandbox is killed, the harness and its supervisor
among them, and every task the agent started is cancelled. Raises `AssertionError` if a process or a task is
left.

```python
nemo_gym.agents.testing.checkpoint_conformance._kill(
    pid: int
) -> None
```

```python
nemo_gym.agents.testing.checkpoint_conformance._kill_sandbox_processes(
    provider: nemo_gym.agents.testing.checkpoint_conformance._LocalExecProvider,
    fixture: nemo_gym.agents.testing.checkpoint_conformance.HarnessFixture,
    timeout: float = 30
) -> list[int]
```

async

Kill every process the provider's commands started and every harness process, stopped ones included.

Returns the processes still alive after `timeout`: an empty list when all are gone.

```python
nemo_gym.agents.testing.checkpoint_conformance._until(
    condition: collections.abc.Callable[[], object],
    timeout: float = 120
) -> None
```

async

```python
nemo_gym.agents.testing.checkpoint_conformance.check_harness_checkpointing(
    fixture: nemo_gym.agents.testing.checkpoint_conformance.HarnessFixture,
    state_dir: pathlib.Path
) -> None
```

async

Check a harness adapter's checkpoint contract with its real supervisor; raise `AssertionError` on a violation.

Checked, through the adapter's routes and checkpoint control routes:

* a checkpoint pauses the running harness in place (every harness process stopped) and parks the activation;
  the commit exports the session, and the resume continues the same launch, which then finishes;
* a re-sent activation gets the reply it already gave, without another launch;
* a pause that is not confirmed stops the harness and parks the activation, and the resume relaunches it;
* after a crash, a restored session closed before its replacement attempt re-binds it returns its saved evidence;
* after a crash, the replacement attempt's seed re-binds the restored session to the sandbox, and its re-sent
  activation relaunches the harness, which continues the saved conversation
  (`HarnessFixture.crash_recovery_assertion`);
* a session closed while its activation runs stops the harness.

The crash ends the agent that took the checkpoint before the session is restored: its sandbox handles stop
working, every process it started in the sandbox is killed, and its tasks are cancelled and must end. Whether
the checks pass or fail, no process started through `fixture.provider` and no harness process is left running.

```python
nemo_gym.agents.testing.checkpoint_conformance.process_state(
    pid: int
) -> str | None
```

The state letter of process `pid` in `/proc`: `R`, `S`, `T` when stopped, `Z` for a zombie.

`None` when there is no such process. For a fixture's `harness_pids`: a process whose state is `None`,
`Z` or `X` is no longer alive.

```python
nemo_gym.agents.testing.checkpoint_conformance._AUTH = {'Authorization': f'Bearer {_TOKEN}'}
```

```python
nemo_gym.agents.testing.checkpoint_conformance._CRASH_TIMEOUT_S = 60.0
```

```python
nemo_gym.agents.testing.checkpoint_conformance._MARKER_VARIABLE = 'NEMO_GYM_CONFORMANCE_SANDBOX'
```

```python
nemo_gym.agents.testing.checkpoint_conformance._TOKEN = 'conformance-token'
```