> This page is for version 0.3.0.
> For other versions, use one of these documentation indexes:
> - Main (default): https://docs.nvidia.com/nemo/gym/main/llms.txt
> - 0.7.0: https://docs.nvidia.com/nemo/gym/v0.7.0/llms.txt
> - 0.6.0: https://docs.nvidia.com/nemo/gym/v0.6.0/llms.txt
> - 0.5.1: https://docs.nvidia.com/nemo/gym/v0.5.1/llms.txt
> - 0.5.0: https://docs.nvidia.com/nemo/gym/v0.5.0/llms.txt
> - 0.4.0: https://docs.nvidia.com/nemo/gym/v0.4.0/llms.txt
> - 0.3.0: https://docs.nvidia.com/nemo/gym/v0.3.0/llms.txt
> - 0.2.1: https://docs.nvidia.com/nemo/gym/v0.2.1/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/gym/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/gym/_mcp/server.

# nemo_gym.agents.utils.supervisor_client

Controller-side transport for the shared sandbox process supervisor.

Session ownership and cancellation live in sandbox_session.py; output parsing stays with the adapter.
Unlike process_supervisor.sh, this module is not uploaded to the task sandbox.

Session control files (all paths are relative to session_dir):

| Filename              | Writer               | When                                             |
| --------------------- | -------------------- | ------------------------------------------------ |
| process_supervisor.sh | Session              | Uploaded before launch                           |
| launch.claim          | Launch or stop shell | Atomically claims launch or fences it            |
| supervisor.pid        | Launch shell         | After claiming launch, before exec (see below)   |
| stop.request          | Stop shell           | Polled by the supervisor to stop the harness     |
| expire.request        | Session              | Polled by the supervisor to time out the harness |
| cleanup.json          | Supervisor or stop   | After cleanup, or when stop fences launch        |
| output.log            | Launch shell         | Captures supervisor and harness diagnostics      |

Agents in the task sandbox often clean up with `pkill -f python3`, `pkill python3` or
`killall python3`. These must not stop the supervisor or a runner, which would end the
harness. The supervisor and the shared harness runner are bash scripts, so their names
never mention python; a harness argv that runs its own python (Hermes, NOOA, Stirrup) still
appears in the supervisor's command line. Gym stops the supervisor through the stop file,
which does not depend on these names.

`supervisor.pid` holds the PID that executes the supervisor. When the provider did not
make the launch shell a process group leader, that process starts the supervisor in a new
group, passes TERM on to it and exits with its status.

## Module Contents

### Classes

| Name                                                                        | Description                                                                     |
| --------------------------------------------------------------------------- | ------------------------------------------------------------------------------- |
| [`CleanupReceipt`](#nemo_gym-agents-utils-supervisor_client-CleanupReceipt) | Outcome after bounded harness termination, as written by process_supervisor.sh. |

### Functions

| Name                                                                                                        | Description                                                                            |
| ----------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------- |
| [`_raise_terminal_failure`](#nemo_gym-agents-utils-supervisor_client-_raise_terminal_failure)               | -                                                                                      |
| [`_seconds`](#nemo_gym-agents-utils-supervisor_client-_seconds)                                             | Seconds as the supervisor parses them: fixed-point, at least one centisecond.          |
| [`exec_timeout`](#nemo_gym-agents-utils-supervisor_client-exec_timeout)                                     | Reserve TERM grace, direct-process reaping, and process-group reaping.                 |
| [`harness_termination_failure`](#nemo_gym-agents-utils-supervisor_client-harness_termination_failure)       | The harness boundary's failure for a negative cleanup receipt.                         |
| [`is_harness_termination_failure`](#nemo_gym-agents-utils-supervisor_client-is_harness_termination_failure) | Whether `error` reports that a harness's termination was not confirmed.                |
| [`parse_cleanup_receipt`](#nemo_gym-agents-utils-supervisor_client-parse_cleanup_receipt)                   | Require positive cleanup evidence, tolerating malformed optional diagnostics.          |
| [`remove_session_directory`](#nemo_gym-agents-utils-supervisor_client-remove_session_directory)             | Remove adapter-owned files after cleanup, keeping delayed launches fenced.             |
| [`stop_and_confirm_cleanup`](#nemo_gym-agents-utils-supervisor_client-stop_and_confirm_cleanup)             | Fence a pending launch or require explicit supervisor acknowledgement before teardown. |
| [`supervised_launch_command`](#nemo_gym-agents-utils-supervisor_client-supervised_launch_command)           | Fence delayed launches and run a harness command under the shared supervisor.          |
| [`supervision_timeouts`](#nemo_gym-agents-utils-supervisor_client-supervision_timeouts)                     | Return the per-cleanup-phase and provider execution deadlines.                         |

### Data

[`CLEANUP_PHASE_COUNT`](#nemo_gym-agents-utils-supervisor_client-CLEANUP_PHASE_COUNT)

[`CLEANUP_RECEIPT_FILE`](#nemo_gym-agents-utils-supervisor_client-CLEANUP_RECEIPT_FILE)

[`DEFAULT_CLEANUP_TIMEOUT`](#nemo_gym-agents-utils-supervisor_client-DEFAULT_CLEANUP_TIMEOUT)

[`EXPIRE_REQUEST_FILE`](#nemo_gym-agents-utils-supervisor_client-EXPIRE_REQUEST_FILE)

[`LAUNCH_CLAIM_FILE`](#nemo_gym-agents-utils-supervisor_client-LAUNCH_CLAIM_FILE)

[`LOG`](#nemo_gym-agents-utils-supervisor_client-LOG)

[`OUTPUT_LOG_FILE`](#nemo_gym-agents-utils-supervisor_client-OUTPUT_LOG_FILE)

[`STOP_REQUEST_FILE`](#nemo_gym-agents-utils-supervisor_client-STOP_REQUEST_FILE)

[`SUPERVISOR_FILE`](#nemo_gym-agents-utils-supervisor_client-SUPERVISOR_FILE)

[`SUPERVISOR_PID_FILE`](#nemo_gym-agents-utils-supervisor_client-SUPERVISOR_PID_FILE)

[`SUPERVISOR_SCRIPT`](#nemo_gym-agents-utils-supervisor_client-SUPERVISOR_SCRIPT)

[`_CLEANUP_RECEIPT`](#nemo_gym-agents-utils-supervisor_client-_CLEANUP_RECEIPT)

### API

```python
class nemo_gym.agents.utils.supervisor_client.CleanupReceipt
```

**Bases:** `typing.TypedDict`

Outcome after bounded harness termination, as written by process_supervisor.sh.

`cleanup_confirmed` means no live harness-group members remain. After a forced
stop it also means that no process the stop reached still runs: the harness's
descendants, including tools in their own sessions, and processes that inherited
the harness's `NEMO_GYM_SUPERVISOR` marker without having left the harness
before the stop. Processes that left the harness before the stop, and a process
that left between two scans without the marker, remain until the owner closes the
sandbox; this is not a sandbox-empty check. `return_code` is the shell status:
128+N for a harness killed by signal N, and 126 or 127 when the command could not
be run.

**`cleanup_confirmed`** `bool`

---

**`error`** `str | None`

---

**`return_code`** `int | None`

---

**`timed_out`** `bool`

---

```python
nemo_gym.agents.utils.supervisor_client._raise_terminal_failure(
    payload: object,
    harness: str
) -> None
```

```python
nemo_gym.agents.utils.supervisor_client._seconds(
    value: float
) -> str
```

Seconds as the supervisor parses them: fixed-point, at least one centisecond.

```python
nemo_gym.agents.utils.supervisor_client.exec_timeout(
    timeout: float,
    cleanup_timeout: float = DEFAULT_CLEANUP_TIMEOUT
) -> float
```

Reserve TERM grace, direct-process reaping, and process-group reaping.

The extra 30 seconds cover startup, polling, and receipt I/O beyond those
bounded cleanup phases. Keep their count shared with the controller budget.

```python
nemo_gym.agents.utils.supervisor_client.harness_termination_failure(
    harness: str,
    receipt: nemo_gym.agents.utils.supervisor_client.CleanupReceipt
) -> nemo_gym.core.typed_exceptions.NgFailureEvent
```

The harness boundary's failure for a negative cleanup receipt.

Its event names the harness but not the receipt's text, which stays in `str()` and on `receipt`
(runtime evidence for teardown, not part of the serialized event).

```python
nemo_gym.agents.utils.supervisor_client.is_harness_termination_failure(
    error: BaseException
) -> bool
```

Whether `error` reports that a harness's termination was not confirmed.

```python
nemo_gym.agents.utils.supervisor_client.parse_cleanup_receipt(
    payload: object
) -> nemo_gym.agents.utils.supervisor_client.CleanupReceipt
```

Require positive cleanup evidence, tolerating malformed optional diagnostics.

```python
nemo_gym.agents.utils.supervisor_client.remove_session_directory(
    sandbox: nemo_gym.sandbox.api.AsyncSandbox,
    session_dir: str,
    workdir: str | None,
    timeout: float,
    harness: str
) -> None
```

async

Remove adapter-owned files after cleanup, keeping delayed launches fenced.

Retire the directory atomically before unlinking its claim. Otherwise a
delayed launch could win the claim while recursive deletion is in progress.
A failed removal leaves the retired path available for a close retry.

```python
nemo_gym.agents.utils.supervisor_client.stop_and_confirm_cleanup(
    sandbox: nemo_gym.sandbox.api.AsyncSandbox,
    session_dir: str,
    workdir: str | None,
    timeout: float,
    harness: str
) -> nemo_gym.agents.utils.supervisor_client.CleanupReceipt
```

async

Fence a pending launch or require explicit supervisor acknowledgement before teardown.

A stop that wins the launch claim writes the same receipt shape as the
supervisor, with no return code because no harness process ran.

```python
nemo_gym.agents.utils.supervisor_client.supervised_launch_command(
    session_dir: str,
    command: list[str],
    timeout: float,
    cleanup_timeout: float
) -> str
```

Fence delayed launches and run a harness command under the shared supervisor.

The supervisor needs only bash. Check that bash can parse it before claiming a
launch: a failed bootstrap cannot write a cleanup receipt. Supervisor and harness
diagnostics share output.log. The supervisor leaves the provider's process group and
stops the harness group, and after a forced stop the processes it reached; the
sandbox owner stops services that left the harness earlier. Stops reach the
supervisor through the stop file, not a signal, so the harness keeps the default
TERM disposition.

```python
nemo_gym.agents.utils.supervisor_client.supervision_timeouts(
    timeout: float,
    close_timeout: float
) -> tuple[float, float]
```

Return the per-cleanup-phase and provider execution deadlines.

Divide the close budget across TERM grace, direct-process and group reaping.
The provider deadline reserves all phases after
the harness timeout, plus 30 seconds for startup, polling, and receipt I/O.

```python
nemo_gym.agents.utils.supervisor_client.CLEANUP_PHASE_COUNT = 3
```

```python
nemo_gym.agents.utils.supervisor_client.CLEANUP_RECEIPT_FILE = 'cleanup.json'
```

```python
nemo_gym.agents.utils.supervisor_client.DEFAULT_CLEANUP_TIMEOUT = 10.0
```

```python
nemo_gym.agents.utils.supervisor_client.EXPIRE_REQUEST_FILE = 'expire.request'
```

```python
nemo_gym.agents.utils.supervisor_client.LAUNCH_CLAIM_FILE = 'launch.claim'
```

```python
nemo_gym.agents.utils.supervisor_client.LOG = logging.getLogger(__name__)
```

```python
nemo_gym.agents.utils.supervisor_client.OUTPUT_LOG_FILE = 'output.log'
```

```python
nemo_gym.agents.utils.supervisor_client.STOP_REQUEST_FILE = 'stop.request'
```

```python
nemo_gym.agents.utils.supervisor_client.SUPERVISOR_FILE = 'process_supervisor.sh'
```

```python
nemo_gym.agents.utils.supervisor_client.SUPERVISOR_PID_FILE = 'supervisor.pid'
```

```python
nemo_gym.agents.utils.supervisor_client.SUPERVISOR_SCRIPT = Path(__file__).with_name(SUPERVISOR_FILE)
```

```python
nemo_gym.agents.utils.supervisor_client._CLEANUP_RECEIPT = TypeAdapter(CleanupReceipt)
```