nemo_gym.agents.utils.supervisor_client

Controller-side transport for the shared sandbox process supervisor.

Session ownership and cancellation live in sandbox_session.py; output parsing stays with the adapter. Unlike process_supervisor.sh, this module is not uploaded to the task sandbox.

Session control files (all paths are relative to session_dir):

Filename	Writer	When
process_supervisor.sh	Session	Uploaded before launch
launch.claim	Launch or stop shell	Atomically claims launch or fences it
supervisor.pid	Launch shell	After claiming launch, before exec (see below)
stop.request	Stop shell	Polled by the supervisor to stop the harness
expire.request	Session	Polled by the supervisor to time out the harness
cleanup.json	Supervisor or stop	After cleanup, or when stop fences launch
output.log	Launch shell	Captures supervisor and harness diagnostics

Agents in the task sandbox often clean up with pkill -f python3, pkill python3 or killall python3. These must not stop the supervisor or a runner, which would end the harness. The supervisor and the shared harness runner are bash scripts, so their names never mention python; a harness argv that runs its own python (Hermes, NOOA, Stirrup) still appears in the supervisor’s command line. Gym stops the supervisor through the stop file, which does not depend on these names.

supervisor.pid holds the PID that executes the supervisor. When the provider did not make the launch shell a process group leader, that process starts the supervisor in a new group, passes TERM on to it and exits with its status.

Module Contents

Classes

Name	Description
`CleanupReceipt`	Outcome after bounded harness termination, as written by process_supervisor.sh.

Functions

Name	Description
`_raise_terminal_failure`	-
`_seconds`	Seconds as the supervisor parses them: fixed-point, at least one centisecond.
`exec_timeout`	Reserve TERM grace, direct-process reaping, and process-group reaping.
`harness_termination_failure`	The harness boundary’s failure for a negative cleanup receipt.
`is_harness_termination_failure`	Whether `error` reports that a harness’s termination was not confirmed.
`parse_cleanup_receipt`	Require positive cleanup evidence, tolerating malformed optional diagnostics.
`remove_session_directory`	Remove adapter-owned files after cleanup, keeping delayed launches fenced.
`stop_and_confirm_cleanup`	Fence a pending launch or require explicit supervisor acknowledgement before teardown.
`supervised_launch_command`	Fence delayed launches and run a harness command under the shared supervisor.
`supervision_timeouts`	Return the per-cleanup-phase and provider execution deadlines.

Data

CLEANUP_PHASE_COUNT

CLEANUP_RECEIPT_FILE

DEFAULT_CLEANUP_TIMEOUT

EXPIRE_REQUEST_FILE

SUPERVISOR_PID_FILE

SUPERVISOR_SCRIPT

_CLEANUP_RECEIPT

API

class nemo_gym.agents.utils.supervisor_client.CleanupReceipt

Bases: typing.TypedDict

Outcome after bounded harness termination, as written by process_supervisor.sh.

cleanup_confirmed means no live harness-group members remain. After a forced stop it also means that no process the stop reached still runs: the harness’s descendants, including tools in their own sessions, and processes that inherited the harness’s NEMO_GYM_SUPERVISOR marker without having left the harness before the stop. Processes that left the harness before the stop, and a process that left between two scans without the marker, remain until the owner closes the sandbox; this is not a sandbox-empty check. return_code is the shell status: 128+N for a harness killed by signal N, and 126 or 127 when the command could not be run.

cleanup_confirmed

bool

error

str | None

return_code

int | None

timed_out

bool

nemo_gym.agents.utils.supervisor_client._raise_terminal_failure(
    payload: object,
    harness: str
) -> None

nemo_gym.agents.utils.supervisor_client._seconds(
    value: float
) -> str

Seconds as the supervisor parses them: fixed-point, at least one centisecond.

nemo_gym.agents.utils.supervisor_client.exec_timeout(
    timeout: float,
    cleanup_timeout: float = DEFAULT_CLEANUP_TIMEOUT
) -> float

Reserve TERM grace, direct-process reaping, and process-group reaping.

The extra 30 seconds cover startup, polling, and receipt I/O beyond those bounded cleanup phases. Keep their count shared with the controller budget.

nemo_gym.agents.utils.supervisor_client.harness_termination_failure(
    harness: str,
    receipt: nemo_gym.agents.utils.supervisor_client.CleanupReceipt
) -> nemo_gym.core.typed_exceptions.NgFailureEvent

The harness boundary’s failure for a negative cleanup receipt.

Its event names the harness but not the receipt’s text, which stays in str() and on receipt (runtime evidence for teardown, not part of the serialized event).

nemo_gym.agents.utils.supervisor_client.is_harness_termination_failure(
    error: BaseException
) -> bool

Whether error reports that a harness’s termination was not confirmed.

nemo_gym.agents.utils.supervisor_client.parse_cleanup_receipt(
    payload: object
) -> nemo_gym.agents.utils.supervisor_client.CleanupReceipt

Require positive cleanup evidence, tolerating malformed optional diagnostics.

nemo_gym.agents.utils.supervisor_client.remove_session_directory(
    sandbox: nemo_gym.sandbox.api.AsyncSandbox,
    session_dir: str,
    workdir: str | None,
    timeout: float,
    harness: str
) -> None

async

Remove adapter-owned files after cleanup, keeping delayed launches fenced.

Retire the directory atomically before unlinking its claim. Otherwise a delayed launch could win the claim while recursive deletion is in progress. A failed removal leaves the retired path available for a close retry.

nemo_gym.agents.utils.supervisor_client.stop_and_confirm_cleanup(
    sandbox: nemo_gym.sandbox.api.AsyncSandbox,
    session_dir: str,
    workdir: str | None,
    timeout: float,
    harness: str
) -> nemo_gym.agents.utils.supervisor_client.CleanupReceipt

async

Fence a pending launch or require explicit supervisor acknowledgement before teardown.

A stop that wins the launch claim writes the same receipt shape as the supervisor, with no return code because no harness process ran.

nemo_gym.agents.utils.supervisor_client.supervised_launch_command(
    session_dir: str,
    command: list[str],
    timeout: float,
    cleanup_timeout: float
) -> str

Fence delayed launches and run a harness command under the shared supervisor.

The supervisor needs only bash. Check that bash can parse it before claiming a launch: a failed bootstrap cannot write a cleanup receipt. Supervisor and harness diagnostics share output.log. The supervisor leaves the provider’s process group and stops the harness group, and after a forced stop the processes it reached; the sandbox owner stops services that left the harness earlier. Stops reach the supervisor through the stop file, not a signal, so the harness keeps the default TERM disposition.

nemo_gym.agents.utils.supervisor_client.supervision_timeouts(
    timeout: float,
    close_timeout: float
) -> tuple[float, float]

Return the per-cleanup-phase and provider execution deadlines.

Divide the close budget across TERM grace, direct-process and group reaping. The provider deadline reserves all phases after the harness timeout, plus 30 seconds for startup, polling, and receipt I/O.

nemo_gym.agents.utils.supervisor_client.CLEANUP_PHASE_COUNT = 3

nemo_gym.agents.utils.supervisor_client.CLEANUP_RECEIPT_FILE = 'cleanup.json'

nemo_gym.agents.utils.supervisor_client.DEFAULT_CLEANUP_TIMEOUT = 10.0

nemo_gym.agents.utils.supervisor_client.EXPIRE_REQUEST_FILE = 'expire.request'

nemo_gym.agents.utils.supervisor_client.LAUNCH_CLAIM_FILE = 'launch.claim'

nemo_gym.agents.utils.supervisor_client.LOG = logging.getLogger(__name__)

nemo_gym.agents.utils.supervisor_client.OUTPUT_LOG_FILE = 'output.log'

nemo_gym.agents.utils.supervisor_client.STOP_REQUEST_FILE = 'stop.request'

nemo_gym.agents.utils.supervisor_client.SUPERVISOR_FILE = 'process_supervisor.sh'

nemo_gym.agents.utils.supervisor_client.SUPERVISOR_PID_FILE = 'supervisor.pid'

nemo_gym.agents.utils.supervisor_client.SUPERVISOR_SCRIPT = Path(__file__).with_name(SUPERVISOR_FILE)

nemo_gym.agents.utils.supervisor_client._CLEANUP_RECEIPT = TypeAdapter(CleanupReceipt)