nemo_gym.orchestration.server_supervisor

Bases: ServerStartFailedError

The head server’s thread is running but it never answered within head_server_spinup_timeout_seconds.

Not resumable: the CLI exits with stop_exit_code (76), which a job chain reads as “another attempt cannot help”, instead of holding its allocation in the wait.

Bases: RuntimeError

Gym servers did not become ready within server_spinup_timeout_seconds.

Bases: RuntimeError

A Gym server process (or the head server) exited before it was ready.

failure_kind is server_start_failed. errno is the errno of the bind error the server printed, when it printed one: errno.EADDRINUSE means another process took its port between the port check and the bind.

Bases: NamedTuple

Forward a server’s output pipe to sink as it arrives, and keep its last max_bytes.

The operator still sees every line live, and the supervisor can read the error a server printed before it exited, such as the bind error that names an occupied port.

nemo_gym.orchestration.server_supervisor._OutputTail._forward(
    chunk: bytes
) -> None

nemo_gym.orchestration.server_supervisor._OutputTail._pump() -> None

nemo_gym.orchestration.server_supervisor._OutputTail.text(
    timeout_s: float = 5.0
) -> str

The errno of the bind error a server printed on stderr, or None when it printed none.

The upstream model endpoints named in the resolved config, as (config key, url) pairs.

wait_for_spinup only polls Gym’s own servers, and the Gym-side model server answers as soon as it starts whether or not anything is behind it, so these are the URLs nothing checks.

base_url is typed Union[str, List[str]] on vllm_model and the local vLLM servers, where the list form spreads load across replicas. The local ones default to an empty list and are filled in after Gym launches vLLM, so an empty list means “not yet” rather than “misconfigured” and is skipped, as is an unset value.

The key travels with the url because the endpoint that fails may be a judge or user model, and a message naming policy_base_url would then be wrong.

Yield each server instance in the config that has an entrypoint to launch.

What to GET to find out whether base_url is being served.

GET /v1/models is part of the OpenAI API, so most endpoints behind a /v1 base URL answer it, but nothing here depends on that: any HTTP response other than a gateway status counts as answering, so an endpoint without it replies 404 and still passes. URLs that do not end in /v1 are probed at their root.

Whether a requests connection error was a name that does not resolve.

requests reports this as a ConnectionError like any other, with the resolver failure nested inside, so the cause chain has to be walked rather than the class inspected.

How long to wait for model endpoints, from config. 0 or a negative value skips the check.

An unset key falls back to the same default the config parser applies. A value that is not a number is a configuration mistake, so it is reported as one rather than surfacing as a ValueError traceback from float().

The errno of binding host:port now, as uvicorn binds it, or None when the port is free.

Whether anything answers at base_url, and if not, whether waiting could help.

Answering is the bar, not healthy: a 401 or 404 means something is there, and requiring a 200 would reject endpoints that need auth. The exception is 502, 503 and 504, which a proxy or load balancer returns while the server behind it is still starting; those are waited on like a refused connection. A completed TLS handshake counts too, even against a certificate this process does not trust, which is why SSLError is checked before ConnectionError it inherits from.

The probe connects directly, ignoring HTTP_PROXY and the like, because model requests go through Gym’s aiohttp client, which ignores them too. Through a proxy, an endpoint the proxy cannot reach would come back as 502 or 504 and be waited on until the timeout.

Names of the configured servers that declare ray_enabled = True, so this run must start or join Ray.

A declaration that can’t be read from the entrypoint’s source, such as one inherited from another module, doesn’t count. Such a server fails at startup with an error asking it to declare ray_enabled = True itself.

Resolve a relative server dir (e.g. resources_servers/<name>) to an absolute path.

Searches NEMO_GYM_EXTRA_ROOTS, the current working directory (a user’s local server), then the Gym install root (PARENT_DIR) where built-in servers live in both editable and wheel installs. A directory counts as a server only if it ships an install marker for one of our two venv setups. This lets gym env test find and run built-in (and plugin) servers from any cwd, not just a repo checkout.

Shell command that sets up a server’s venv and starts its entrypoint with that venv’s own interpreter.

The interpreter is named explicitly instead of relying on the bare python that source bin/activate puts on PATH: a venv copied or moved after creation still names its original prefix in bin/activate, so activating it would silently run a different interpreter than uv_venv_dir selected.

The middle line records when venv setup finished. The server uses it to separate venv setup from interpreter start and import time in its startup stages (see nemo_gym.telemetry.startup). It reads the clock with the venv’s own interpreter because date +%s%N does not work on macOS.

The finite shared grace period for managed server process groups.

How long to wait for Gym servers to become ready, from config. 0 or a negative value waits forever.

An unset or null key falls back to the same default the config parser applies. Environment interpolation yields a string, so numeric strings are accepted. Anything else is reported as a ConfigError rather than a TypeError traceback mid-startup.

How a startup that raised error ended, as the startup telemetry reports it.

Wait for every endpoint to answer. Returns the ones that never did.

Unresolvable names are reported once and not waited on. Refused connections and gateway statuses are waited on, because an inference server can take minutes to load weights and someone who starts thirty seconds early should not have to start over.