> This page is for version 0.3.0. > For other versions, use one of these documentation indexes: > - Main (default): https://docs.nvidia.com/nemo/gym/main/llms.txt > - 0.7.0: https://docs.nvidia.com/nemo/gym/v0.7.0/llms.txt > - 0.6.0: https://docs.nvidia.com/nemo/gym/v0.6.0/llms.txt > - 0.5.1: https://docs.nvidia.com/nemo/gym/v0.5.1/llms.txt > - 0.5.0: https://docs.nvidia.com/nemo/gym/v0.5.0/llms.txt > - 0.4.0: https://docs.nvidia.com/nemo/gym/v0.4.0/llms.txt > - 0.3.0: https://docs.nvidia.com/nemo/gym/v0.3.0/llms.txt > - 0.2.1: https://docs.nvidia.com/nemo/gym/v0.2.1/llms.txt > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/nemo/gym/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/gym/_mcp/server. # nemo_gym.orchestration.server_supervisor Starting, watching, and stopping the servers of a Gym run: the supervisor behind `gym env start`. ## Module Contents ### Classes | Name | Description | | ------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------ | | [`HeadServerUnresponsiveError`](#nemo_gym-orchestration-server_supervisor-HeadServerUnresponsiveError) | The head server's thread is running but it never answered within `head_server_spinup_timeout_seconds`. | | [`RunHelper`](#nemo_gym-orchestration-server_supervisor-RunHelper) | - | | [`ServerSpinupTimeoutError`](#nemo_gym-orchestration-server_supervisor-ServerSpinupTimeoutError) | Gym servers did not become ready within `server_spinup_timeout_seconds`. | | [`ServerStartFailedError`](#nemo_gym-orchestration-server_supervisor-ServerStartFailedError) | A Gym server process (or the head server) exited before it was ready. | | [`_ConfiguredServer`](#nemo_gym-orchestration-server_supervisor-_ConfiguredServer) | - | | [`_OutputTail`](#nemo_gym-orchestration-server_supervisor-_OutputTail) | Forward a server's output pipe to `sink` as it arrives, and keep its last `max_bytes`. | ### Functions | Name | Description | | ---------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------ | | [`_bind_errno`](#nemo_gym-orchestration-server_supervisor-_bind_errno) | The errno of the bind error a server printed on stderr, or None when it printed none. | | [`_collect_model_endpoints`](#nemo_gym-orchestration-server_supervisor-_collect_model_endpoints) | The upstream model endpoints named in the resolved config, as (config key, url) pairs. | | [`_configured_servers`](#nemo_gym-orchestration-server_supervisor-_configured_servers) | Yield each server instance in the config that has an entrypoint to launch. | | [`_endpoint_probe_url`](#nemo_gym-orchestration-server_supervisor-_endpoint_probe_url) | What to GET to find out whether `base_url` is being served. | | [`_head_server_spinup_timeout_seconds`](#nemo_gym-orchestration-server_supervisor-_head_server_spinup_timeout_seconds) | - | | [`_is_dns_failure`](#nemo_gym-orchestration-server_supervisor-_is_dns_failure) | Whether a `requests` connection error was a name that does not resolve. | | [`_model_endpoint_timeout_seconds`](#nemo_gym-orchestration-server_supervisor-_model_endpoint_timeout_seconds) | How long to wait for model endpoints, from config. 0 or a negative value skips the check. | | [`_port_bind_errno`](#nemo_gym-orchestration-server_supervisor-_port_bind_errno) | The errno of binding `host:port` now, as uvicorn binds it, or None when the port is free. | | [`_probe_endpoint`](#nemo_gym-orchestration-server_supervisor-_probe_endpoint) | Whether anything answers at `base_url`, and if not, whether waiting could help. | | [`_ray_server_names`](#nemo_gym-orchestration-server_supervisor-_ray_server_names) | Names of the configured servers that declare `ray_enabled = True`, so this run must start or join Ray. | | [`_resolve_server_dir`](#nemo_gym-orchestration-server_supervisor-_resolve_server_dir) | Resolve a relative server dir (e.g. `resources_servers/<name>`) to an absolute path. | | [`_server_group_exists`](#nemo_gym-orchestration-server_supervisor-_server_group_exists) | - | | [`_server_launch_command`](#nemo_gym-orchestration-server_supervisor-_server_launch_command) | Shell command that sets up a server's venv and starts its entrypoint with that venv's own interpreter. | | [`_server_shutdown_timeout_seconds`](#nemo_gym-orchestration-server_supervisor-_server_shutdown_timeout_seconds) | The finite shared grace period for managed server process groups. | | [`_server_spinup_timeout_seconds`](#nemo_gym-orchestration-server_supervisor-_server_spinup_timeout_seconds) | How long to wait for Gym servers to become ready, from config. 0 or a negative value waits forever. | | [`_signal_server_group`](#nemo_gym-orchestration-server_supervisor-_signal_server_group) | - | | [`_startup_outcome`](#nemo_gym-orchestration-server_supervisor-_startup_outcome) | How a startup that raised `error` ended, as the startup telemetry reports it. | | [`_wait_for_model_endpoints`](#nemo_gym-orchestration-server_supervisor-_wait_for_model_endpoints) | Wait for every endpoint to answer. Returns the ones that never did. | ### Data [`HEAD_SERVER_SPINUP_TIMEOUT_SECONDS_KEY_NAME`](#nemo_gym-orchestration-server_supervisor-HEAD_SERVER_SPINUP_TIMEOUT_SECONDS_KEY_NAME) [`RAY_ENABLE_AUTO_CONNECT_ENV_VAR_NAME`](#nemo_gym-orchestration-server_supervisor-RAY_ENABLE_AUTO_CONNECT_ENV_VAR_NAME) [`_BASE_URL_KEY_SUFFIX`](#nemo_gym-orchestration-server_supervisor-_BASE_URL_KEY_SUFFIX) [`_BIND_ERRNO_PATTERN`](#nemo_gym-orchestration-server_supervisor-_BIND_ERRNO_PATTERN) [`_DEFAULT_HEAD_SERVER_SPINUP_TIMEOUT_SEC`](#nemo_gym-orchestration-server_supervisor-_DEFAULT_HEAD_SERVER_SPINUP_TIMEOUT_SEC) [`_DEFAULT_MODEL_ENDPOINT_READINESS_TIMEOUT_SEC`](#nemo_gym-orchestration-server_supervisor-_DEFAULT_MODEL_ENDPOINT_READINESS_TIMEOUT_SEC) [`_DEFAULT_SERVER_SPINUP_TIMEOUT_SEC`](#nemo_gym-orchestration-server_supervisor-_DEFAULT_SERVER_SPINUP_TIMEOUT_SEC) [`_ENDPOINT_ANSWERING`](#nemo_gym-orchestration-server_supervisor-_ENDPOINT_ANSWERING) [`_ENDPOINT_POLL_INTERVAL_SEC`](#nemo_gym-orchestration-server_supervisor-_ENDPOINT_POLL_INTERVAL_SEC) [`_ENDPOINT_PROBE_TIMEOUT_SEC`](#nemo_gym-orchestration-server_supervisor-_ENDPOINT_PROBE_TIMEOUT_SEC) [`_ENDPOINT_REFUSED`](#nemo_gym-orchestration-server_supervisor-_ENDPOINT_REFUSED) [`_ENDPOINT_STARTING`](#nemo_gym-orchestration-server_supervisor-_ENDPOINT_STARTING) [`_ENDPOINT_STARTING_STATUS_CODES`](#nemo_gym-orchestration-server_supervisor-_ENDPOINT_STARTING_STATUS_CODES) [`_ENDPOINT_UNRESOLVABLE`](#nemo_gym-orchestration-server_supervisor-_ENDPOINT_UNRESOLVABLE) [`_FORCE_KILL_REAP_TIMEOUT_SEC`](#nemo_gym-orchestration-server_supervisor-_FORCE_KILL_REAP_TIMEOUT_SEC) [`_GRACEFUL_SHUTDOWN_TIMEOUT_SEC`](#nemo_gym-orchestration-server_supervisor-_GRACEFUL_SHUTDOWN_TIMEOUT_SEC) [`_MODEL_SERVER_TYPE`](#nemo_gym-orchestration-server_supervisor-_MODEL_SERVER_TYPE) [`_SERVER_OUTPUT_TAIL_BYTES`](#nemo_gym-orchestration-server_supervisor-_SERVER_OUTPUT_TAIL_BYTES) [`_SERVER_PROBE_TIMEOUT_SEC`](#nemo_gym-orchestration-server_supervisor-_SERVER_PROBE_TIMEOUT_SEC) ### API ```python class nemo_gym.orchestration.server_supervisor.HeadServerUnresponsiveError() ``` **Bases:** [ServerStartFailedError](#nemo_gym-orchestration-server_supervisor-ServerStartFailedError) The head server's thread is running but it never answered within `head_server_spinup_timeout_seconds`. Not resumable: the CLI exits with `stop_exit_code` (76), which a job chain reads as "another attempt cannot help", instead of holding its allocation in the wait. **`stop_exit_code`** `= 76` --- ```python class nemo_gym.orchestration.server_supervisor.RunHelper() ``` **`_h2_ping_sidecar`** `H2PingSidecarManager | None` --- **`_head_server`** `Server` --- **`_head_server_instance`** `HeadServer` --- **`_head_server_thread`** `Thread` --- **`_memory_profiler`** `MemoryProfiler | None` --- **`_memory_profiling_config`** `MemoryProfilingConfig` --- **`_output_tails`** `Dict[str, _OutputTail]` --- **`_processes`** `Dict[str, Popen]` --- **`_server_client`** `ServerClient` --- **`_server_instance_display_configs`** `List[ServerInstanceDisplayConfig]` --- **`_server_shutdown_timeout_seconds`** `float = _GRACEFUL_SHUTDOWN_TIMEOUT_SEC` --- **`_startup`** `Optional[SupervisorStartup] = None` --- **`_telemetry_metrics_enabled`** `bool` --- ```python nemo_gym.orchestration.server_supervisor.RunHelper._move_to_a_fresh_port( error: BaseException ) -> bool ``` After a server found its port in use, give it a fresh one in the config every server is started from. Each server learns every other server's port from that config when it starts, so the caller starts them all again. Only a bind that failed with EADDRINUSE qualifies. The head server moves the same way: every server and the collection client read its port from that config. ```python nemo_gym.orchestration.server_supervisor.RunHelper._spawn_server( server_name: str, command: str, dir_path: pathlib.Path, project_root: pathlib.Path | None, extra_env: typing.Dict[str, str], global_config_dict: omegaconf.DictConfig | None = None, start_new_session: bool = False ) -> subprocess.Popen ``` Start one server, forwarding the stream its errors go to and keeping that stream's tail for poll(). ```python nemo_gym.orchestration.server_supervisor.RunHelper._start( global_config_dict_parser_config: nemo_gym.config.global_config.GlobalConfigDictParserConfig ) -> None ``` ```python nemo_gym.orchestration.server_supervisor.RunHelper._start_memory_profiler() -> None ``` ```python nemo_gym.orchestration.server_supervisor.RunHelper._wait_for_head_server() -> None ``` Wait until the head server answers, for at most `head_server_spinup_timeout_seconds`. A head server thread that ended (a failed bind) never will: that is a typed start failure, with EADDRINUSE when its port is taken now, so `start` moves it to a fresh port once. A live head server that never answers is a `HeadServerUnresponsiveError`, which stops the run instead of holding its allocation. ```python nemo_gym.orchestration.server_supervisor.RunHelper.check_http_server_statuses( successful_servers: typing.List[str], deadline: typing.Optional[float] = None ) -> typing.List[typing.Tuple[str, nemo_gym.client.server_client.ServerStatus]] ``` Probe every server not yet ready. With a `deadline` from `monotonic()`, each probe is capped at the time left before it. Servers still unprobed once it passes are reported as `timeout` without a request. ```python nemo_gym.orchestration.server_supervisor.RunHelper.display_server_instance_info() -> None ``` ```python nemo_gym.orchestration.server_supervisor.RunHelper.poll() -> None ``` ```python nemo_gym.orchestration.server_supervisor.RunHelper.run_forever() -> None ``` ```python nemo_gym.orchestration.server_supervisor.RunHelper.shutdown() -> None ``` ```python nemo_gym.orchestration.server_supervisor.RunHelper.start( global_config_dict_parser_config: nemo_gym.config.global_config.GlobalConfigDictParserConfig ) -> None ``` Start the head server and every configured server, and wait until all of them are ready. Any failure or interrupt before readiness shuts down everything started so far, then re-raises. Callers reach their own `shutdown()` only after this returns. Managed servers run in their own sessions, so shutdown owns delivery of their interrupt. ```python nemo_gym.orchestration.server_supervisor.RunHelper.wait_for_dry_run_spinup() -> None ``` Wait for every dry-run process to finish, and fail if any of them did. A dry run builds each server's venv and exits, so unlike poll() a finished process is the expected outcome here and only the exit code separates success from failure. The code has to be checked: uv creates the venv before installing into it, so a failed install still leaves an interpreter and an activate script behind. That venv then satisfies skip_venv_if_present on the next run, and the first symptom is an ImportError from a server long after the install that caused it. ```python nemo_gym.orchestration.server_supervisor.RunHelper.wait_for_model_endpoints( global_config_dict: omegaconf.DictConfig ) -> None ``` Block until every model endpoint named in the config answers, then return. Reads the bound from `model_endpoint_readiness_timeout_seconds`, where 0 skips the check entirely. Raises `ConfigError` naming the endpoints that never answered and the config key each came from. It raises rather than exits because `RunHelper` is imported and driven as a library, so the caller decides what an unreachable endpoint means; the CLI entrypoints turn it into an exit. `start()` shuts down the servers it already spawned before the error reaches the caller. ```python nemo_gym.orchestration.server_supervisor.RunHelper.wait_for_server_readiness( global_config_dict: omegaconf.DictConfig ) -> None ``` Mark the head ready only after every managed server and model endpoint is reachable. ```python nemo_gym.orchestration.server_supervisor.RunHelper.wait_for_spinup() -> None ``` ```python class nemo_gym.orchestration.server_supervisor.ServerSpinupTimeoutError() ``` **Bases:** `RuntimeError` Gym servers did not become ready within `server_spinup_timeout_seconds`. ```python class nemo_gym.orchestration.server_supervisor.ServerStartFailedError( message: str, server_name: str, errno: int | None ) ``` **Bases:** `RuntimeError` A Gym server process (or the head server) exited before it was ready. `failure_kind` is `server_start_failed`. `errno` is the errno of the bind error the server printed, when it printed one: `errno.EADDRINUSE` means another process took its port between the port check and the bind. ```python class nemo_gym.orchestration.server_supervisor._ConfiguredServer() ``` **Bases:** `NamedTuple` **`config`** `DictConfig` --- **`dir_path`** `Path` --- **`entrypoint_fpath`** `Path` --- **`name`** `str` --- **`server_type`** `str` --- **`top_level_path`** `str` --- ```python class nemo_gym.orchestration.server_supervisor._OutputTail( source: typing.IO[bytes], sink: typing.IO, max_bytes: int = _SERVER_OUTPUT_TAIL_BYTES ) ``` Forward a server's output pipe to `sink` as it arrives, and keep its last `max_bytes`. The operator still sees every line live, and the supervisor can read the error a server printed before it exited, such as the bind error that names an occupied port. **`_lock`** `= Lock()` --- **`_tail`** `= bytearray()` --- **`_thread`** --- ```python nemo_gym.orchestration.server_supervisor._OutputTail._forward( chunk: bytes ) -> None ``` ```python nemo_gym.orchestration.server_supervisor._OutputTail._pump() -> None ``` ```python nemo_gym.orchestration.server_supervisor._OutputTail.text( timeout_s: float = 5.0 ) -> str ``` The kept output, after waiting up to `timeout_s` for an exited server's pipe to drain. ```python nemo_gym.orchestration.server_supervisor._bind_errno( stderr: str | None ) -> int | None ``` The errno of the bind error a server printed on stderr, or None when it printed none. ```python nemo_gym.orchestration.server_supervisor._collect_model_endpoints( global_config_dict: omegaconf.DictConfig ) -> typing.List[typing.Tuple[str, str]] ``` The upstream model endpoints named in the resolved config, as (config key, url) pairs. `wait_for_spinup` only polls Gym's own servers, and the Gym-side model server answers as soon as it starts whether or not anything is behind it, so these are the URLs nothing checks. `base_url` is typed `Union[str, List[str]]` on vllm_model and the local vLLM servers, where the list form spreads load across replicas. The local ones default to an empty list and are filled in after Gym launches vLLM, so an empty list means "not yet" rather than "misconfigured" and is skipped, as is an unset value. The key travels with the url because the endpoint that fails may be a judge or user model, and a message naming `policy_base_url` would then be wrong. ```python nemo_gym.orchestration.server_supervisor._configured_servers( global_config_dict: omegaconf.DictConfig ) -> typing.Iterator[nemo_gym.orchestration.server_supervisor._ConfiguredServer] ``` Yield each server instance in the config that has an entrypoint to launch. ```python nemo_gym.orchestration.server_supervisor._endpoint_probe_url( base_url: str ) -> str ``` What to GET to find out whether `base_url` is being served. `GET /v1/models` is part of the OpenAI API, so most endpoints behind a `/v1` base URL answer it, but nothing here depends on that: any HTTP response other than a gateway status counts as answering, so an endpoint without it replies 404 and still passes. URLs that do not end in `/v1` are probed at their root. ```python nemo_gym.orchestration.server_supervisor._head_server_spinup_timeout_seconds( global_config_dict: omegaconf.DictConfig ) -> float ``` ```python nemo_gym.orchestration.server_supervisor._is_dns_failure( error: BaseException ) -> bool ``` Whether a `requests` connection error was a name that does not resolve. `requests` reports this as a `ConnectionError` like any other, with the resolver failure nested inside, so the cause chain has to be walked rather than the class inspected. ```python nemo_gym.orchestration.server_supervisor._model_endpoint_timeout_seconds( global_config_dict: omegaconf.DictConfig ) -> float ``` How long to wait for model endpoints, from config. 0 or a negative value skips the check. An unset key falls back to the same default the config parser applies. A value that is not a number is a configuration mistake, so it is reported as one rather than surfacing as a `ValueError` traceback from `float()`. ```python nemo_gym.orchestration.server_supervisor._port_bind_errno( host: str, port: int ) -> int | None ``` The errno of binding `host:port` now, as uvicorn binds it, or None when the port is free. ```python nemo_gym.orchestration.server_supervisor._probe_endpoint( base_url: str, timeout_seconds: float = _ENDPOINT_PROBE_TIMEOUT_SEC ) -> str ``` Whether anything answers at `base_url`, and if not, whether waiting could help. Answering is the bar, not healthy: a 401 or 404 means something is there, and requiring a 200 would reject endpoints that need auth. The exception is 502, 503 and 504, which a proxy or load balancer returns while the server behind it is still starting; those are waited on like a refused connection. A completed TLS handshake counts too, even against a certificate this process does not trust, which is why `SSLError` is checked before `ConnectionError` it inherits from. The probe connects directly, ignoring `HTTP_PROXY` and the like, because model requests go through Gym's aiohttp client, which ignores them too. Through a proxy, an endpoint the proxy cannot reach would come back as 502 or 504 and be waited on until the timeout. ```python nemo_gym.orchestration.server_supervisor._ray_server_names( configured_servers: typing.List[nemo_gym.orchestration.server_supervisor._ConfiguredServer] ) -> typing.List[str] ``` Names of the configured servers that declare `ray_enabled = True`, so this run must start or join Ray. A declaration that can't be read from the entrypoint's source, such as one inherited from another module, doesn't count. Such a server fails at startup with an error asking it to declare `ray_enabled = True` itself. ```python nemo_gym.orchestration.server_supervisor._resolve_server_dir( rel_path: pathlib.Path ) -> pathlib.Path ``` Resolve a relative server dir (e.g. `resources_servers/`) to an absolute path. Searches NEMO_GYM_EXTRA_ROOTS, the current working directory (a user's local server), then the Gym install root (`PARENT_DIR`) where built-in servers live in both editable and wheel installs. A directory counts as a server only if it ships an install marker for one of our two venv setups. This lets `gym env test` find and run built-in (and plugin) servers from any cwd, not just a repo checkout. ```python nemo_gym.orchestration.server_supervisor._server_group_exists( process: subprocess.Popen ) -> bool ``` ```python nemo_gym.orchestration.server_supervisor._server_launch_command( dir_path: pathlib.Path, global_config_dict: omegaconf.DictConfig, server_name: str, entrypoint_fpath: pathlib.Path ) -> str ``` Shell command that sets up a server's venv and starts its entrypoint with that venv's own interpreter. The interpreter is named explicitly instead of relying on the bare `python` that `source bin/activate` puts on PATH: a venv copied or moved after creation still names its original prefix in `bin/activate`, so activating it would silently run a different interpreter than `uv_venv_dir` selected. The middle line records when venv setup finished. The server uses it to separate venv setup from interpreter start and import time in its startup stages (see `nemo_gym.telemetry.startup`). It reads the clock with the venv's own interpreter because `date +%s%N` does not work on macOS. ```python nemo_gym.orchestration.server_supervisor._server_shutdown_timeout_seconds( global_config_dict: omegaconf.DictConfig ) -> float ``` The finite shared grace period for managed server process groups. ```python nemo_gym.orchestration.server_supervisor._server_spinup_timeout_seconds( global_config_dict: omegaconf.DictConfig ) -> float ``` How long to wait for Gym servers to become ready, from config. 0 or a negative value waits forever. An unset or null key falls back to the same default the config parser applies. Environment interpolation yields a string, so numeric strings are accepted. Anything else is reported as a `ConfigError` rather than a `TypeError` traceback mid-startup. ```python nemo_gym.orchestration.server_supervisor._signal_server_group( process: subprocess.Popen, signal_number: int ) -> None ``` ```python nemo_gym.orchestration.server_supervisor._startup_outcome( error: BaseException ) -> str ``` How a startup that raised `error` ended, as the startup telemetry reports it. ```python nemo_gym.orchestration.server_supervisor._wait_for_model_endpoints( endpoints: typing.List[typing.Tuple[str, str]], timeout_seconds: float, poll_interval_seconds: float = _ENDPOINT_POLL_INTERVAL_SEC, monotonic = monotonic, sleep_fn = sleep ) -> typing.List[typing.Tuple[str, str]] ``` Wait for every endpoint to answer. Returns the ones that never did. Unresolvable names are reported once and not waited on. Refused connections and gateway statuses are waited on, because an inference server can take minutes to load weights and someone who starts thirty seconds early should not have to start over. ```python nemo_gym.orchestration.server_supervisor.HEAD_SERVER_SPINUP_TIMEOUT_SECONDS_KEY_NAME = 'head_server_spinup_timeout_seconds' ``` ```python nemo_gym.orchestration.server_supervisor.RAY_ENABLE_AUTO_CONNECT_ENV_VAR_NAME = 'RAY_ENABLE_AUTO_CONNECT' ``` ```python nemo_gym.orchestration.server_supervisor._BASE_URL_KEY_SUFFIX = 'base_url' ``` ```python nemo_gym.orchestration.server_supervisor._BIND_ERRNO_PATTERN = re.compile('\\[Errno (\\d+)\\] error while attempting to bind') ``` ```python nemo_gym.orchestration.server_supervisor._DEFAULT_HEAD_SERVER_SPINUP_TIMEOUT_SEC = 900.0 ``` ```python nemo_gym.orchestration.server_supervisor._DEFAULT_MODEL_ENDPOINT_READINESS_TIMEOUT_SEC: float = 600.0 ``` ```python nemo_gym.orchestration.server_supervisor._DEFAULT_SERVER_SPINUP_TIMEOUT_SEC: float = float(DEFAULT_SERVER_SPINUP_TIMEOUT_SECONDS) ``` ```python nemo_gym.orchestration.server_supervisor._ENDPOINT_ANSWERING = 'answering' ``` ```python nemo_gym.orchestration.server_supervisor._ENDPOINT_POLL_INTERVAL_SEC: float = 3.0 ``` ```python nemo_gym.orchestration.server_supervisor._ENDPOINT_PROBE_TIMEOUT_SEC: float = 5.0 ``` ```python nemo_gym.orchestration.server_supervisor._ENDPOINT_REFUSED = 'refused' ``` ```python nemo_gym.orchestration.server_supervisor._ENDPOINT_STARTING = 'starting' ``` ```python nemo_gym.orchestration.server_supervisor._ENDPOINT_STARTING_STATUS_CODES = frozenset({502, 503, 504}) ``` ```python nemo_gym.orchestration.server_supervisor._ENDPOINT_UNRESOLVABLE = 'unresolvable' ``` ```python nemo_gym.orchestration.server_supervisor._FORCE_KILL_REAP_TIMEOUT_SEC: int = 2 ``` ```python nemo_gym.orchestration.server_supervisor._GRACEFUL_SHUTDOWN_TIMEOUT_SEC: int = 1 ``` ```python nemo_gym.orchestration.server_supervisor._MODEL_SERVER_TYPE = 'responses_api_models' ``` ```python nemo_gym.orchestration.server_supervisor._SERVER_OUTPUT_TAIL_BYTES = 64 * 1024 ``` ```python nemo_gym.orchestration.server_supervisor._SERVER_PROBE_TIMEOUT_SEC: float = 5.0 ```