> This page is for version 0.5.0. > For other versions, use one of these documentation indexes: > - Main (default): https://docs.nvidia.com/nemo/gym/main/llms.txt > - 0.7.0: https://docs.nvidia.com/nemo/gym/v0.7.0/llms.txt > - 0.6.0: https://docs.nvidia.com/nemo/gym/v0.6.0/llms.txt > - 0.5.1: https://docs.nvidia.com/nemo/gym/v0.5.1/llms.txt > - 0.5.0: https://docs.nvidia.com/nemo/gym/v0.5.0/llms.txt > - 0.4.0: https://docs.nvidia.com/nemo/gym/v0.4.0/llms.txt > - 0.3.0: https://docs.nvidia.com/nemo/gym/v0.3.0/llms.txt > - 0.2.1: https://docs.nvidia.com/nemo/gym/v0.2.1/llms.txt > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/nemo/gym/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/gym/_mcp/server. # nemo_gym.server_runtime.base_server ## Module Contents ### Classes | Name | Description | | ----------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------- | | [`BaseServer`](#nemo_gym-server_runtime-base_server-BaseServer) | All instances of BaseServer are queryable using ServerClient. | | [`KeepaliveHttpToolsProtocol`](#nemo_gym-server_runtime-base_server-KeepaliveHttpToolsProtocol) | Uvicorn's httptools protocol with TCP keepalive on every accepted connection. | | [`RewritableRequest`](#nemo_gym-server_runtime-base_server-RewritableRequest) | The part of an `httpx.Request` the rollout-prefix hooks use: a URL whose type parses a string. | | [`SimpleServer`](#nemo_gym-server_runtime-base_server-SimpleServer) | - | | [`UvicornBacklogConfig`](#nemo_gym-server_runtime-base_server-UvicornBacklogConfig) | - | | [`UvicornLoggingConfig`](#nemo_gym-server_runtime-base_server-UvicornLoggingConfig) | - | | [`UvicornProxyHeadersConfig`](#nemo_gym-server_runtime-base_server-UvicornProxyHeadersConfig) | - | | [`UvicornReusePortConfig`](#nemo_gym-server_runtime-base_server-UvicornReusePortConfig) | - | ### Functions | Name | Description | | ------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------ | | [`_end_of_segments`](#nemo_gym-server_runtime-base_server-_end_of_segments) | The index in `path` just past its first `count` non-empty segments; repeated slashes are kept as sent. | | [`_telemetry_server_type`](#nemo_gym-server_runtime-base_server-_telemetry_server_type) | Return the Gym server-type name for *server_cls*, or None if it is not one. | | [`_url_root`](#nemo_gym-server_runtime-base_server-_url_root) | Scheme, lowercased host, effective port and path segments, as HTTP clients normalize them. | | [`async_rollout_prefix_hook`](#nemo_gym-server_runtime-base_server-async_rollout_prefix_hook) | An `httpx.AsyncClient` request event hook that applies `with_rollout_prefix` to every request. | | [`refuse_unsupported_num_workers`](#nemo_gym-server_runtime-base_server-refuse_unsupported_num_workers) | Raise when the server asks for several workers but cannot run with them. | | [`rollout_prefix_hook`](#nemo_gym-server_runtime-base_server-rollout_prefix_hook) | An `httpx.Client` request event hook that applies `with_rollout_prefix` to every request. | | [`with_rollout_prefix`](#nemo_gym-server_runtime-base_server-with_rollout_prefix) | Insert `/ng-rollout/<current rollout>` after the matching base URL, for clients with a fixed base URL. | ### Data [`_ALL_V4_MAPPED`](#nemo_gym-server_runtime-base_server-_ALL_V4_MAPPED) [`_DEFAULT_PORTS`](#nemo_gym-server_runtime-base_server-_DEFAULT_PORTS) [`_ORIGINAL_URL_EXTENSION`](#nemo_gym-server_runtime-base_server-_ORIGINAL_URL_EXTENSION) [`_TELEMETRY_SERVER_TYPE_BY_BASE`](#nemo_gym-server_runtime-base_server-_TELEMETRY_SERVER_TYPE_BY_BASE) [`__getattr__`](#nemo_gym-server_runtime-base_server-__getattr__) [`logger`](#nemo_gym-server_runtime-base_server-logger) ### API ```python class nemo_gym.server_runtime.base_server.BaseServer() ``` **Bases:** `BaseModel` All instances of BaseServer are queryable using ServerClient. **`config`** `BaseRunServerInstanceConfig` --- ```python nemo_gym.server_runtime.base_server.BaseServer.load_config_from_global_config() -> nemo_gym.config.config_types.BaseRunServerInstanceConfig ``` classmethod ```python nemo_gym.server_runtime.base_server.BaseServer.setup_liveness( app: fastapi.FastAPI ) -> None ``` ```python class nemo_gym.server_runtime.base_server.KeepaliveHttpToolsProtocol( args: typing.Any = (), keepalive: typing.Tuple[int, int, int], kwargs: typing.Any = {} ) ``` Protocol **Bases:** `HttpToolsProtocol` Uvicorn's httptools protocol with TCP keepalive on every accepted connection. A model server answers only after generation finishes, so a client connection can carry no bytes for many minutes. Stateful network hops on some paths evict flows that stay idle that long, which silently drops the eventual reply and leaves the client waiting on a half-open connection. Keepalive probes keep the flow alive and let the kernel reap peers that are really gone. Uses the same `global_aiohttp_tcp_keepalive_*` settings as the outgoing client connections. Handed to uvicorn as `partial(KeepaliveHttpToolsProtocol, keepalive=(idle, interval, probes))`: multi-worker uvicorn pickles its config into worker processes, so this class must stay importable at module level. ```python nemo_gym.server_runtime.base_server.KeepaliveHttpToolsProtocol.connection_made( transport: asyncio.Transport ) -> None ``` ```python class nemo_gym.server_runtime.base_server.RewritableRequest() ``` Protocol The part of an `httpx.Request` the rollout-prefix hooks use: a URL whose type parses a string. **`url`** `object` --- ```python class nemo_gym.server_runtime.base_server.SimpleServer() ``` **Bases:** [BaseServer](#nemo_gym-server_runtime-base_server-BaseServer) **`ray_enabled`** `bool = False` --- **`routes_sessions_to_owner`** `bool = False` --- **`server_client`** `ServerClient` --- ```python nemo_gym.server_runtime.base_server.SimpleServer._failure_reply( e: BaseException ) -> fastapi.responses.JSONResponse ``` The reply for an exception the route did not handle. ```python nemo_gym.server_runtime.base_server.SimpleServer.get_session_middleware_key() -> str ``` ```python nemo_gym.server_runtime.base_server.SimpleServer.instrument_app_for_telemetry( app: fastapi.FastAPI ) -> None ``` Apply OTel FastAPI auto-instrumentation to *app*, if telemetry is exporting traces. Gives every server the inbound half of context propagation plus dimensioned `http.server.*` metrics, which is why Gym does not use nemo-lens's undimensioned `gym.server.request_duration_ms` (see `nemo_gym/telemetry/metrics.py`). The instrumentor costs tens of microseconds per request even when no span is exported, so a metrics-only run skips it and does without the `http.server.*` metrics. A server whose instrumentation could not be applied keeps serving; it just does not extract inbound trace context. ```python nemo_gym.server_runtime.base_server.SimpleServer.multiple_workers_unsupported_reason() -> typing.Optional[str] ``` Why this server cannot run with `num_workers > 1`, or None when it can. With several workers, uvicorn imports the entrypoint in each worker and serves its module-level `app`. A server whose state or resources several workers cannot share overrides this and returns the reason, and `run_webserver` refuses `num_workers > 1` with it before uvicorn starts. The reason reads as the end of the sentence "\ needs num_workers=1: \". It is a method rather than a class attribute so that a server can decide from its config. ```python nemo_gym.server_runtime.base_server.SimpleServer.owned_session_routing() -> dict[str, typing.Any] ``` Arguments to `install_session_routing`: where requests name their session, and how long ownership lasts. A server that sets `routes_sessions_to_owner` implements it. ```python nemo_gym.server_runtime.base_server.SimpleServer.prefix_server_logs() -> None ``` ```python nemo_gym.server_runtime.base_server.SimpleServer.run_webserver() -> typing.Optional[fastapi.FastAPI] ``` classmethod ```python nemo_gym.server_runtime.base_server.SimpleServer.set_ulimit( target_soft_limit: int = 65535 ) ``` ```python nemo_gym.server_runtime.base_server.SimpleServer.setup_cancellation_middleware( app: fastapi.FastAPI ) -> None ``` ```python nemo_gym.server_runtime.base_server.SimpleServer.setup_checkpoint_participant( app: fastapi.FastAPI ) -> None ``` Take part in partial-rollout checkpoints on the app `setup_webserver` built, if it does not yet. Called for every server, so a server whose `setup_webserver` builds its own app takes part too. ```python nemo_gym.server_runtime.base_server.SimpleServer.setup_exception_middleware( app: fastapi.FastAPI ) -> None ``` ```python nemo_gym.server_runtime.base_server.SimpleServer.setup_profiling( app: fastapi.FastAPI, profiling_config: nemo_gym.telemetry.profiling.ProfilingMiddlewareConfig ) -> None ``` ```python nemo_gym.server_runtime.base_server.SimpleServer.setup_session_middleware( app: fastapi.FastAPI ) -> None ``` ```python nemo_gym.server_runtime.base_server.SimpleServer.setup_telemetry() -> None ``` Initialise this process's nemo-lens telemetry. Idempotent, once per process. Every Gym server is its own process with its own providers — there is no parent handle to inherit, only the `NEMO_GYM_OTEL_*` environment the supervisor exported before spawning it. A failure here is logged and swallowed: telemetry must never stop a server from serving. ```python nemo_gym.server_runtime.base_server.SimpleServer.setup_webserver() -> fastapi.FastAPI ``` abstract ```python class nemo_gym.server_runtime.base_server.UvicornBacklogConfig() ``` **Bases:** `BaseModel` **`uvicorn_backlog`** `PositiveInt = DEFAULT_UVICORN_BACKLOG` --- ```python class nemo_gym.server_runtime.base_server.UvicornLoggingConfig() ``` **Bases:** `BaseModel` **`uvicorn_logging_show_200_ok`** `bool = False` --- ```python class nemo_gym.server_runtime.base_server.UvicornProxyHeadersConfig() ``` **Bases:** `BaseModel` **`uvicorn_forwarded_allow_ips`** `Optional[List[str]] = None` --- **`uvicorn_proxy_headers`** `bool = False` --- ```python nemo_gym.server_runtime.base_server.UvicornProxyHeadersConfig._require_trusted_proxy_allowlist() -> nemo_gym.server_runtime.base_server.UvicornProxyHeadersConfig ``` ```python class nemo_gym.server_runtime.base_server.UvicornReusePortConfig() ``` **Bases:** `BaseModel` **`uvicorn_reuse_port`** `bool = False` --- ```python nemo_gym.server_runtime.base_server._end_of_segments( path: str, count: int ) -> int ``` The index in `path` just past its first `count` non-empty segments; repeated slashes are kept as sent. ```python nemo_gym.server_runtime.base_server._telemetry_server_type( server_cls: typing.Type ) -> typing.Optional[str] ``` Return the Gym server-type name for *server_cls*, or None if it is not one. ```python nemo_gym.server_runtime.base_server._url_root( url: str ) -> tuple[str, typing.Optional[str], typing.Optional[int], list[str]] ``` Scheme, lowercased host, effective port and path segments, as HTTP clients normalize them. ```python nemo_gym.server_runtime.base_server.async_rollout_prefix_hook( base_urls: typing.Sequence[str] ) -> typing.Callable[[RewritableRequest], typing.Awaitable[None]] ``` An `httpx.AsyncClient` request event hook that applies `with_rollout_prefix` to every request. ```python nemo_gym.server_runtime.base_server.refuse_unsupported_num_workers( server: nemo_gym.server_runtime.base_server.SimpleServer ) -> None ``` Raise when the server asks for several workers but cannot run with them. ```python nemo_gym.server_runtime.base_server.rollout_prefix_hook( base_urls: typing.Sequence[str] ) -> typing.Callable[[RewritableRequest], None] ``` An `httpx.Client` request event hook that applies `with_rollout_prefix` to every request. ```python nemo_gym.server_runtime.base_server.with_rollout_prefix( url: str, base_urls: typing.Sequence[str] ) -> str ``` Insert `/ng-rollout/` after the matching base URL, for clients with a fixed base URL. A long-lived client (for example one a resources server calls from `verify()`) keeps one base URL; this rewrites each request URL instead, so the call reaches the Gym model server exactly like a prefixed one. Only URLs under one of `base_urls` are changed, compared by scheme, host (case-insensitive), effective port and whole path segments. A URL whose path below the base already carries a rollout prefix is left as it is. Without a current rollout the URL is unchanged, and the model server forwards the call without capture. The current rollout is a `ContextVar`: a synchronous client run through `asyncio.to_thread` or `contextvars.copy_context().run` sees it, one run through `loop.run_in_executor` or a thread pool does not. ```python nemo_gym.server_runtime.base_server._ALL_V4_MAPPED = ip_network('::ffff:0:0/96') ``` ```python nemo_gym.server_runtime.base_server._DEFAULT_PORTS = {'http': 80, 'https': 443} ``` ```python nemo_gym.server_runtime.base_server._ORIGINAL_URL_EXTENSION = 'nemo_gym.original_url' ``` ```python nemo_gym.server_runtime.base_server._TELEMETRY_SERVER_TYPE_BY_BASE = {'SimpleResourcesServer': 'resources_servers', 'SimpleResponsesAPIAgent': 'respo... ``` ```python nemo_gym.server_runtime.base_server.__getattr__ = moved_names_getattr(__name__) ``` ```python nemo_gym.server_runtime.base_server.logger = logging.getLogger(__name__) ```