nemo_gym.server_runtime.base_server

Module Contents

Classes

Name	Description
`BaseServer`	All instances of BaseServer are queryable using ServerClient.
`KeepaliveHttpToolsProtocol`	Uvicorn’s httptools protocol with TCP keepalive on every accepted connection.
`RewritableRequest`	The part of an `httpx.Request` the rollout-prefix hooks use: a URL whose type parses a string.
`SimpleServer`	-
`UvicornBacklogConfig`	-
`UvicornLoggingConfig`	-
`UvicornProxyHeadersConfig`	-
`UvicornReusePortConfig`	-

Functions

Name	Description
`_end_of_segments`	The index in `path` just past its first `count` non-empty segments; repeated slashes are kept as sent.
`_telemetry_server_type`	Return the Gym server-type name for server_cls, or None if it is not one.
`_url_root`	Scheme, lowercased host, effective port and path segments, as HTTP clients normalize them.
`async_rollout_prefix_hook`	An `httpx.AsyncClient` request event hook that applies `with_rollout_prefix` to every request.
`refuse_unsupported_num_workers`	Raise when the server asks for several workers but cannot run with them.
`rollout_prefix_hook`	An `httpx.Client` request event hook that applies `with_rollout_prefix` to every request.
`with_rollout_prefix`	Insert `/ng-rollout/<current rollout>` after the matching base URL, for clients with a fixed base URL.

Data

_ALL_V4_MAPPED

_DEFAULT_PORTS

_ORIGINAL_URL_EXTENSION

_TELEMETRY_SERVER_TYPE_BY_BASE

__getattr__

logger

API

class nemo_gym.server_runtime.base_server.BaseServer()

Bases: BaseModel

All instances of BaseServer are queryable using ServerClient.

config

BaseRunServerInstanceConfig

nemo_gym.server_runtime.base_server.BaseServer.load_config_from_global_config() -> nemo_gym.config.config_types.BaseRunServerInstanceConfig

classmethod

nemo_gym.server_runtime.base_server.BaseServer.setup_liveness(
    app: fastapi.FastAPI
) -> None

class nemo_gym.server_runtime.base_server.KeepaliveHttpToolsProtocol(
    args: typing.Any = (),
    keepalive: typing.Tuple[int, int, int],
    kwargs: typing.Any = {}
)

Protocol

Bases: HttpToolsProtocol

Uvicorn’s httptools protocol with TCP keepalive on every accepted connection.

A model server answers only after generation finishes, so a client connection can carry no bytes for many minutes. Stateful network hops on some paths evict flows that stay idle that long, which silently drops the eventual reply and leaves the client waiting on a half-open connection. Keepalive probes keep the flow alive and let the kernel reap peers that are really gone. Uses the same global_aiohttp_tcp_keepalive_* settings as the outgoing client connections.

Handed to uvicorn as partial(KeepaliveHttpToolsProtocol, keepalive=(idle, interval, probes)): multi-worker uvicorn pickles its config into worker processes, so this class must stay importable at module level.

nemo_gym.server_runtime.base_server.KeepaliveHttpToolsProtocol.connection_made(
    transport: asyncio.Transport
) -> None

class nemo_gym.server_runtime.base_server.RewritableRequest()

Protocol

The part of an httpx.Request the rollout-prefix hooks use: a URL whose type parses a string.

url

object

class nemo_gym.server_runtime.base_server.SimpleServer()

Bases: BaseServer

ray_enabled

bool = False

routes_sessions_to_owner

bool = False

server_client

ServerClient

nemo_gym.server_runtime.base_server.SimpleServer._failure_reply(
    e: BaseException
) -> fastapi.responses.JSONResponse

The reply for an exception the route did not handle.

nemo_gym.server_runtime.base_server.SimpleServer.get_session_middleware_key() -> str

nemo_gym.server_runtime.base_server.SimpleServer.instrument_app_for_telemetry(
    app: fastapi.FastAPI
) -> None

Apply OTel FastAPI auto-instrumentation to app, if telemetry is exporting traces.

Gives every server the inbound half of context propagation plus dimensioned http.server.* metrics, which is why Gym does not use nemo-lens’s undimensioned gym.server.request_duration_ms (see nemo_gym/telemetry/metrics.py).

The instrumentor costs tens of microseconds per request even when no span is exported, so a metrics-only run skips it and does without the http.server.* metrics.

A server whose instrumentation could not be applied keeps serving; it just does not extract inbound trace context.

nemo_gym.server_runtime.base_server.SimpleServer.multiple_workers_unsupported_reason() -> typing.Optional[str]

Why this server cannot run with num_workers > 1, or None when it can.

With several workers, uvicorn imports the entrypoint in each worker and serves its module-level app. A server whose state or resources several workers cannot share overrides this and returns the reason, and run_webserver refuses num_workers > 1 with it before uvicorn starts. The reason reads as the end of the sentence “<server> needs num_workers=1: <reason>”. It is a method rather than a class attribute so that a server can decide from its config.

nemo_gym.server_runtime.base_server.SimpleServer.owned_session_routing() -> dict[str, typing.Any]

Arguments to install_session_routing: where requests name their session, and how long ownership lasts.

A server that sets routes_sessions_to_owner implements it.

nemo_gym.server_runtime.base_server.SimpleServer.prefix_server_logs() -> None

nemo_gym.server_runtime.base_server.SimpleServer.run_webserver() -> typing.Optional[fastapi.FastAPI]

classmethod

nemo_gym.server_runtime.base_server.SimpleServer.set_ulimit(
    target_soft_limit: int = 65535
)

nemo_gym.server_runtime.base_server.SimpleServer.setup_cancellation_middleware(
    app: fastapi.FastAPI
) -> None

nemo_gym.server_runtime.base_server.SimpleServer.setup_checkpoint_participant(
    app: fastapi.FastAPI
) -> None

Take part in partial-rollout checkpoints on the app setup_webserver built, if it does not yet.

Called for every server, so a server whose setup_webserver builds its own app takes part too.

nemo_gym.server_runtime.base_server.SimpleServer.setup_exception_middleware(
    app: fastapi.FastAPI
) -> None

nemo_gym.server_runtime.base_server.SimpleServer.setup_profiling(
    app: fastapi.FastAPI,
    profiling_config: nemo_gym.telemetry.profiling.ProfilingMiddlewareConfig
) -> None

nemo_gym.server_runtime.base_server.SimpleServer.setup_session_middleware(
    app: fastapi.FastAPI
) -> None

nemo_gym.server_runtime.base_server.SimpleServer.setup_telemetry() -> None

Initialise this process’s nemo-lens telemetry. Idempotent, once per process.

Every Gym server is its own process with its own providers — there is no parent handle to inherit, only the NEMO_GYM_OTEL_* environment the supervisor exported before spawning it. A failure here is logged and swallowed: telemetry must never stop a server from serving.

nemo_gym.server_runtime.base_server.SimpleServer.setup_webserver() -> fastapi.FastAPI

abstract

class nemo_gym.server_runtime.base_server.UvicornBacklogConfig()

Bases: BaseModel

uvicorn_backlog

PositiveInt = DEFAULT_UVICORN_BACKLOG

class nemo_gym.server_runtime.base_server.UvicornLoggingConfig()

Bases: BaseModel

uvicorn_logging_show_200_ok

bool = False

class nemo_gym.server_runtime.base_server.UvicornProxyHeadersConfig()

Bases: BaseModel

uvicorn_forwarded_allow_ips

Optional[List[str]] = None

uvicorn_proxy_headers

bool = False

nemo_gym.server_runtime.base_server.UvicornProxyHeadersConfig._require_trusted_proxy_allowlist() -> nemo_gym.server_runtime.base_server.UvicornProxyHeadersConfig

class nemo_gym.server_runtime.base_server.UvicornReusePortConfig()

Bases: BaseModel

uvicorn_reuse_port

bool = False

nemo_gym.server_runtime.base_server._end_of_segments(
    path: str,
    count: int
) -> int

The index in path just past its first count non-empty segments; repeated slashes are kept as sent.

nemo_gym.server_runtime.base_server._telemetry_server_type(
    server_cls: typing.Type
) -> typing.Optional[str]

Return the Gym server-type name for server_cls, or None if it is not one.

nemo_gym.server_runtime.base_server._url_root(
    url: str
) -> tuple[str, typing.Optional[str], typing.Optional[int], list[str]]

Scheme, lowercased host, effective port and path segments, as HTTP clients normalize them.

nemo_gym.server_runtime.base_server.async_rollout_prefix_hook(
    base_urls: typing.Sequence[str]
) -> typing.Callable[[RewritableRequest], typing.Awaitable[None]]

An httpx.AsyncClient request event hook that applies with_rollout_prefix to every request.

nemo_gym.server_runtime.base_server.refuse_unsupported_num_workers(
    server: nemo_gym.server_runtime.base_server.SimpleServer
) -> None

Raise when the server asks for several workers but cannot run with them.

nemo_gym.server_runtime.base_server.rollout_prefix_hook(
    base_urls: typing.Sequence[str]
) -> typing.Callable[[RewritableRequest], None]

An httpx.Client request event hook that applies with_rollout_prefix to every request.

nemo_gym.server_runtime.base_server.with_rollout_prefix(
    url: str,
    base_urls: typing.Sequence[str]
) -> str

Insert /ng-rollout/<current rollout> after the matching base URL, for clients with a fixed base URL.

A long-lived client (for example one a resources server calls from verify()) keeps one base URL; this rewrites each request URL instead, so the call reaches the Gym model server exactly like a prefixed one. Only URLs under one of base_urls are changed, compared by scheme, host (case-insensitive), effective port and whole path segments. A URL whose path below the base already carries a rollout prefix is left as it is. Without a current rollout the URL is unchanged, and the model server forwards the call without capture.

The current rollout is a ContextVar: a synchronous client run through asyncio.to_thread or contextvars.copy_context().run sees it, one run through loop.run_in_executor or a thread pool does not.

nemo_gym.server_runtime.base_server._ALL_V4_MAPPED = ip_network('::ffff:0:0/96')

nemo_gym.server_runtime.base_server._DEFAULT_PORTS = {'http': 80, 'https': 443}

nemo_gym.server_runtime.base_server._ORIGINAL_URL_EXTENSION = 'nemo_gym.original_url'

nemo_gym.server_runtime.base_server._TELEMETRY_SERVER_TYPE_BY_BASE = {'SimpleResourcesServer': 'resources_servers', 'SimpleResponsesAPIAgent': 'respo...

nemo_gym.server_runtime.base_server.__getattr__ = moved_names_getattr(__name__)

nemo_gym.server_runtime.base_server.logger = logging.getLogger(__name__)