nemo_gym.server_runtime.request_handling.model_sampling

The middleware that handles a model call’s /ng-sampling/<values|null> segment on each kind of server.

nemo_gym.core.model_call_sampling describes the values and where they come from.

Module Contents

Classes

Name	Description
`CallerSamplingMiddleware`	Give every model call a resources server makes the `null` segment, so it keeps its caller’s sampling.
`ModelSamplingContextMiddleware`	Strip an inbound `/ng-sampling/` segment on an agent server and give the agent’s model calls the same one.
`ModelSamplingMiddleware`	Apply a model call’s sampling segment on a Model Server, before policy admission and capture see the call.
`_SamplingPath`	-

Functions

Name	Description
`_api_path`	The path a request routes to after its correlation prefix, such as /v1/responses.
`_parse_sampling_path`	Split off a request path’s sampling segment, or return `None` for a path without one.
`_reject`	-
`_route_path`	Split the request path into a relay’s mount path (`root_path`) and the path this app routes.

Data

_TOKEN_LIMIT_FIELDS

API

class nemo_gym.server_runtime.request_handling.model_sampling.CallerSamplingMiddleware(
    app: starlette.types.ASGIApp
)

Give every model call a resources server makes the null segment, so it keeps its caller’s sampling.

Installed in a run that sets model_call_sampling. A resources server never runs an agent that the framework trains, so its judges and tools keep their own sampling, or their Model Server’s sampling_overrides.

nemo_gym.server_runtime.request_handling.model_sampling.CallerSamplingMiddleware.__call__(
    scope: starlette.types.Scope,
    receive: starlette.types.Receive,
    send: starlette.types.Send
) -> None

async

class nemo_gym.server_runtime.request_handling.model_sampling.ModelSamplingContextMiddleware(
    app: starlette.types.ASGIApp
)

Strip an inbound /ng-sampling/ segment on an agent server and give the agent’s model calls the same one.

An environment server’s activation of the agent, or the agent’s call to itself, carries the rollout’s segment. The agent’s URL helpers then copy it onto every model URL the agent builds while handling the request.

nemo_gym.server_runtime.request_handling.model_sampling.ModelSamplingContextMiddleware.__call__(
    scope: starlette.types.Scope,
    receive: starlette.types.Receive,
    send: starlette.types.Send
) -> None

async

class nemo_gym.server_runtime.request_handling.model_sampling.ModelSamplingMiddleware(
    app: starlette.types.ASGIApp,
    required: bool = False
)

Apply a model call’s sampling segment on a Model Server, before policy admission and capture see the call.

Values replace the request’s temperature and top_p. Their max_output_tokens caps the request’s own output-token limit, so a smaller limit the caller set is kept. null leaves the request as its caller sent it. Either way the segment is stripped from the path and stays the current segment while the server handles the call. With required, a generation call without a segment fails.

Only the body of a POST to a generation route is rewritten; other routes keep their body. Responses, streamed or not, pass through unchanged.

nemo_gym.server_runtime.request_handling.model_sampling.ModelSamplingMiddleware.__call__(
    scope: starlette.types.Scope,
    receive: starlette.types.Receive,
    send: starlette.types.Send
) -> None

async

nemo_gym.server_runtime.request_handling.model_sampling.ModelSamplingMiddleware._apply(
    scope: starlette.types.Scope,
    receive: starlette.types.Receive,
    send: starlette.types.Send,
    sampling: nemo_gym.core.model_call_sampling.ModelCallSampling,
    limit_fields: tuple[str, ...]
) -> None

async

class nemo_gym.server_runtime.request_handling.model_sampling._SamplingPath()

Bases: NamedTuple

api_path

str

routed_path

str

sampling

Optional[ModelCallSampling]

segment

str

nemo_gym.server_runtime.request_handling.model_sampling._api_path(
    route_path: str
) -> str

The path a request routes to after its correlation prefix, such as /v1/responses.

nemo_gym.server_runtime.request_handling.model_sampling._parse_sampling_path(
    scope: starlette.types.Scope
) -> typing.Optional[nemo_gym.server_runtime.request_handling.model_sampling._SamplingPath]

Split off a request path’s sampling segment, or return None for a path without one.

Raises ValueError for a malformed segment.

nemo_gym.server_runtime.request_handling.model_sampling._reject(
    scope: starlette.types.Scope,
    receive: starlette.types.Receive,
    send: starlette.types.Send,
    detail: str
) -> None

async

nemo_gym.server_runtime.request_handling.model_sampling._route_path(
    scope: starlette.types.Scope
) -> tuple[str, str]

Split the request path into a relay’s mount path (root_path) and the path this app routes.

nemo_gym.server_runtime.request_handling.model_sampling._TOKEN_LIMIT_FIELDS = {'/v1/chat/completions': ('max_tokens', 'max_completion_tokens'), '/v1/responses...