> This page is for version 0.4.0.
> For other versions, use one of these documentation indexes:
> - Main (default): https://docs.nvidia.com/nemo/gym/main/llms.txt
> - 0.7.0: https://docs.nvidia.com/nemo/gym/v0.7.0/llms.txt
> - 0.6.0: https://docs.nvidia.com/nemo/gym/v0.6.0/llms.txt
> - 0.5.1: https://docs.nvidia.com/nemo/gym/v0.5.1/llms.txt
> - 0.5.0: https://docs.nvidia.com/nemo/gym/v0.5.0/llms.txt
> - 0.4.0: https://docs.nvidia.com/nemo/gym/v0.4.0/llms.txt
> - 0.3.0: https://docs.nvidia.com/nemo/gym/v0.3.0/llms.txt
> - 0.2.1: https://docs.nvidia.com/nemo/gym/v0.2.1/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/gym/llms.txt.

# nemo_gym.server_runtime.request_handling.model_sampling

The middleware that handles a model call's `/ng-sampling/<values|null>` segment on each kind of server.

`nemo_gym.core.model_call_sampling` describes the values and where they come from.

## Module Contents

### Classes

| Name                                                                                                                        | Description                                                                                                |
| --------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------- |
| [`CallerSamplingMiddleware`](#nemo_gym-server_runtime-request_handling-model_sampling-CallerSamplingMiddleware)             | Give every model call a resources server makes the `null` segment, so it keeps its caller's sampling.      |
| [`ModelSamplingContextMiddleware`](#nemo_gym-server_runtime-request_handling-model_sampling-ModelSamplingContextMiddleware) | Strip an inbound `/ng-sampling/` segment on an agent server and give the agent's model calls the same one. |
| [`ModelSamplingMiddleware`](#nemo_gym-server_runtime-request_handling-model_sampling-ModelSamplingMiddleware)               | Apply a model call's sampling segment on a Model Server, before policy admission and capture see the call. |
| [`_SamplingPath`](#nemo_gym-server_runtime-request_handling-model_sampling-_SamplingPath)                                   | -                                                                                                          |

### Functions

| Name                                                                                                    | Description                                                                                  |
| ------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------- |
| [`_api_path`](#nemo_gym-server_runtime-request_handling-model_sampling-_api_path)                       | The path a request routes to after its correlation prefix, such as /v1/responses.            |
| [`_parse_sampling_path`](#nemo_gym-server_runtime-request_handling-model_sampling-_parse_sampling_path) | Split off a request path's sampling segment, or return `None` for a path without one.        |
| [`_reject`](#nemo_gym-server_runtime-request_handling-model_sampling-_reject)                           | -                                                                                            |
| [`_route_path`](#nemo_gym-server_runtime-request_handling-model_sampling-_route_path)                   | Split the request path into a relay's mount path (`root_path`) and the path this app routes. |

### Data

[`_TOKEN_LIMIT_FIELDS`](#nemo_gym-server_runtime-request_handling-model_sampling-_TOKEN_LIMIT_FIELDS)

### API

```python
class nemo_gym.server_runtime.request_handling.model_sampling.CallerSamplingMiddleware(
    app: starlette.types.ASGIApp
)
```

Give every model call a resources server makes the `null` segment, so it keeps its caller's sampling.

Installed in a run that sets `model_call_sampling`. A resources server never runs an agent that the framework
trains, so its judges and tools keep their own sampling, or their Model Server's `sampling_overrides`.

```python
nemo_gym.server_runtime.request_handling.model_sampling.CallerSamplingMiddleware.__call__(
    scope: starlette.types.Scope,
    receive: starlette.types.Receive,
    send: starlette.types.Send
) -> None
```

async

```python
class nemo_gym.server_runtime.request_handling.model_sampling.ModelSamplingContextMiddleware(
    app: starlette.types.ASGIApp
)
```

Strip an inbound `/ng-sampling/` segment on an agent server and give the agent's model calls the same one.

An environment server's activation of the agent, or the agent's call to itself, carries the rollout's segment.
The agent's URL helpers then copy it onto every model URL the agent builds while handling the request.

```python
nemo_gym.server_runtime.request_handling.model_sampling.ModelSamplingContextMiddleware.__call__(
    scope: starlette.types.Scope,
    receive: starlette.types.Receive,
    send: starlette.types.Send
) -> None
```

async

```python
class nemo_gym.server_runtime.request_handling.model_sampling.ModelSamplingMiddleware(
    app: starlette.types.ASGIApp,
    required: bool = False
)
```

Apply a model call's sampling segment on a Model Server, before policy admission and capture see the call.

Values replace the request's `temperature` and `top_p`. Their `max_output_tokens` caps the request's own
output-token limit, so a smaller limit the caller set is kept. `null` leaves the request as its caller sent it.
Either way the segment is stripped from the path and stays the current segment while the server handles the
call. With `required`, a generation call without a segment fails.

Only the body of a POST to a generation route is rewritten; other routes keep their body. Responses, streamed or
not, pass through unchanged.

```python
nemo_gym.server_runtime.request_handling.model_sampling.ModelSamplingMiddleware.__call__(
    scope: starlette.types.Scope,
    receive: starlette.types.Receive,
    send: starlette.types.Send
) -> None
```

async

```python
nemo_gym.server_runtime.request_handling.model_sampling.ModelSamplingMiddleware._apply(
    scope: starlette.types.Scope,
    receive: starlette.types.Receive,
    send: starlette.types.Send,
    sampling: nemo_gym.core.model_call_sampling.ModelCallSampling,
    limit_fields: tuple[str, ...]
) -> None
```

async

```python
class nemo_gym.server_runtime.request_handling.model_sampling._SamplingPath()
```

**Bases:** `NamedTuple`

**`api_path`** `str`

---

**`routed_path`** `str`

---

**`sampling`** `Optional[ModelCallSampling]`

---

**`segment`** `str`

---

```python
nemo_gym.server_runtime.request_handling.model_sampling._api_path(
    route_path: str
) -> str
```

The path a request routes to after its correlation prefix, such as /v1/responses.

```python
nemo_gym.server_runtime.request_handling.model_sampling._parse_sampling_path(
    scope: starlette.types.Scope
) -> typing.Optional[nemo_gym.server_runtime.request_handling.model_sampling._SamplingPath]
```

Split off a request path's sampling segment, or return `None` for a path without one.

Raises `ValueError` for a malformed segment.

```python
nemo_gym.server_runtime.request_handling.model_sampling._reject(
    scope: starlette.types.Scope,
    receive: starlette.types.Receive,
    send: starlette.types.Send,
    detail: str
) -> None
```

async

```python
nemo_gym.server_runtime.request_handling.model_sampling._route_path(
    scope: starlette.types.Scope
) -> tuple[str, str]
```

Split the request path into a relay's mount path (`root_path`) and the path this app routes.

```python
nemo_gym.server_runtime.request_handling.model_sampling._TOKEN_LIMIT_FIELDS = {'/v1/chat/completions': ('max_tokens', 'max_completion_tokens'), '/v1/responses...
```