nemo_gym.h2_ping_sidecar.config

Schema for the sidecar: block of a NeMo Gym config.

NVCF endpoints sit behind AWS Global Accelerator, which drops a connection that carries no application data for 340s. Gym’s aiohttp client cannot send HTTP/2 PINGs, so Gym points its model base URLs at a local sidecar that forwards over HTTP/2 and sends the PINGs itself.

This module imports only pydantic, so it is safe to import from Gym’s config machinery.

Module Contents

Classes

Name	Description
`H2PingSidecarConfig`	Configuration of the HTTP/2 PING sidecar that `gym env start` can run.
`SidecarInstanceConfig`	One proxy: a loopback listener that forwards to a single HTTPS upstream.

Functions

Name	Description
`_normalize_duration`	-
`is_loopback_host`	Whether `host` (a name or IP literal, optionally in brackets) only accepts local connections.
`parse_duration_seconds`	Seconds in a Go-style duration such as `60s`, `15m` or `1m30s`.
`upstream_origin`	`scheme://host[:port]` of an https URL; raises `ValueError` for anything else.

Data

GLOBAL_ACCELERATOR_IDLE_TIMEOUT_SECONDS

H2_PING_SIDECAR_KEY_NAME

LOAD_BALANCER_KEEP_ALIVE_SECONDS

_DEFAULT_LISTEN

_DURATION_PART

_DURATION_UNIT_SECONDS

_MEMORY_LIMIT_PATTERN

_NAME_PATTERN

API

class nemo_gym.h2_ping_sidecar.config.H2PingSidecarConfig()

Bases: BaseModel

Configuration of the HTTP/2 PING sidecar that gym env start can run.

binary

Optional[str] = None

Path to the compiled h2-ping-sidecar. Defaults to a build in Gym’s cache directory (see build_if_missing). It only needs to exist on the node that runs gym env start.

build_if_missing

bool = True

Run go build on the launching node when the binary does not exist yet. Needs Go on PATH.

enabled

bool = False

Master switch. When false Gym starts nothing and leaves every URL untouched.

gomaxprocs

Optional[int] = Field(default=None, ge=1)

CPU threads the Go runtime may use, passed as GOMAXPROCS. Only needed to cap the Go runtime below the container’s CPU quota; setting it disables the runtime’s automatic updates.

gomemlimit

Optional[str] = None

Soft memory limit for the Go runtime, passed as GOMEMLIMIT (for example 1GiB).

insecure_skip_verify

bool = False

Skip upstream TLS verification. For tests only.

instances

List[SidecarInstanceConfig] = Field(default_factory=list)

One entry per upstream the sidecar forwards to; required when enabled. A model URL is routed through an instance when its host matches the instance’s upstream. Give each instance its own listen port.

log_dir

Optional[str] = None

Directory for h2ping-<instance>-<host>.log. Defaults to nemo_gym_log_dir, then a temp dir.

max_conn_age

str = '50m'

Retire the upstream connection after 90-100% of this age: in-flight requests finish on it and new requests use a fresh connection, at the cost of one TLS handshake. Keep it below the load balancer’s client keep-alive limit (3600s by default) so large bodies never meet its GOAWAY. 0s disables.

ping_interval

str = '60s'

Send an HTTP/2 PING after this much read-idle time. Must stay well below 340s.

ping_timeout

str = '15s'

Close the upstream connection if a PING is not acknowledged within this time.

retry_body_limit

int = Field(default=(16 * 1024 * 1024), ge=0)

Request bodies up to this many bytes are buffered in memory so a request refused with a graceful GOAWAY can be re-sent on a fresh connection. Larger bodies stream through and are not retried. Peak memory is roughly this limit times the number of concurrent requests. 0 disables buffering.

rewrite_base_urls

bool = True

Rewrite policy_base_url and every *base_url under responses_api_models that points at an instance’s upstream so it targets that instance’s local address instead.

shutdown_grace

str = '60s'

On shutdown, let in-flight requests finish for up to this long before the sidecar exits.

startup_timeout_seconds

float = Field(default=30.0, gt=0)

Wait this long for each sidecar to bind its port before failing the run.

nemo_gym.h2_ping_sidecar.config.H2PingSidecarConfig._check_consistency() -> nemo_gym.h2_ping_sidecar.config.H2PingSidecarConfig

nemo_gym.h2_ping_sidecar.config.H2PingSidecarConfig._check_duration(
    value: typing.Union[str, int, float]
) -> str

classmethod

nemo_gym.h2_ping_sidecar.config.H2PingSidecarConfig._check_gomemlimit(
    value: typing.Optional[str]
) -> typing.Optional[str]

classmethod

class nemo_gym.h2_ping_sidecar.config.SidecarInstanceConfig()

Bases: BaseModel

One proxy: a loopback listener that forwards to a single HTTPS upstream.

listen

str = _DEFAULT_LISTEN

host:port the sidecar binds. Keep it on loopback: the sidecar adds no authentication.

name

str = 'nvcf'

Used in log file names and error messages.

upstream

str

HTTPS URL to forward to. Only scheme://host[:port] is used; a path such as /v1 is dropped, because clients keep sending it on every request.

nemo_gym.h2_ping_sidecar.config.SidecarInstanceConfig._check_listen(
    value: str
) -> str

classmethod

nemo_gym.h2_ping_sidecar.config.SidecarInstanceConfig._check_name(
    value: str
) -> str

classmethod

nemo_gym.h2_ping_sidecar.config.SidecarInstanceConfig._check_upstream(
    value: str
) -> str

classmethod

nemo_gym.h2_ping_sidecar.config._normalize_duration(
    value: typing.Union[str, int, float]
) -> str

nemo_gym.h2_ping_sidecar.config.is_loopback_host(
    host: str
) -> bool

Whether host (a name or IP literal, optionally in brackets) only accepts local connections.

nemo_gym.h2_ping_sidecar.config.parse_duration_seconds(
    value: str
) -> float

Seconds in a Go-style duration such as 60s, 15m or 1m30s.

nemo_gym.h2_ping_sidecar.config.upstream_origin(
    url: str
) -> str

scheme://host[:port] of an https URL; raises ValueError for anything else.

nemo_gym.h2_ping_sidecar.config.GLOBAL_ACCELERATOR_IDLE_TIMEOUT_SECONDS = 340.0

nemo_gym.h2_ping_sidecar.config.H2_PING_SIDECAR_KEY_NAME = 'sidecar'

nemo_gym.h2_ping_sidecar.config.LOAD_BALANCER_KEEP_ALIVE_SECONDS = 3600.0

nemo_gym.h2_ping_sidecar.config._DEFAULT_LISTEN = '127.0.0.1:1250'

nemo_gym.h2_ping_sidecar.config._DURATION_PART = re.compile('(\\d+(?:\\.\\d+)?)(ms|s|m|h)')

nemo_gym.h2_ping_sidecar.config._DURATION_UNIT_SECONDS = {'ms': 0.001, 's': 1.0, 'm': 60.0, 'h': 3600.0}

nemo_gym.h2_ping_sidecar.config._MEMORY_LIMIT_PATTERN = re.compile('^\\d+(B|KiB|MiB|GiB|TiB)?$')

nemo_gym.h2_ping_sidecar.config._NAME_PATTERN = re.compile('^[A-Za-z0-9][A-Za-z0-9_-]*$')