nemo_gym.telemetry.gym_metrics

Attributed OTel instruments that nemo.lens.instruments.gym.record_gym_metrics cannot express.

nemo_gym.telemetry.metrics forwards to the five fixed, undimensioned instruments nemo-lens declares. Anything that needs an attribute (a provider, a site, a class) is created here, directly on the lens meter, and cached per meter so a re-initialised telemetry handle gets fresh instruments. Every recorder is a no-op unless telemetry is initialised and exporting, and never raises into its caller. Call sites use the relevant span-group or metrics-export gate so disabled telemetry stays off the hot path.

Sandbox lifecycle

gym.sandbox.active (up-down counter): sandboxes this process currently holds, +1 when a provider hands back a handle, -1 when Gym releases it. Summed by the backend across the processes that hold sandboxes; a gauge would show whichever process wrote last.

gym.sandbox.startup_duration_ms / gym.sandbox.exec_duration_ms (histograms): wall-clock of one provisioning and one command. Explicit bucket boundaries up to thirty minutes: the SDK default stops at ten seconds and a sandbox start routinely takes a minute.

gym.sandbox.create_retry_total (counter): one per create attempt a provider retried. Retrying is provider-internal, so each provider that retries records it from its own loop.

All four carry nemo.gym.sandbox.provider.

Server startup

gym.server.startup_stage_duration_ms (histogram): one stage of one server’s own startup, recorded by the process that ran it. nemo.gym.startup.stage names the stage, and nemo.gym.server.worker marks a Uvicorn worker of a multi-worker server. Each point also carries nemo.gym.server.name and nemo.gym.server.type. Stages are contiguous, so one process’s stages add up to the time from its start until it accepts connections. A server process starts when the supervisor spawns it, and a worker starts when its main process hands off to Uvicorn. The same stages are spans under the startup span group. There is no supervisor-side total, because the supervisor only learns that a server is ready from a periodic health poll. That total would be these stages plus polling delay.

HTTP connection pool

gym.http.connection_pool.queue_duration_ms (histogram): connection-acquisition wait for one queued connection acquisition, combining repeated waiter wakeups within it. Queued redirect hops, aiohttp reconnects, and Gym retries each record separate samples. Connection attempts that acquire a slot immediately do not record a queue-duration histogram sample. Bounded attributes identify the binding connector limit, queue outcome, and destination server.

gym.http.connection_pool.connect_total (observable counter): all connection attempts (connect() calls), including attempts that later fail or are abandoned. Compare its value with the queue-duration histogram count to calculate the queued fraction.

Rollout outcomes

gym.rollout.completed_total (counter): one per rollout attempt the collection driver finished handling, recorded after it wrote the attempt or decided to omit it. An attempt is one dispatch of one row in one collection run; a retry on a later resume run is another attempt, and an HTTP-level re-send of /run inside one dispatch is not. Only run_from_config (gym eval run with serving) records it: --no-serve initialises no telemetry, and run_examples callers, reverification and the GDPVal multistage driver do not call the recorder.

nemo.gym.rollout.outcome is scored (main output, masked and forced-zero results included), failed (failures sidecar) or omitted (no-persist, e.g. kill-shaped). Rows drained by the dispatch budget never ran and are not counted, unlike the progress metrics, whose dropped groups failed with an omitted that includes drained rows. A failed /run is a failed attempt with class agent_request_failed only with route_failures_to_sidecar on; otherwise it aborts the run and is not counted. Coverage comes from the output files, not this counter.

Attributes are bounded: nemo.gym.rollout.dispatch.name (the environment server, or the agent when rows name no environment server) and, on attempts that carry a failure class, nemo.gym.failure_class, nemo.gym.failure_kind (registered or <server>:<kind> namespaced, any other value recorded as unregistered) and nemo.gym.failure_type (exception class name). Free-text reasons are never attributes.

Module Contents

Functions

Name	Description
`_get_or_create`	-
`_meter`	-
`_record_counter`	-
`_record_histogram`	-
`_record_up_down_counter`	-
`_reset_for_testing`	Drop cached instruments. Test-only.
`_server_attributes`	-
`record_http_connection_pool_queue_duration`	Record one queued connection acquisition.
`record_rollout_completed`	Count one finished rollout attempt in `gym.rollout.completed_total`.
`record_sandbox_active`	Add `delta` (`+1` on start, `-1` on stop) to `gym.sandbox.active` for `provider`.
`record_sandbox_create_retry`	Count one retried sandbox-create attempt, by provider.
`record_sandbox_exec_duration`	Record one command’s wall-clock inside an already-provisioned sandbox, by provider.
`record_sandbox_startup`	Record one sandbox’s provisioning wall-clock, by provider.
`record_server_startup_stage`	Record one stage of one server’s own startup, from the process that ran it.
`register_http_connection_pool_connect_counter`	Export cumulative connection-attempt counts without an OTel call on each connect.

Data

DISPATCH_NAME_ATTRIBUTE

FAILURE_CLASS_ATTRIBUTE

FAILURE_KIND_ATTRIBUTE

FAILURE_TYPE_ATTRIBUTE

HTTP_CONNECTION_POOL_CONNECT_INSTRUMENT

HTTP_CONNECTION_POOL_QUEUE_CONSTRAINT_ATTRIBUTE

HTTP_CONNECTION_POOL_QUEUE_DURATION_BOUNDARIES_MS

HTTP_CONNECTION_POOL_QUEUE_DURATION_INSTRUMENT

HTTP_CONNECTION_POOL_QUEUE_OUTCOME_ATTRIBUTE

HTTP_DESTINATION_SERVER_NAME_ATTRIBUTE

ROLLOUT_COMPLETED_INSTRUMENT

ROLLOUT_OUTCOME_ATTRIBUTE

SANDBOX_ACTIVE_INSTRUMENT

SANDBOX_CREATE_RETRY_INSTRUMENT

SANDBOX_DURATION_BOUNDARIES_MS

SANDBOX_EXEC_INSTRUMENT

SANDBOX_PROVIDER_ATTRIBUTE

SANDBOX_STARTUP_INSTRUMENT

SERVER_NAME_ATTRIBUTE

SERVER_STARTUP_BOUNDARIES_MS

SERVER_STARTUP_STAGE_INSTRUMENT

SERVER_TYPE_ATTRIBUTE

STARTUP_STAGE_ATTRIBUTE

UNREGISTERED_FAILURE_KIND

API

nemo_gym.telemetry.gym_metrics._get_or_create(
    meter: typing.Any,
    name: str,
    factory: collections.abc.Callable[[], typing.Any]
) -> typing.Any

nemo_gym.telemetry.gym_metrics._meter() -> typing.Optional[typing.Any]

nemo_gym.telemetry.gym_metrics._record_counter(
    name: str,
    description: str,
    attributes: dict[str, typing.Any],
    amount: int = 1
) -> None

nemo_gym.telemetry.gym_metrics._record_histogram(
    name: str,
    unit: str,
    description: str,
    value: float,
    attributes: dict[str, typing.Any],
    boundaries: collections.abc.Sequence[float] | None = None
) -> None

nemo_gym.telemetry.gym_metrics._record_up_down_counter(
    name: str,
    unit: str,
    description: str,
    delta: int,
    attributes: dict[str, typing.Any]
) -> None

nemo_gym.telemetry.gym_metrics._reset_for_testing() -> None

Drop cached instruments. Test-only.

nemo_gym.telemetry.gym_metrics._server_attributes(
    server_name: str,
    server_type: typing.Optional[str]
) -> dict[str, typing.Any]

nemo_gym.telemetry.gym_metrics.record_http_connection_pool_queue_duration(
    duration_ms: float,
    queue_constraint: str,
    queue_outcome: str,
    server_name: str
) -> None

Record one queued connection acquisition.

nemo_gym.telemetry.gym_metrics.record_rollout_completed(
    outcome: str,
    dispatch_name: str,
    failure_class: str | None = None,
    failure_kind: str | None = None,
    failure_type: str | None = None
) -> None

Count one finished rollout attempt in gym.rollout.completed_total.

nemo_gym.telemetry.gym_metrics.record_sandbox_active(
    delta: int,
    provider: str
) -> None

Add delta (+1 on start, -1 on stop) to gym.sandbox.active for provider.

nemo_gym.telemetry.gym_metrics.record_sandbox_create_retry(
    provider: str
) -> None

Count one retried sandbox-create attempt, by provider.

nemo_gym.telemetry.gym_metrics.record_sandbox_exec_duration(
    duration_ms: float,
    provider: str
) -> None

Record one command’s wall-clock inside an already-provisioned sandbox, by provider.

nemo_gym.telemetry.gym_metrics.record_sandbox_startup(
    duration_ms: float,
    provider: str
) -> None

Record one sandbox’s provisioning wall-clock, by provider.

nemo_gym.telemetry.gym_metrics.record_server_startup_stage(
    duration_ms: float,
    stage: str,
    server_name: str,
    server_type: typing.Optional[str],
    worker: bool = False
) -> None

Record one stage of one server’s own startup, from the process that ran it.

worker is true for a Uvicorn worker of a multi-worker server. A worker’s stages are timed separately from its main process’s.

nemo_gym.telemetry.gym_metrics.register_http_connection_pool_connect_counter(
    snapshot: collections.abc.Callable[[], dict[str, int]]
) -> None

Export cumulative connection-attempt counts without an OTel call on each connect.

nemo_gym.telemetry.gym_metrics.DISPATCH_NAME_ATTRIBUTE = 'nemo.gym.rollout.dispatch.name'

nemo_gym.telemetry.gym_metrics.FAILURE_CLASS_ATTRIBUTE = 'nemo.gym.failure_class'

nemo_gym.telemetry.gym_metrics.FAILURE_KIND_ATTRIBUTE = 'nemo.gym.failure_kind'

nemo_gym.telemetry.gym_metrics.FAILURE_TYPE_ATTRIBUTE = 'nemo.gym.failure_type'

nemo_gym.telemetry.gym_metrics.HTTP_CONNECTION_POOL_CONNECT_INSTRUMENT = 'gym.http.connection_pool.connect_total'

nemo_gym.telemetry.gym_metrics.HTTP_CONNECTION_POOL_QUEUE_CONSTRAINT_ATTRIBUTE = 'nemo.gym.http.connection_pool.queue_constraint'

nemo_gym.telemetry.gym_metrics.HTTP_CONNECTION_POOL_QUEUE_DURATION_BOUNDARIES_MS: tuple[float, ...] = (0.01, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10, 25, 50, 100, 250, 500, 1000, 5000, 3...

nemo_gym.telemetry.gym_metrics.HTTP_CONNECTION_POOL_QUEUE_DURATION_INSTRUMENT = 'gym.http.connection_pool.queue_duration_ms'

nemo_gym.telemetry.gym_metrics.HTTP_CONNECTION_POOL_QUEUE_OUTCOME_ATTRIBUTE = 'nemo.gym.http.connection_pool.queue_outcome'

nemo_gym.telemetry.gym_metrics.HTTP_DESTINATION_SERVER_NAME_ATTRIBUTE = 'nemo.gym.http.destination.server.name'

nemo_gym.telemetry.gym_metrics.ROLLOUT_COMPLETED_INSTRUMENT = 'gym.rollout.completed_total'

nemo_gym.telemetry.gym_metrics.ROLLOUT_OUTCOME_ATTRIBUTE = 'nemo.gym.rollout.outcome'

nemo_gym.telemetry.gym_metrics.SANDBOX_ACTIVE_INSTRUMENT = 'gym.sandbox.active'

nemo_gym.telemetry.gym_metrics.SANDBOX_CREATE_RETRY_INSTRUMENT = 'gym.sandbox.create_retry_total'

nemo_gym.telemetry.gym_metrics.SANDBOX_DURATION_BOUNDARIES_MS: tuple[float, ...] = (250, 500, 1000, 2000, 5000, 10000, 30000, 60000, 120000, 300000, 600000, 180000...

nemo_gym.telemetry.gym_metrics.SANDBOX_EXEC_INSTRUMENT = 'gym.sandbox.exec_duration_ms'

nemo_gym.telemetry.gym_metrics.SANDBOX_PROVIDER_ATTRIBUTE = 'nemo.gym.sandbox.provider'

nemo_gym.telemetry.gym_metrics.SANDBOX_STARTUP_INSTRUMENT = 'gym.sandbox.startup_duration_ms'

nemo_gym.telemetry.gym_metrics.SERVER_NAME_ATTRIBUTE = 'nemo.gym.server.name'

nemo_gym.telemetry.gym_metrics.SERVER_STARTUP_BOUNDARIES_MS: tuple[float, ...] = (100, 250, 500, 1000, 2500, 5000, 10000, 30000, 60000, 120000, 300000, 600000, 1...

nemo_gym.telemetry.gym_metrics.SERVER_STARTUP_STAGE_INSTRUMENT = 'gym.server.startup_stage_duration_ms'

nemo_gym.telemetry.gym_metrics.SERVER_TYPE_ATTRIBUTE = 'nemo.gym.server.type'

nemo_gym.telemetry.gym_metrics.STARTUP_STAGE_ATTRIBUTE = 'nemo.gym.startup.stage'

nemo_gym.telemetry.gym_metrics.UNREGISTERED_FAILURE_KIND = 'unregistered'

nemo_gym.telemetry.gym_metrics.WORKER_ATTRIBUTE = 'nemo.gym.server.worker'

nemo_gym.telemetry.gym_metrics._INSTRUMENTS: dict[int, dict[str, Any]] = {}

nemo_gym.telemetry.gym_metrics._INSTRUMENT_LOCK = threading.Lock()

nemo_gym.telemetry.gym_metrics.logger = logging.getLogger(__name__)