nemo_gym.telemetry.gym_metrics

Attributed OTel instruments that nemo.lens.instruments.gym.record_gym_metrics cannot express.

nemo_gym.telemetry.metrics forwards to the five fixed, undimensioned instruments nemo-lens declares. Anything that needs an attribute (a provider, a site, a class) is created here, directly on the lens meter, and cached per meter so a re-initialised telemetry handle gets fresh instruments. Every recorder is a no-op unless telemetry is initialised and exporting, and never raises into its caller. Call sites use the relevant span-group or metrics-export gate so disabled telemetry stays off the hot path.

Sandbox lifecycle

gym.sandbox.active (up-down counter): sandboxes this process currently holds, +1 when a provider hands back a handle, -1 when Gym releases it. Summed by the backend across the processes that hold sandboxes; a gauge would show whichever process wrote last.

gym.sandbox.startup_duration_ms / gym.sandbox.exec_duration_ms (histograms): wall-clock of one provisioning and one command. Explicit bucket boundaries up to thirty minutes: the SDK default stops at ten seconds and a sandbox start routinely takes a minute.

gym.sandbox.create_retry_total (counter): one per create attempt a provider retried. Retrying is provider-internal, so each provider that retries records it from its own loop.

All four carry nemo.gym.sandbox.provider.

Server startup

gym.server.startup_stage_duration_ms (histogram): one stage of one server’s own startup, recorded by the process that ran it. nemo.gym.startup.stage names the stage, and nemo.gym.server.worker marks a Uvicorn worker of a multi-worker server. Each point also carries nemo.gym.server.name and nemo.gym.server.type. Stages are contiguous, so one process’s stages add up to the time from its start until it accepts connections. A server process starts when the supervisor spawns it, and a worker starts when its main process hands off to Uvicorn. The same stages are spans under the startup span group. There is no supervisor-side total, because the supervisor only learns that a server is ready from a periodic health poll. That total would be these stages plus polling delay.

HTTP connection pool

gym.http.connection_pool.queue_duration_ms (histogram): connection-acquisition wait for one queued connection acquisition, combining repeated waiter wakeups within it. Queued redirect hops, aiohttp reconnects, and Gym retries each record separate samples. Connection attempts that acquire a slot immediately do not record a queue-duration histogram sample. Bounded attributes identify the binding connector limit, queue outcome, and destination server.

gym.http.connection_pool.connect_duration_ms (histogram): time to open one new connection, including DNS, TCP, and TLS setup. Reused connections record nothing, so the count is the number of connections the pool had to open. A run that opens connections throughout, rather than only at its start, is losing them, for example to keep-alive timeouts. nemo.gym.http.connection_pool.connect_outcome is ok or failed.

gym.http.connection_pool.connect_total (observable counter): all connection attempts (connect() calls), including attempts that later fail or are abandoned. Compare its value with the queue-duration histogram count to calculate the queued fraction.

Rollout outcomes

gym.rollout.completed_total (counter): one per rollout attempt the collection driver finished handling, recorded after it wrote the attempt or decided to omit it. An attempt is one dispatch of one row in one collection run; a retry on a later resume run is another attempt, and an HTTP-level re-send of /run inside one dispatch is not. Only run_from_config (gym eval run with serving) records it: --no-serve initialises no telemetry, and run_examples callers, reverification and the GDPVal multistage driver do not call the recorder.

nemo.gym.rollout.outcome is scored (main output, masked and forced-zero results included), failed (failures sidecar) or omitted (no-persist, e.g. kill-shaped). Rows drained by the dispatch budget never ran and are not counted, unlike the progress metrics, whose dropped groups failed with an omitted that includes drained rows. A failed /run is a failed attempt with class agent_request_failed only with route_failures_to_sidecar on; otherwise it aborts the run and is not counted. Coverage comes from the output files, not this counter.

Attributes are bounded: nemo.gym.rollout.dispatch.name (the environment server, or the agent when rows name no environment server) and, on attempts that carry a failure class, nemo.gym.failure_class, nemo.gym.failure_kind (registered or <server>:<kind> namespaced, any other value recorded as unregistered) and nemo.gym.failure_type (exception class name). Free-text reasons are never attributes.

Rollout rewards

gym.rollout.rewarded_total (counter): one per scored attempt that carries a numeric reward, recorded with the outcome counter; a result without one is unscored, not a zero. Rewards are environment-defined floats: binary, partial credit, or negative penalties. nemo.gym.rollout.reward.range sorts each one into negative, zero, partial (strictly between 0 and 1), full (exactly 1) or above_one. Also carries nemo.gym.rollout.dispatch.name and nemo.gym.rollout.masked (the result’s mask_sample), so masked rewards can be left out as the aggregate metrics leave them out. A live view of the run, not its score: benchmark metrics such as pass@k come from /aggregate_metrics.

Rollout health

The trajectory-level checks of the rollout health report, evaluated as each rollout finishes. gym.rollout.health_verdict_total (counter): one per evaluated rollout, by nemo.gym.health.verdict (healthy, unhealthy or unobserved, as in the report) and nemo.gym.rollout.outcome (scored or failed; the post-run report reads scored rollouts only). gym.rollout.health_flagged_total (counter): one per rollout and check with at least one finding, by nemo.gym.health.check (a check id from the health-check registry) and nemo.gym.rollout.outcome, so a check’s count is the number of rollouts it flagged, not the number of findings. Checks in health_check_ignored_checks are not evaluated.

Module Contents

Functions

Name	Description
`_get_or_create`	-
`_meter`	-
`_record_counter`	-
`_record_histogram`	-
`_record_up_down_counter`	-
`_reset_for_testing`	Drop cached instruments. Test-only.
`_reward_range`	-
`_server_attributes`	-
`record_http_connection_pool_connect_duration`	Record the setup time of one new pooled connection.
`record_http_connection_pool_queue_duration`	Record one queued connection acquisition.
`record_rollout_completed`	Count one finished rollout attempt in `gym.rollout.completed_total`.
`record_rollout_health`	Count one health-evaluated rollout and each distinct check that flagged it.
`record_rollout_reward`	Count one scored attempt in `gym.rollout.rewarded_total` by the range its reward falls in.
`record_sandbox_active`	Add `delta` (`+1` on start, `-1` on stop) to `gym.sandbox.active` for `provider`.
`record_sandbox_create_retry`	Count one retried sandbox-create attempt, by provider.
`record_sandbox_exec_duration`	Record one command’s wall-clock inside an already-provisioned sandbox, by provider.
`record_sandbox_startup`	Record one sandbox’s provisioning wall-clock, by provider.
`record_server_startup_stage`	Record one stage of one server’s own startup, from the process that ran it.
`register_http_connection_pool_connect_counter`	Export cumulative connection-attempt counts without an OTel call on each connect.

Data

DISPATCH_NAME_ATTRIBUTE

FAILURE_CLASS_ATTRIBUTE

FAILURE_KIND_ATTRIBUTE

FAILURE_TYPE_ATTRIBUTE

HEALTH_CHECK_ATTRIBUTE

HEALTH_VERDICT_ATTRIBUTE

HTTP_CONNECTION_POOL_CONNECT_DURATION_INSTRUMENT

HTTP_CONNECTION_POOL_CONNECT_INSTRUMENT

HTTP_CONNECTION_POOL_CONNECT_OUTCOME_ATTRIBUTE

HTTP_CONNECTION_POOL_QUEUE_CONSTRAINT_ATTRIBUTE

HTTP_CONNECTION_POOL_QUEUE_DURATION_BOUNDARIES_MS

HTTP_CONNECTION_POOL_QUEUE_DURATION_INSTRUMENT

HTTP_CONNECTION_POOL_QUEUE_OUTCOME_ATTRIBUTE

HTTP_DESTINATION_SERVER_NAME_ATTRIBUTE

REWARD_RANGE_ATTRIBUTE

ROLLOUT_COMPLETED_INSTRUMENT

ROLLOUT_HEALTH_FLAGGED_INSTRUMENT

ROLLOUT_HEALTH_VERDICT_INSTRUMENT

ROLLOUT_MASKED_ATTRIBUTE

ROLLOUT_OUTCOME_ATTRIBUTE

ROLLOUT_REWARDED_INSTRUMENT

SANDBOX_ACTIVE_INSTRUMENT

SANDBOX_CREATE_RETRY_INSTRUMENT

SANDBOX_DURATION_BOUNDARIES_MS

SANDBOX_EXEC_INSTRUMENT

SANDBOX_PROVIDER_ATTRIBUTE

SANDBOX_STARTUP_INSTRUMENT

SERVER_NAME_ATTRIBUTE

SERVER_STARTUP_BOUNDARIES_MS

SERVER_STARTUP_STAGE_INSTRUMENT

SERVER_TYPE_ATTRIBUTE

STARTUP_STAGE_ATTRIBUTE

UNREGISTERED_FAILURE_KIND

API

nemo_gym.telemetry.gym_metrics._get_or_create(
    meter: typing.Any,
    name: str,
    factory: collections.abc.Callable[[], typing.Any]
) -> typing.Any

nemo_gym.telemetry.gym_metrics._meter() -> typing.Optional[typing.Any]

nemo_gym.telemetry.gym_metrics._record_counter(
    name: str,
    description: str,
    attributes: dict[str, typing.Any],
    amount: int = 1,
    unit: str = '1'
) -> None

nemo_gym.telemetry.gym_metrics._record_histogram(
    name: str,
    unit: str,
    description: str,
    value: float,
    attributes: dict[str, typing.Any],
    boundaries: collections.abc.Sequence[float] | None = None
) -> None

nemo_gym.telemetry.gym_metrics._record_up_down_counter(
    name: str,
    unit: str,
    description: str,
    delta: int,
    attributes: dict[str, typing.Any]
) -> None

nemo_gym.telemetry.gym_metrics._reset_for_testing() -> None

Drop cached instruments. Test-only.

nemo_gym.telemetry.gym_metrics._reward_range(
    reward: float
) -> str

nemo_gym.telemetry.gym_metrics._server_attributes(
    server_name: str,
    server_type: typing.Optional[str]
) -> dict[str, typing.Any]

nemo_gym.telemetry.gym_metrics.record_http_connection_pool_connect_duration(
    duration_ms: float,
    outcome: str,
    server_name: str
) -> None

Record the setup time of one new pooled connection.

nemo_gym.telemetry.gym_metrics.record_http_connection_pool_queue_duration(
    duration_ms: float,
    queue_constraint: str,
    queue_outcome: str,
    server_name: str
) -> None

Record one queued connection acquisition.

nemo_gym.telemetry.gym_metrics.record_rollout_completed(
    outcome: str,
    dispatch_name: str,
    failure_class: str | None = None,
    failure_kind: str | None = None,
    failure_type: str | None = None
) -> None

Count one finished rollout attempt in gym.rollout.completed_total.

nemo_gym.telemetry.gym_metrics.record_rollout_health(
    verdict: str,
    outcome: str,
    flagged_checks: collections.abc.Iterable[str]
) -> None

Count one health-evaluated rollout and each distinct check that flagged it.

nemo_gym.telemetry.gym_metrics.record_rollout_reward(
    reward: float,
    dispatch_name: str,
    masked: bool
) -> None

Count one scored attempt in gym.rollout.rewarded_total by the range its reward falls in.

nemo_gym.telemetry.gym_metrics.record_sandbox_active(
    delta: int,
    provider: str
) -> None

Add delta (+1 on start, -1 on stop) to gym.sandbox.active for provider.

nemo_gym.telemetry.gym_metrics.record_sandbox_create_retry(
    provider: str
) -> None

Count one retried sandbox-create attempt, by provider.

nemo_gym.telemetry.gym_metrics.record_sandbox_exec_duration(
    duration_ms: float,
    provider: str
) -> None

Record one command’s wall-clock inside an already-provisioned sandbox, by provider.

nemo_gym.telemetry.gym_metrics.record_sandbox_startup(
    duration_ms: float,
    provider: str
) -> None

Record one sandbox’s provisioning wall-clock, by provider.

nemo_gym.telemetry.gym_metrics.record_server_startup_stage(
    duration_ms: float,
    stage: str,
    server_name: str,
    server_type: typing.Optional[str],
    worker: bool = False
) -> None

Record one stage of one server’s own startup, from the process that ran it.

worker is true for a Uvicorn worker of a multi-worker server. A worker’s stages are timed separately from its main process’s.

nemo_gym.telemetry.gym_metrics.register_http_connection_pool_connect_counter(
    snapshot: collections.abc.Callable[[], dict[str, int]]
) -> None

Export cumulative connection-attempt counts without an OTel call on each connect.

nemo_gym.telemetry.gym_metrics.DISPATCH_NAME_ATTRIBUTE = 'nemo.gym.rollout.dispatch.name'

nemo_gym.telemetry.gym_metrics.FAILURE_CLASS_ATTRIBUTE = 'nemo.gym.failure_class'

nemo_gym.telemetry.gym_metrics.FAILURE_KIND_ATTRIBUTE = 'nemo.gym.failure_kind'

nemo_gym.telemetry.gym_metrics.FAILURE_TYPE_ATTRIBUTE = 'nemo.gym.failure_type'

nemo_gym.telemetry.gym_metrics.HEALTH_CHECK_ATTRIBUTE = 'nemo.gym.health.check'

nemo_gym.telemetry.gym_metrics.HEALTH_VERDICT_ATTRIBUTE = 'nemo.gym.health.verdict'

nemo_gym.telemetry.gym_metrics.HTTP_CONNECTION_POOL_CONNECT_DURATION_INSTRUMENT = 'gym.http.connection_pool.connect_duration_ms'

nemo_gym.telemetry.gym_metrics.HTTP_CONNECTION_POOL_CONNECT_INSTRUMENT = 'gym.http.connection_pool.connect_total'

nemo_gym.telemetry.gym_metrics.HTTP_CONNECTION_POOL_CONNECT_OUTCOME_ATTRIBUTE = 'nemo.gym.http.connection_pool.connect_outcome'

nemo_gym.telemetry.gym_metrics.HTTP_CONNECTION_POOL_QUEUE_CONSTRAINT_ATTRIBUTE = 'nemo.gym.http.connection_pool.queue_constraint'

nemo_gym.telemetry.gym_metrics.HTTP_CONNECTION_POOL_QUEUE_DURATION_BOUNDARIES_MS: tuple[float, ...] = (0.01, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10, 25, 50, 100, 250, 500, 1000, 5000, 3...

nemo_gym.telemetry.gym_metrics.HTTP_CONNECTION_POOL_QUEUE_DURATION_INSTRUMENT = 'gym.http.connection_pool.queue_duration_ms'

nemo_gym.telemetry.gym_metrics.HTTP_CONNECTION_POOL_QUEUE_OUTCOME_ATTRIBUTE = 'nemo.gym.http.connection_pool.queue_outcome'

nemo_gym.telemetry.gym_metrics.HTTP_DESTINATION_SERVER_NAME_ATTRIBUTE = 'nemo.gym.http.destination.server.name'

nemo_gym.telemetry.gym_metrics.REWARD_RANGE_ATTRIBUTE = 'nemo.gym.rollout.reward.range'

nemo_gym.telemetry.gym_metrics.ROLLOUT_COMPLETED_INSTRUMENT = 'gym.rollout.completed_total'

nemo_gym.telemetry.gym_metrics.ROLLOUT_HEALTH_FLAGGED_INSTRUMENT = 'gym.rollout.health_flagged_total'

nemo_gym.telemetry.gym_metrics.ROLLOUT_HEALTH_VERDICT_INSTRUMENT = 'gym.rollout.health_verdict_total'

nemo_gym.telemetry.gym_metrics.ROLLOUT_MASKED_ATTRIBUTE = 'nemo.gym.rollout.masked'

nemo_gym.telemetry.gym_metrics.ROLLOUT_OUTCOME_ATTRIBUTE = 'nemo.gym.rollout.outcome'

nemo_gym.telemetry.gym_metrics.ROLLOUT_REWARDED_INSTRUMENT = 'gym.rollout.rewarded_total'

nemo_gym.telemetry.gym_metrics.SANDBOX_ACTIVE_INSTRUMENT = 'gym.sandbox.active'

nemo_gym.telemetry.gym_metrics.SANDBOX_CREATE_RETRY_INSTRUMENT = 'gym.sandbox.create_retry_total'

nemo_gym.telemetry.gym_metrics.SANDBOX_DURATION_BOUNDARIES_MS: tuple[float, ...] = (250, 500, 1000, 2000, 5000, 10000, 30000, 60000, 120000, 300000, 600000, 180000...

nemo_gym.telemetry.gym_metrics.SANDBOX_EXEC_INSTRUMENT = 'gym.sandbox.exec_duration_ms'

nemo_gym.telemetry.gym_metrics.SANDBOX_PROVIDER_ATTRIBUTE = 'nemo.gym.sandbox.provider'

nemo_gym.telemetry.gym_metrics.SANDBOX_STARTUP_INSTRUMENT = 'gym.sandbox.startup_duration_ms'

nemo_gym.telemetry.gym_metrics.SERVER_NAME_ATTRIBUTE = 'nemo.gym.server.name'

nemo_gym.telemetry.gym_metrics.SERVER_STARTUP_BOUNDARIES_MS: tuple[float, ...] = (100, 250, 500, 1000, 2500, 5000, 10000, 30000, 60000, 120000, 300000, 600000, 1...

nemo_gym.telemetry.gym_metrics.SERVER_STARTUP_STAGE_INSTRUMENT = 'gym.server.startup_stage_duration_ms'

nemo_gym.telemetry.gym_metrics.SERVER_TYPE_ATTRIBUTE = 'nemo.gym.server.type'

nemo_gym.telemetry.gym_metrics.STARTUP_STAGE_ATTRIBUTE = 'nemo.gym.startup.stage'

nemo_gym.telemetry.gym_metrics.UNREGISTERED_FAILURE_KIND = 'unregistered'

nemo_gym.telemetry.gym_metrics.WORKER_ATTRIBUTE = 'nemo.gym.server.worker'

nemo_gym.telemetry.gym_metrics._INSTRUMENTS: dict[int, dict[str, Any]] = {}

nemo_gym.telemetry.gym_metrics._INSTRUMENT_LOCK = threading.Lock()

nemo_gym.telemetry.gym_metrics.logger = logging.getLogger(__name__)