nemo_gym.telemetry.gym_metrics

Attributed OTel instruments that nemo.lens.instruments.gym.record_gym_metrics cannot express.

nemo_gym.telemetry.metrics forwards to the five fixed, undimensioned instruments nemo-lens declares. Anything that needs an attribute (a provider, a site, a class) is created here, directly on the lens meter, and cached per meter so a re-initialised telemetry handle gets fresh instruments. Every recorder is a no-op unless telemetry is initialised and exporting, and never raises into its caller. Call sites use the relevant span-group or metrics-export gate so disabled telemetry stays off the hot path.

Sandbox lifecycle

gym.sandbox.active (up-down counter): sandboxes this process currently holds, +1 when a provider hands back a handle, -1 when Gym releases it. Summed by the backend across the processes that hold sandboxes; a gauge would show whichever process wrote last.

gym.sandbox.startup_duration_ms / gym.sandbox.exec_duration_ms (histograms): wall-clock of one provisioning and one command. Explicit bucket boundaries up to thirty minutes: the SDK default stops at ten seconds and a sandbox start routinely takes a minute.

gym.sandbox.create_retry_total (counter): one per create attempt a provider retried. Retrying is provider-internal, so each provider that retries records it from its own loop.

All four carry nemo.gym.sandbox.provider.

HTTP connection pool

gym.http.connection_pool.queue_duration_ms (histogram): connection-acquisition wait for one queued connection acquisition, combining repeated waiter wakeups within it. Queued redirect hops, aiohttp reconnects, and Gym retries each record separate samples. Connection attempts that acquire a slot immediately do not record a queue-duration histogram sample. Bounded attributes identify the binding connector limit, queue outcome, and destination server.

gym.http.connection_pool.connect_total (observable counter): all connection attempts (connect() calls), including attempts that later fail or are abandoned. Compare its value with the queue-duration histogram count to calculate the queued fraction.

Module Contents

Functions

Name	Description
`_get_or_create`	-
`_meter`	-
`_record_counter`	-
`_record_histogram`	-
`_record_up_down_counter`	-
`_reset_for_testing`	Drop cached instruments. Test-only.
`record_http_connection_pool_queue_duration`	Record one queued connection acquisition.
`record_sandbox_active`	Add `delta` (`+1` on start, `-1` on stop) to `gym.sandbox.active` for `provider`.
`record_sandbox_create_retry`	Count one retried sandbox-create attempt, by provider.
`record_sandbox_exec_duration`	Record one command’s wall-clock inside an already-provisioned sandbox, by provider.
`record_sandbox_startup`	Record one sandbox’s provisioning wall-clock, by provider.
`register_http_connection_pool_connect_counter`	Export cumulative connection-attempt counts without an OTel call on each connect.

Data

HTTP_CONNECTION_POOL_CONNECT_INSTRUMENT

HTTP_CONNECTION_POOL_QUEUE_CONSTRAINT_ATTRIBUTE

HTTP_CONNECTION_POOL_QUEUE_DURATION_BOUNDARIES_MS

HTTP_CONNECTION_POOL_QUEUE_DURATION_INSTRUMENT

HTTP_CONNECTION_POOL_QUEUE_OUTCOME_ATTRIBUTE

HTTP_DESTINATION_SERVER_NAME_ATTRIBUTE

SANDBOX_ACTIVE_INSTRUMENT

SANDBOX_CREATE_RETRY_INSTRUMENT

SANDBOX_DURATION_BOUNDARIES_MS

SANDBOX_EXEC_INSTRUMENT

SANDBOX_PROVIDER_ATTRIBUTE

SANDBOX_STARTUP_INSTRUMENT

_INSTRUMENTS

_INSTRUMENT_LOCK

logger

API

nemo_gym.telemetry.gym_metrics._get_or_create(
    meter: typing.Any,
    name: str,
    factory: collections.abc.Callable[[], typing.Any]
) -> typing.Any

nemo_gym.telemetry.gym_metrics._meter() -> typing.Optional[typing.Any]

nemo_gym.telemetry.gym_metrics._record_counter(
    name: str,
    description: str,
    attributes: dict[str, typing.Any],
    amount: int = 1
) -> None

nemo_gym.telemetry.gym_metrics._record_histogram(
    name: str,
    unit: str,
    description: str,
    value: float,
    attributes: dict[str, typing.Any],
    boundaries: collections.abc.Sequence[float] | None = None
) -> None

nemo_gym.telemetry.gym_metrics._record_up_down_counter(
    name: str,
    unit: str,
    description: str,
    delta: int,
    attributes: dict[str, typing.Any]
) -> None

nemo_gym.telemetry.gym_metrics._reset_for_testing() -> None

Drop cached instruments. Test-only.

nemo_gym.telemetry.gym_metrics.record_http_connection_pool_queue_duration(
    duration_ms: float,
    queue_constraint: str,
    queue_outcome: str,
    server_name: str
) -> None

Record one queued connection acquisition.

nemo_gym.telemetry.gym_metrics.record_sandbox_active(
    delta: int,
    provider: str
) -> None

Add delta (+1 on start, -1 on stop) to gym.sandbox.active for provider.

nemo_gym.telemetry.gym_metrics.record_sandbox_create_retry(
    provider: str
) -> None

Count one retried sandbox-create attempt, by provider.

nemo_gym.telemetry.gym_metrics.record_sandbox_exec_duration(
    duration_ms: float,
    provider: str
) -> None

Record one command’s wall-clock inside an already-provisioned sandbox, by provider.

nemo_gym.telemetry.gym_metrics.record_sandbox_startup(
    duration_ms: float,
    provider: str
) -> None

Record one sandbox’s provisioning wall-clock, by provider.

nemo_gym.telemetry.gym_metrics.register_http_connection_pool_connect_counter(
    snapshot: collections.abc.Callable[[], dict[str, int]]
) -> None

Export cumulative connection-attempt counts without an OTel call on each connect.

nemo_gym.telemetry.gym_metrics.HTTP_CONNECTION_POOL_CONNECT_INSTRUMENT = 'gym.http.connection_pool.connect_total'

nemo_gym.telemetry.gym_metrics.HTTP_CONNECTION_POOL_QUEUE_CONSTRAINT_ATTRIBUTE = 'nemo.gym.http.connection_pool.queue_constraint'

nemo_gym.telemetry.gym_metrics.HTTP_CONNECTION_POOL_QUEUE_DURATION_BOUNDARIES_MS: tuple[float, ...] = (0.01, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10, 25, 50, 100, 250, 500, 1000, 5000, 3...

nemo_gym.telemetry.gym_metrics.HTTP_CONNECTION_POOL_QUEUE_DURATION_INSTRUMENT = 'gym.http.connection_pool.queue_duration_ms'

nemo_gym.telemetry.gym_metrics.HTTP_CONNECTION_POOL_QUEUE_OUTCOME_ATTRIBUTE = 'nemo.gym.http.connection_pool.queue_outcome'

nemo_gym.telemetry.gym_metrics.HTTP_DESTINATION_SERVER_NAME_ATTRIBUTE = 'nemo.gym.http.destination.server.name'

nemo_gym.telemetry.gym_metrics.SANDBOX_ACTIVE_INSTRUMENT = 'gym.sandbox.active'

nemo_gym.telemetry.gym_metrics.SANDBOX_CREATE_RETRY_INSTRUMENT = 'gym.sandbox.create_retry_total'

nemo_gym.telemetry.gym_metrics.SANDBOX_DURATION_BOUNDARIES_MS: tuple[float, ...] = (250, 500, 1000, 2000, 5000, 10000, 30000, 60000, 120000, 300000, 600000, 180000...

nemo_gym.telemetry.gym_metrics.SANDBOX_EXEC_INSTRUMENT = 'gym.sandbox.exec_duration_ms'

nemo_gym.telemetry.gym_metrics.SANDBOX_PROVIDER_ATTRIBUTE = 'nemo.gym.sandbox.provider'

nemo_gym.telemetry.gym_metrics.SANDBOX_STARTUP_INSTRUMENT = 'gym.sandbox.startup_duration_ms'

nemo_gym.telemetry.gym_metrics._INSTRUMENTS: dict[int, dict[str, Any]] = {}

nemo_gym.telemetry.gym_metrics._INSTRUMENT_LOCK = threading.Lock()

nemo_gym.telemetry.gym_metrics.logger = logging.getLogger(__name__)