nemo_gym.telemetry.gym_metrics

Attributed OTel instruments that nemo.lens.instruments.gym.record_gym_metrics cannot express.

nemo_gym.telemetry.metrics forwards to the five fixed, undimensioned instruments nemo-lens declares. Anything that needs an attribute (a provider, a site, a class) is created here, directly on the lens meter, and cached per meter so a re-initialised telemetry handle gets fresh instruments. Every recorder is a no-op unless telemetry is initialised and exporting, and never raises into its caller. Call sites use the relevant span-group or metrics-export gate so disabled telemetry stays off the hot path.

Sandbox lifecycle

gym.sandbox.active (up-down counter): sandboxes this process currently holds, +1 when a provider hands back a handle, -1 when Gym releases it. Summed by the backend across the processes that hold sandboxes; a gauge would show whichever process wrote last.

gym.sandbox.startup_duration_ms / gym.sandbox.exec_duration_ms (histograms): wall-clock of one provisioning and one command. Explicit bucket boundaries up to thirty minutes: the SDK default stops at ten seconds and a sandbox start routinely takes a minute.

gym.sandbox.create_retry_total (counter): one per create attempt a provider retried. Retrying is provider-internal, so each provider that retries records it from its own loop.

gym.sandbox.transfer_duration_ms (histogram): wall-clock of one file transfer into or out of a sandbox, including the seed files AsyncSandbox.start() writes after the provider returns the sandbox. It carries nemo.gym.sandbox.transfer.direction (upload or download) and nemo.gym.sandbox.transfer.outcome (ok, error or cancelled). Sidecar downloads are not counted.

gym.sandbox.transfer_bytes_total (counter, bytes): the size of each regular file a transfer moved successfully, by direction.

All six carry nemo.gym.sandbox.provider.

Server startup

gym.server.startup_stage_duration_ms (histogram): one stage of one server’s own startup, recorded by the process that ran it. nemo.gym.startup.stage names the stage, and nemo.gym.server.worker marks a Uvicorn worker of a multi-worker server. Each point also carries nemo.gym.server.name and nemo.gym.server.type. Stages are contiguous, so one process’s stages add up to the time from its start until it accepts connections. A server process starts when the supervisor spawns it, and a worker starts when its main process hands off to Uvicorn. The same stages are spans under the startup span group. There is no supervisor-side total, because the supervisor only learns that a server is ready from a periodic health poll. That total would be these stages plus polling delay.

HTTP connection pool

gym.http.connection_pool.queue_duration_ms (histogram): connection-acquisition wait for one queued connection acquisition, combining repeated waiter wakeups within it. Queued redirect hops, aiohttp reconnects, and Gym retries each record separate samples. Connection attempts that acquire a slot immediately do not record a queue-duration histogram sample. Bounded attributes identify the binding connector limit, queue outcome, and destination server.

gym.http.connection_pool.connect_duration_ms (histogram): time to open one new connection, including DNS, TCP, and TLS setup. Reused connections record nothing, so the count is the number of connections the pool had to open. A run that opens connections throughout, rather than only at its start, is losing them, for example to keep-alive timeouts. nemo.gym.http.connection_pool.connect_outcome is ok or failed.

gym.http.connection_pool.connect_total (observable counter): all connection attempts (connect() calls), including attempts that later fail or are abandoned. Compare its value with the queue-duration histogram count to calculate the queued fraction.

Rollout outcomes

gym.rollout.completed_total (counter): one per rollout attempt the collection driver finished handling, recorded (for the collection driver) after the recovery store accepts the saved outcome. An attempt is one dispatch of one row in one collection run; a retry on a later resume run is another attempt, and an HTTP-level re-send of /run inside one dispatch is not. Two producers record it. The collection driver is run_from_config (gym eval run with serving): --no-serve initialises no telemetry, and run_examples callers, reverification and the GDPVal multistage driver do not call the recorder. TaskRunner.run_task, the in-process API, also records each rollout outcome it returns, without a recovery-store write: for it failed means the outcome has a failure (labeled with its kind only), scored a reward and no failure, and unscored neither. gym.rollout.rewarded_total counts only the finite rewards of scored rollouts, masked ones included.

nemo.gym.rollout.outcome is scored (main output, including masked, forced-zero and supported unscored completions), failed (saved failures, including kill-shaped) or omitted (saved intentional omissions, including terminal skips and bare no-persist replies). The historical scored label does not guarantee a measured reward. Rows drained by the dispatch budget never ran and are not counted. Reused cached results do not increment the counter. Request errors count only when failure routing saves them: an unavailable server uses class agent_request_failed, while an agent that ran and failed uses agent_run_error (including HTTP 500). Otherwise the request error aborts collection and is not counted. Coverage comes from the output files.

Attributes include nemo.gym.rollout.dispatch.name (the environment server, or the agent when rows name no environment server) and, on attempts that carry a failure class, nemo.gym.failure_class, nemo.gym.failure_kind (registered or <server>:<kind> namespaced, any other value recorded as unregistered) and nemo.gym.failure_type (exception class name). Labels come from the dispatch’s original routing fields, before persistence maps kinds to sidecar classes. Older legacy producers retain their routing class and exception type; these two attributes remain producer-controlled, as on main, and must use stable category names. A missing producer kind adds no kind label, even when the saved record uses a class as its fallback. Native failures and the legacy Environment Server adapter route by the shared kind, bucketed when it also supplies the class. The adapter previously used environment_server_failed without a kind label; it now labels failures like native episodes. Free-text kinds and reasons are never promoted to class labels, but arbitrary strings supplied directly as legacy class/type fields are retained.

Rollout rewards

gym.rollout.rewarded_total (counter): one per scored attempt that carries a numeric reward, recorded with the outcome counter; a result without one is unscored, not a zero. Rewards are environment-defined floats: binary, partial credit, or negative penalties. nemo.gym.rollout.reward.range sorts each one into negative, zero, partial (strictly between 0 and 1), full (exactly 1) or above_one. Also carries nemo.gym.rollout.dispatch.name and nemo.gym.rollout.masked (the result’s mask_sample), so masked rewards can be left out as the aggregate metrics leave them out. A live view of the run, not its score: benchmark metrics such as pass@k come from /aggregate_metrics.

Rollout health

The trajectory-level checks of the rollout health report, evaluated as each rollout finishes. gym.rollout.health_verdict_total (counter): one per evaluated rollout, by nemo.gym.health.verdict (healthy, unhealthy or unobserved, as in the report) and nemo.gym.rollout.outcome (scored or failed; the post-run report reads scored rollouts only). gym.rollout.health_flagged_total (counter): one per rollout and check with at least one finding, by nemo.gym.health.check (a check id from the health-check registry) and nemo.gym.rollout.outcome, so a check’s count is the number of rollouts it flagged, not the number of findings. Checks in health_check_ignored_checks are not evaluated.

Partial-rollout checkpoints

gym.checkpoint.operation_duration_ms (histogram): wall-clock of one checkpoint operation. nemo.gym.checkpoint.operation is a participant’s control operation (prepare, commit, restore, resume, retire, forget, or release), lease_reopen for one attempt to reopen a participant whose lease expired, or coordinate.<operation> for one call the controller makes to every participant. It also carries nemo.gym.checkpoint.participant_kind (controller for coordination) and outcome (ok or error).

gym.checkpoint.records_total / gym.checkpoint.bytes_total (counters): the records a participant wrote at commit or restored, and the bytes of the records file it wrote or read, by nemo.gym.checkpoint.operation and nemo.gym.checkpoint.participant_kind. A commit counts once its write finishes, and a restore counts once it succeeds, so a retried call counts nothing again.

gym.checkpoint.control_errors_total (counter): control calls that failed, by nemo.gym.checkpoint.error_code, the code of the error in the reply, such as checkpoint_deadline_exceeded. Each failed call to a participant’s control routes counts once. Data-plane requests refused because of a checkpoint are not counted.

gym.checkpoint.events_total (counter): things that change a checkpoint’s outcome without failing a call, by nemo.gym.checkpoint.event:

lease_expired: a participant’s lease ran out before its controller resumed it, so the participant reopens on its own. Counted once per checkpoint, with the participant kind and phase. Each attempt to reopen is a lease_reopen operation.
prepare_not_ready: a prepare reached its deadline with blockers.
generation_cut: one in-flight model call’s cut, with its disposition.

Module Contents

Functions

Name	Description
`_get_or_create`	-
`_meter`	-
`_record_counter`	-
`_record_histogram`	-
`_record_up_down_counter`	-
`_reset_for_testing`	Drop cached instruments. Test-only.
`_reward_range`	-
`_server_attributes`	-
`record_checkpoint_control_error`	Count one control call that failed, by the error code of its reply.
`record_checkpoint_event`	Count a checkpoint event (`lease_expired`, `prepare_not_ready`, `generation_cut`).
`record_checkpoint_operation`	Record one checkpoint operation’s wall-clock, by operation, participant kind, and outcome.
`record_checkpoint_volume`	Count the records a participant wrote at commit or restored, and the bytes of the records file.
`record_http_connection_pool_connect_duration`	Record the setup time of one new pooled connection.
`record_http_connection_pool_queue_duration`	Record one queued connection acquisition.
`record_rollout_completed`	Count one finished rollout attempt in `gym.rollout.completed_total`.
`record_rollout_health`	Count one health-evaluated rollout and each distinct check that flagged it.
`record_rollout_reward`	Count one scored attempt in `gym.rollout.rewarded_total` by the range its reward falls in.
`record_run_summary`	Record a finished run’s coverage and retry counts as gauges, one per key, under its run id.
`record_sandbox_active`	Add `delta` (`+1` on start, `-1` on stop) to `gym.sandbox.active` for `provider`.
`record_sandbox_create_retry`	Count one retried sandbox-create attempt, by provider.
`record_sandbox_exec_duration`	Record one command’s wall-clock inside an already-provisioned sandbox, by provider.
`record_sandbox_resource_sample`	Export execd snapshots as gauges; timestamped raw observations remain on resource spans.
`record_sandbox_startup`	Record one sandbox’s provisioning wall-clock, by provider.
`record_sandbox_transfer`	Record one file transfer’s wall-clock and, when it succeeded, the bytes it moved.
`record_server_startup_stage`	Record one stage of one server’s own startup, from the process that ran it.
`register_http_connection_pool_connect_counter`	Export cumulative connection-attempt counts without an OTel call on each connect.

Data

CHECKPOINT_ATTRIBUTE_PREFIX

CHECKPOINT_BYTES_INSTRUMENT

CHECKPOINT_CONTROL_ERRORS_INSTRUMENT

CHECKPOINT_DURATION_BOUNDARIES_MS

CHECKPOINT_ERROR_CODE_ATTRIBUTE

CHECKPOINT_EVENTS_INSTRUMENT

CHECKPOINT_KIND_ATTRIBUTE

CHECKPOINT_OPERATION_ATTRIBUTE

CHECKPOINT_OPERATION_INSTRUMENT

CHECKPOINT_RECORDS_INSTRUMENT

DISPATCH_NAME_ATTRIBUTE

FAILURE_CLASS_ATTRIBUTE

FAILURE_KIND_ATTRIBUTE

FAILURE_TYPE_ATTRIBUTE

HEALTH_CHECKS_ATTRIBUTE

HEALTH_CHECK_ATTRIBUTE

HEALTH_VERDICT_ATTRIBUTE

HTTP_CONNECTION_POOL_CONNECT_DURATION_INSTRUMENT

HTTP_CONNECTION_POOL_CONNECT_INSTRUMENT

HTTP_CONNECTION_POOL_CONNECT_OUTCOME_ATTRIBUTE

HTTP_CONNECTION_POOL_QUEUE_CONSTRAINT_ATTRIBUTE

HTTP_CONNECTION_POOL_QUEUE_DURATION_BOUNDARIES_MS

HTTP_CONNECTION_POOL_QUEUE_DURATION_INSTRUMENT

HTTP_CONNECTION_POOL_QUEUE_OUTCOME_ATTRIBUTE

HTTP_DESTINATION_SERVER_NAME_ATTRIBUTE

REWARD_RANGE_ATTRIBUTE

ROLLOUT_COMPLETED_INSTRUMENT

ROLLOUT_HEALTH_FLAGGED_INSTRUMENT

ROLLOUT_HEALTH_VERDICT_INSTRUMENT

ROLLOUT_MASKED_ATTRIBUTE

ROLLOUT_OUTCOME_ATTRIBUTE

ROLLOUT_REWARDED_INSTRUMENT

RUN_ID_ATTRIBUTE

RUN_INFO_INSTRUMENT

RUN_METRIC_BUCKET_ATTRIBUTE

RUN_METRIC_PREFIX

RUN_METRIC_UNREGISTERED_BUCKET

SANDBOX_ACTIVE_INSTRUMENT

SANDBOX_CREATE_RETRY_INSTRUMENT

SANDBOX_DURATION_BOUNDARIES_MS

SANDBOX_EXEC_INSTRUMENT

SANDBOX_PROVIDER_ATTRIBUTE

SANDBOX_STARTUP_INSTRUMENT

SANDBOX_TRANSFER_BYTES_INSTRUMENT

SANDBOX_TRANSFER_DIRECTION_ATTRIBUTE

SANDBOX_TRANSFER_DURATION_BOUNDARIES_MS

SANDBOX_TRANSFER_DURATION_INSTRUMENT

SANDBOX_TRANSFER_OUTCOME_ATTRIBUTE

SERVER_NAME_ATTRIBUTE

SERVER_STARTUP_BOUNDARIES_MS

SERVER_STARTUP_STAGE_INSTRUMENT

SERVER_TYPE_ATTRIBUTE

STARTUP_STAGE_ATTRIBUTE

SandboxTransferDirection

SandboxTransferOutcome

UNREGISTERED_FAILURE_KIND

WORKER_ATTRIBUTE

_INSTRUMENTS

_INSTRUMENT_LOCK

_RUN_METRIC_EXTRA_BUCKETS

logger

API

nemo_gym.telemetry.gym_metrics._get_or_create(
    meter: typing.Any,
    name: str,
    factory: collections.abc.Callable[[], typing.Any]
) -> typing.Any

nemo_gym.telemetry.gym_metrics._meter() -> typing.Optional[typing.Any]

nemo_gym.telemetry.gym_metrics._record_counter(
    name: str,
    description: str,
    attributes: dict[str, typing.Any],
    amount: int = 1,
    unit: str = '1'
) -> None

nemo_gym.telemetry.gym_metrics._record_histogram(
    name: str,
    unit: str,
    description: str,
    value: float,
    attributes: dict[str, typing.Any],
    boundaries: collections.abc.Sequence[float] | None = None
) -> None

nemo_gym.telemetry.gym_metrics._record_up_down_counter(
    name: str,
    unit: str,
    description: str,
    delta: int,
    attributes: dict[str, typing.Any]
) -> None

nemo_gym.telemetry.gym_metrics._reset_for_testing() -> None

Drop cached instruments. Test-only.

nemo_gym.telemetry.gym_metrics._reward_range(
    reward: float
) -> str

nemo_gym.telemetry.gym_metrics._server_attributes(
    server_name: str,
    server_type: typing.Optional[str]
) -> dict[str, typing.Any]

nemo_gym.telemetry.gym_metrics.record_checkpoint_control_error(
    code: str
) -> None

Count one control call that failed, by the error code of its reply.

nemo_gym.telemetry.gym_metrics.record_checkpoint_event(
    event: str,
    amount: int = 1,
    attributes: str = {}
) -> None

Count a checkpoint event (lease_expired, prepare_not_ready, generation_cut).

nemo_gym.telemetry.gym_metrics.record_checkpoint_operation(
    duration_ms: float,
    operation: str,
    kind: str,
    outcome: str
) -> None

Record one checkpoint operation’s wall-clock, by operation, participant kind, and outcome.

nemo_gym.telemetry.gym_metrics.record_checkpoint_volume(
    operation: str,
    kind: str,
    records: int,
    size_bytes: int
) -> None

Count the records a participant wrote at commit or restored, and the bytes of the records file.

nemo_gym.telemetry.gym_metrics.record_http_connection_pool_connect_duration(
    duration_ms: float,
    outcome: str,
    server_name: str
) -> None

Record the setup time of one new pooled connection.

nemo_gym.telemetry.gym_metrics.record_http_connection_pool_queue_duration(
    duration_ms: float,
    queue_constraint: str,
    queue_outcome: str,
    server_name: str
) -> None

Record one queued connection acquisition.

nemo_gym.telemetry.gym_metrics.record_rollout_completed(
    outcome: str,
    dispatch_name: str,
    failure_class: str | None = None,
    failure_kind: str | None = None,
    failure_type: str | None = None
) -> None

Count one finished rollout attempt in gym.rollout.completed_total.

nemo_gym.telemetry.gym_metrics.record_rollout_health(
    verdict: str,
    outcome: str,
    flagged_checks: collections.abc.Iterable[str]
) -> None

Count one health-evaluated rollout and each distinct check that flagged it.

nemo_gym.telemetry.gym_metrics.record_rollout_reward(
    reward: float,
    dispatch_name: str,
    masked: bool
) -> None

Count one scored attempt in gym.rollout.rewarded_total by the range its reward falls in.

nemo_gym.telemetry.gym_metrics.record_run_summary(
    summary: collections.abc.Mapping[str, float],
    run_id: typing.Optional[str]
) -> None

Record a finished run’s coverage and retry counts as gauges, one per key, under its run id.

coverage/scored becomes gym.run.coverage.scored. coverage/failed/<class> becomes gym.run.coverage.failed with the class as the bucket attribute. A class outside the registered failure kinds is counted under unregistered, so the attribute stays bounded. run_id is left off for a merge of several shards; the job’s nemo.run.id resource attribute identifies it.

nemo_gym.telemetry.gym_metrics.record_sandbox_active(
    delta: int,
    provider: str
) -> None

Add delta (+1 on start, -1 on stop) to gym.sandbox.active for provider.

nemo_gym.telemetry.gym_metrics.record_sandbox_create_retry(
    provider: str
) -> None

Count one retried sandbox-create attempt, by provider.

nemo_gym.telemetry.gym_metrics.record_sandbox_exec_duration(
    duration_ms: float,
    provider: str
) -> None

Record one command’s wall-clock inside an already-provisioned sandbox, by provider.

nemo_gym.telemetry.gym_metrics.record_sandbox_resource_sample(
    values: dict[str, typing.Any],
    attributes: dict[str, typing.Any]
) -> None

Export execd snapshots as gauges; timestamped raw observations remain on resource spans.

nemo_gym.telemetry.gym_metrics.record_sandbox_startup(
    duration_ms: float,
    provider: str
) -> None

Record one sandbox’s provisioning wall-clock, by provider.

nemo_gym.telemetry.gym_metrics.record_sandbox_transfer(
    duration_ms: float,
    provider: str,
    direction: nemo_gym.telemetry.gym_metrics.SandboxTransferDirection,
    outcome: nemo_gym.telemetry.gym_metrics.SandboxTransferOutcome,
    size_bytes: int | None
) -> None

Record one file transfer’s wall-clock and, when it succeeded, the bytes it moved.

nemo_gym.telemetry.gym_metrics.record_server_startup_stage(
    duration_ms: float,
    stage: str,
    server_name: str,
    server_type: typing.Optional[str],
    worker: bool = False
) -> None

Record one stage of one server’s own startup, from the process that ran it.

worker is true for a Uvicorn worker of a multi-worker server. A worker’s stages are timed separately from its main process’s.

nemo_gym.telemetry.gym_metrics.register_http_connection_pool_connect_counter(
    snapshot: collections.abc.Callable[[], dict[str, int]]
) -> None

Export cumulative connection-attempt counts without an OTel call on each connect.

nemo_gym.telemetry.gym_metrics.CHECKPOINT_ATTRIBUTE_PREFIX = 'nemo.gym.checkpoint.'

nemo_gym.telemetry.gym_metrics.CHECKPOINT_BYTES_INSTRUMENT = 'gym.checkpoint.bytes_total'

nemo_gym.telemetry.gym_metrics.CHECKPOINT_CONTROL_ERRORS_INSTRUMENT = 'gym.checkpoint.control_errors_total'

nemo_gym.telemetry.gym_metrics.CHECKPOINT_DURATION_BOUNDARIES_MS: tuple[float, ...] = (1, 5, 10, 50, 100, 500, 1000, 5000, 10000, 30000, 120000)

nemo_gym.telemetry.gym_metrics.CHECKPOINT_ERROR_CODE_ATTRIBUTE = f'{CHECKPOINT_ATTRIBUTE_PREFIX}error_code'

nemo_gym.telemetry.gym_metrics.CHECKPOINT_EVENTS_INSTRUMENT = 'gym.checkpoint.events_total'

nemo_gym.telemetry.gym_metrics.CHECKPOINT_KIND_ATTRIBUTE = f'{CHECKPOINT_ATTRIBUTE_PREFIX}participant_kind'

nemo_gym.telemetry.gym_metrics.CHECKPOINT_OPERATION_ATTRIBUTE = f'{CHECKPOINT_ATTRIBUTE_PREFIX}operation'

nemo_gym.telemetry.gym_metrics.CHECKPOINT_OPERATION_INSTRUMENT = 'gym.checkpoint.operation_duration_ms'

nemo_gym.telemetry.gym_metrics.CHECKPOINT_RECORDS_INSTRUMENT = 'gym.checkpoint.records_total'

nemo_gym.telemetry.gym_metrics.DISPATCH_NAME_ATTRIBUTE = 'nemo.gym.rollout.dispatch.name'

nemo_gym.telemetry.gym_metrics.FAILURE_CLASS_ATTRIBUTE = 'nemo.gym.failure_class'

nemo_gym.telemetry.gym_metrics.FAILURE_KIND_ATTRIBUTE = 'nemo.gym.failure_kind'

nemo_gym.telemetry.gym_metrics.FAILURE_TYPE_ATTRIBUTE = 'nemo.gym.failure_type'

nemo_gym.telemetry.gym_metrics.HEALTH_CHECKS_ATTRIBUTE = 'nemo.gym.health.checks'

nemo_gym.telemetry.gym_metrics.HEALTH_CHECK_ATTRIBUTE = 'nemo.gym.health.check'

nemo_gym.telemetry.gym_metrics.HEALTH_VERDICT_ATTRIBUTE = 'nemo.gym.health.verdict'

nemo_gym.telemetry.gym_metrics.HTTP_CONNECTION_POOL_CONNECT_DURATION_INSTRUMENT = 'gym.http.connection_pool.connect_duration_ms'

nemo_gym.telemetry.gym_metrics.HTTP_CONNECTION_POOL_CONNECT_INSTRUMENT = 'gym.http.connection_pool.connect_total'

nemo_gym.telemetry.gym_metrics.HTTP_CONNECTION_POOL_CONNECT_OUTCOME_ATTRIBUTE = 'nemo.gym.http.connection_pool.connect_outcome'

nemo_gym.telemetry.gym_metrics.HTTP_CONNECTION_POOL_QUEUE_CONSTRAINT_ATTRIBUTE = 'nemo.gym.http.connection_pool.queue_constraint'

nemo_gym.telemetry.gym_metrics.HTTP_CONNECTION_POOL_QUEUE_DURATION_BOUNDARIES_MS: tuple[float, ...] = (0.01, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10, 25, 50, 100, 250, 500, 1000, 5000, 3...

nemo_gym.telemetry.gym_metrics.HTTP_CONNECTION_POOL_QUEUE_DURATION_INSTRUMENT = 'gym.http.connection_pool.queue_duration_ms'

nemo_gym.telemetry.gym_metrics.HTTP_CONNECTION_POOL_QUEUE_OUTCOME_ATTRIBUTE = 'nemo.gym.http.connection_pool.queue_outcome'

nemo_gym.telemetry.gym_metrics.HTTP_DESTINATION_SERVER_NAME_ATTRIBUTE = 'nemo.gym.http.destination.server.name'

nemo_gym.telemetry.gym_metrics.REWARD_RANGE_ATTRIBUTE = 'nemo.gym.rollout.reward.range'

nemo_gym.telemetry.gym_metrics.ROLLOUT_COMPLETED_INSTRUMENT = 'gym.rollout.completed_total'

nemo_gym.telemetry.gym_metrics.ROLLOUT_HEALTH_FLAGGED_INSTRUMENT = 'gym.rollout.health_flagged_total'

nemo_gym.telemetry.gym_metrics.ROLLOUT_HEALTH_VERDICT_INSTRUMENT = 'gym.rollout.health_verdict_total'

nemo_gym.telemetry.gym_metrics.ROLLOUT_MASKED_ATTRIBUTE = 'nemo.gym.rollout.masked'

nemo_gym.telemetry.gym_metrics.ROLLOUT_OUTCOME_ATTRIBUTE = 'nemo.gym.rollout.outcome'

nemo_gym.telemetry.gym_metrics.ROLLOUT_REWARDED_INSTRUMENT = 'gym.rollout.rewarded_total'

nemo_gym.telemetry.gym_metrics.RUN_ID_ATTRIBUTE = 'nemo.gym.run.id'

nemo_gym.telemetry.gym_metrics.RUN_INFO_INSTRUMENT = 'gym.run.info'

nemo_gym.telemetry.gym_metrics.RUN_METRIC_BUCKET_ATTRIBUTE = 'nemo.gym.run.bucket'

nemo_gym.telemetry.gym_metrics.RUN_METRIC_PREFIX = 'gym.run.'

nemo_gym.telemetry.gym_metrics.RUN_METRIC_UNREGISTERED_BUCKET = 'unregistered'

nemo_gym.telemetry.gym_metrics.SANDBOX_ACTIVE_INSTRUMENT = 'gym.sandbox.active'

nemo_gym.telemetry.gym_metrics.SANDBOX_CREATE_RETRY_INSTRUMENT = 'gym.sandbox.create_retry_total'

nemo_gym.telemetry.gym_metrics.SANDBOX_DURATION_BOUNDARIES_MS: tuple[float, ...] = (250, 500, 1000, 2000, 5000, 10000, 30000, 60000, 120000, 300000, 600000, 180000...

nemo_gym.telemetry.gym_metrics.SANDBOX_EXEC_INSTRUMENT = 'gym.sandbox.exec_duration_ms'

nemo_gym.telemetry.gym_metrics.SANDBOX_PROVIDER_ATTRIBUTE = 'nemo.gym.sandbox.provider'

nemo_gym.telemetry.gym_metrics.SANDBOX_STARTUP_INSTRUMENT = 'gym.sandbox.startup_duration_ms'

nemo_gym.telemetry.gym_metrics.SANDBOX_TRANSFER_BYTES_INSTRUMENT = 'gym.sandbox.transfer_bytes_total'

nemo_gym.telemetry.gym_metrics.SANDBOX_TRANSFER_DIRECTION_ATTRIBUTE = 'nemo.gym.sandbox.transfer.direction'

nemo_gym.telemetry.gym_metrics.SANDBOX_TRANSFER_DURATION_BOUNDARIES_MS: tuple[float, ...] = (10, 25, 50, 100, 250, 500, 1000, 2500, 5000, 10000, 30000, 60000, 120000, 30000...

nemo_gym.telemetry.gym_metrics.SANDBOX_TRANSFER_DURATION_INSTRUMENT = 'gym.sandbox.transfer_duration_ms'

nemo_gym.telemetry.gym_metrics.SANDBOX_TRANSFER_OUTCOME_ATTRIBUTE = 'nemo.gym.sandbox.transfer.outcome'

nemo_gym.telemetry.gym_metrics.SERVER_NAME_ATTRIBUTE = 'nemo.gym.server.name'

nemo_gym.telemetry.gym_metrics.SERVER_STARTUP_BOUNDARIES_MS: tuple[float, ...] = (100, 250, 500, 1000, 2500, 5000, 10000, 30000, 60000, 120000, 300000, 600000, 1...

nemo_gym.telemetry.gym_metrics.SERVER_STARTUP_STAGE_INSTRUMENT = 'gym.server.startup_stage_duration_ms'

nemo_gym.telemetry.gym_metrics.SERVER_TYPE_ATTRIBUTE = 'nemo.gym.server.type'

nemo_gym.telemetry.gym_metrics.STARTUP_STAGE_ATTRIBUTE = 'nemo.gym.startup.stage'

nemo_gym.telemetry.gym_metrics.SandboxTransferDirection = Literal['upload', 'download']

nemo_gym.telemetry.gym_metrics.SandboxTransferOutcome = Literal['ok', 'error', 'cancelled']

nemo_gym.telemetry.gym_metrics.UNREGISTERED_FAILURE_KIND = 'unregistered'

nemo_gym.telemetry.gym_metrics.WORKER_ATTRIBUTE = 'nemo.gym.server.worker'

nemo_gym.telemetry.gym_metrics._INSTRUMENTS: dict[int, dict[str, Any]] = {}

nemo_gym.telemetry.gym_metrics._INSTRUMENT_LOCK = threading.Lock()

nemo_gym.telemetry.gym_metrics._RUN_METRIC_EXTRA_BUCKETS = frozenset({'skipped', 'unknown'})

nemo_gym.telemetry.gym_metrics.logger = logging.getLogger(__name__)