> This page is for version 0.7.0.
> For other versions, use one of these documentation indexes:
> - Main (default): https://docs.nvidia.com/nemo/gym/main/llms.txt
> - 0.7.0: https://docs.nvidia.com/nemo/gym/v0.7.0/llms.txt
> - 0.6.0: https://docs.nvidia.com/nemo/gym/v0.6.0/llms.txt
> - 0.5.1: https://docs.nvidia.com/nemo/gym/v0.5.1/llms.txt
> - 0.5.0: https://docs.nvidia.com/nemo/gym/v0.5.0/llms.txt
> - 0.4.0: https://docs.nvidia.com/nemo/gym/v0.4.0/llms.txt
> - 0.3.0: https://docs.nvidia.com/nemo/gym/v0.3.0/llms.txt
> - 0.2.1: https://docs.nvidia.com/nemo/gym/v0.2.1/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/gym/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/gym/_mcp/server.

# nemo_gym.telemetry.gym_metrics

Attributed OTel instruments that `nemo.lens.instruments.gym.record_gym_metrics` cannot express.

`nemo_gym.telemetry.metrics` forwards to the five fixed, undimensioned instruments nemo-lens
declares. Anything that needs an attribute (a provider, a site, a class) is created here,
directly on the lens meter, and cached per meter so a re-initialised telemetry handle gets
fresh instruments. Every recorder is a no-op unless telemetry is initialised and exporting,
and never raises into its caller. Call sites use the relevant span-group or metrics-export gate
so disabled telemetry stays off the hot path.

**Sandbox lifecycle**

`gym.sandbox.active` (up-down counter): sandboxes this process currently holds, `+1` when a
provider hands back a handle, `-1` when Gym releases it. Summed by the backend across the
processes that hold sandboxes; a gauge would show whichever process wrote last.

`gym.sandbox.startup_duration_ms` / `gym.sandbox.exec_duration_ms` (histograms): wall-clock
of one provisioning and one command. Explicit bucket boundaries up to thirty minutes: the SDK
default stops at ten seconds and a sandbox start routinely takes a minute.

`gym.sandbox.create_retry_total` (counter): one per create attempt a provider retried.
Retrying is provider-internal, so each provider that retries records it from its own loop.

All four carry `nemo.gym.sandbox.provider`.

**Server startup**

`gym.server.startup_stage_duration_ms` (histogram): one stage of one server's own startup, recorded by the process that ran it.
`nemo.gym.startup.stage` names the stage, and `nemo.gym.server.worker` marks a Uvicorn worker of a multi-worker server.
Each point also carries `nemo.gym.server.name` and `nemo.gym.server.type`.
Stages are contiguous, so one process's stages add up to the time from its start until it accepts connections.
A server process starts when the supervisor spawns it, and a worker starts when its main process hands off to Uvicorn.
The same stages are spans under the `startup` span group.
There is no supervisor-side total, because the supervisor only learns that a server is ready from a periodic health poll.
That total would be these stages plus polling delay.

**HTTP connection pool**

`gym.http.connection_pool.queue_duration_ms` (histogram): connection-acquisition wait
for one queued connection acquisition, combining repeated waiter wakeups within it.
Queued redirect hops, aiohttp reconnects, and Gym retries each record separate samples.
Connection attempts that acquire a slot immediately do not record a queue-duration histogram sample.
Bounded attributes identify the binding connector limit, queue outcome, and destination server.

`gym.http.connection_pool.connect_duration_ms` (histogram): time to open one new connection, including DNS, TCP, and TLS setup.
Reused connections record nothing, so the count is the number of connections the pool had to open.
A run that opens connections throughout, rather than only at its start, is losing them, for example to keep-alive timeouts.
`nemo.gym.http.connection_pool.connect_outcome` is `ok` or `failed`.

`gym.http.connection_pool.connect_total` (observable counter): all connection attempts
(`connect()` calls), including attempts that later fail or are abandoned.
Compare its value with the queue-duration histogram count to calculate the queued fraction.

**Rollout outcomes**

`gym.rollout.completed_total` (counter): one per rollout attempt the collection driver finished
handling, recorded after it wrote the attempt or decided to omit it. An attempt is one dispatch of
one row in one collection run; a retry on a later resume run is another attempt, and an HTTP-level
re-send of `/run` inside one dispatch is not. Only `run_from_config` (`gym eval run` with
serving) records it: `--no-serve` initialises no telemetry, and `run_examples` callers,
reverification and the GDPVal multistage driver do not call the recorder.

`nemo.gym.rollout.outcome` is `scored` (main output, masked and forced-zero results included),
`failed` (failures sidecar) or `omitted` (no-persist, e.g. kill-shaped). Rows drained by the
dispatch budget never ran and are not counted, unlike the progress metrics, whose `dropped`
groups `failed` with an `omitted` that includes drained rows. A failed `/run` is a `failed`
attempt with class `agent_request_failed` only with `route_failures_to_sidecar` on; otherwise
it aborts the run and is not counted. Coverage comes from the output files, not this counter.

Attributes are bounded: `nemo.gym.rollout.dispatch.name` (the environment server, or the agent
when rows name no environment server) and, on attempts that carry a failure class,
`nemo.gym.failure_class`, `nemo.gym.failure_kind` (registered or `<server>:<kind>`
namespaced, any other value recorded as `unregistered`) and `nemo.gym.failure_type` (exception
class name). Free-text reasons are never attributes.

**Rollout rewards**

`gym.rollout.rewarded_total` (counter): one per `scored` attempt that carries a numeric reward,
recorded with the outcome counter; a result without one is unscored, not a zero. Rewards are
environment-defined floats: binary, partial credit, or negative penalties.
`nemo.gym.rollout.reward.range` sorts each one into `negative`, `zero`, `partial` (strictly
between 0 and 1), `full` (exactly 1) or `above_one`. Also carries `nemo.gym.rollout.dispatch.name`
and `nemo.gym.rollout.masked` (the result's `mask_sample`), so masked rewards can be left out as
the aggregate metrics leave them out. A live view of the run, not its score: benchmark metrics such
as pass\@k come from `/aggregate_metrics`.

**Rollout health**

The trajectory-level checks of the rollout health report, evaluated as each rollout finishes.
`gym.rollout.health_verdict_total` (counter): one per evaluated rollout, by `nemo.gym.health.verdict`
(`healthy`, `unhealthy` or `unobserved`, as in the report) and `nemo.gym.rollout.outcome`
(`scored` or `failed`; the post-run report reads scored rollouts only).
`gym.rollout.health_flagged_total` (counter): one per rollout *and* check with at least one finding,
by `nemo.gym.health.check` (a check id from the health-check registry) and `nemo.gym.rollout.outcome`, so a
check's count is the number of rollouts it flagged, not the number of findings. Checks in `health_check_ignored_checks` are not evaluated.

## Module Contents

### Functions

| Name                                                                                                                             | Description                                                                                |
| -------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------ |
| [`_get_or_create`](#nemo_gym-telemetry-gym_metrics-_get_or_create)                                                               | -                                                                                          |
| [`_meter`](#nemo_gym-telemetry-gym_metrics-_meter)                                                                               | -                                                                                          |
| [`_record_counter`](#nemo_gym-telemetry-gym_metrics-_record_counter)                                                             | -                                                                                          |
| [`_record_histogram`](#nemo_gym-telemetry-gym_metrics-_record_histogram)                                                         | -                                                                                          |
| [`_record_up_down_counter`](#nemo_gym-telemetry-gym_metrics-_record_up_down_counter)                                             | -                                                                                          |
| [`_reset_for_testing`](#nemo_gym-telemetry-gym_metrics-_reset_for_testing)                                                       | Drop cached instruments. Test-only.                                                        |
| [`_reward_range`](#nemo_gym-telemetry-gym_metrics-_reward_range)                                                                 | -                                                                                          |
| [`_server_attributes`](#nemo_gym-telemetry-gym_metrics-_server_attributes)                                                       | -                                                                                          |
| [`record_http_connection_pool_connect_duration`](#nemo_gym-telemetry-gym_metrics-record_http_connection_pool_connect_duration)   | Record the setup time of one new pooled connection.                                        |
| [`record_http_connection_pool_queue_duration`](#nemo_gym-telemetry-gym_metrics-record_http_connection_pool_queue_duration)       | Record one queued connection acquisition.                                                  |
| [`record_rollout_completed`](#nemo_gym-telemetry-gym_metrics-record_rollout_completed)                                           | Count one finished rollout attempt in `gym.rollout.completed_total`.                       |
| [`record_rollout_health`](#nemo_gym-telemetry-gym_metrics-record_rollout_health)                                                 | Count one health-evaluated rollout and each distinct check that flagged it.                |
| [`record_rollout_reward`](#nemo_gym-telemetry-gym_metrics-record_rollout_reward)                                                 | Count one scored attempt in `gym.rollout.rewarded_total` by the range its reward falls in. |
| [`record_sandbox_active`](#nemo_gym-telemetry-gym_metrics-record_sandbox_active)                                                 | Add `delta` (`+1` on start, `-1` on stop) to `gym.sandbox.active` for `provider`.          |
| [`record_sandbox_create_retry`](#nemo_gym-telemetry-gym_metrics-record_sandbox_create_retry)                                     | Count one retried sandbox-create attempt, by provider.                                     |
| [`record_sandbox_exec_duration`](#nemo_gym-telemetry-gym_metrics-record_sandbox_exec_duration)                                   | Record one command's wall-clock inside an already-provisioned sandbox, by provider.        |
| [`record_sandbox_startup`](#nemo_gym-telemetry-gym_metrics-record_sandbox_startup)                                               | Record one sandbox's provisioning wall-clock, by provider.                                 |
| [`record_server_startup_stage`](#nemo_gym-telemetry-gym_metrics-record_server_startup_stage)                                     | Record one stage of one server's own startup, from the process that ran it.                |
| [`register_http_connection_pool_connect_counter`](#nemo_gym-telemetry-gym_metrics-register_http_connection_pool_connect_counter) | Export cumulative connection-attempt counts without an OTel call on each connect.          |

### Data

[`DISPATCH_NAME_ATTRIBUTE`](#nemo_gym-telemetry-gym_metrics-DISPATCH_NAME_ATTRIBUTE)

[`FAILURE_CLASS_ATTRIBUTE`](#nemo_gym-telemetry-gym_metrics-FAILURE_CLASS_ATTRIBUTE)

[`FAILURE_KIND_ATTRIBUTE`](#nemo_gym-telemetry-gym_metrics-FAILURE_KIND_ATTRIBUTE)

[`FAILURE_TYPE_ATTRIBUTE`](#nemo_gym-telemetry-gym_metrics-FAILURE_TYPE_ATTRIBUTE)

[`HEALTH_CHECK_ATTRIBUTE`](#nemo_gym-telemetry-gym_metrics-HEALTH_CHECK_ATTRIBUTE)

[`HEALTH_VERDICT_ATTRIBUTE`](#nemo_gym-telemetry-gym_metrics-HEALTH_VERDICT_ATTRIBUTE)

[`HTTP_CONNECTION_POOL_CONNECT_DURATION_INSTRUMENT`](#nemo_gym-telemetry-gym_metrics-HTTP_CONNECTION_POOL_CONNECT_DURATION_INSTRUMENT)

[`HTTP_CONNECTION_POOL_CONNECT_INSTRUMENT`](#nemo_gym-telemetry-gym_metrics-HTTP_CONNECTION_POOL_CONNECT_INSTRUMENT)

[`HTTP_CONNECTION_POOL_CONNECT_OUTCOME_ATTRIBUTE`](#nemo_gym-telemetry-gym_metrics-HTTP_CONNECTION_POOL_CONNECT_OUTCOME_ATTRIBUTE)

[`HTTP_CONNECTION_POOL_QUEUE_CONSTRAINT_ATTRIBUTE`](#nemo_gym-telemetry-gym_metrics-HTTP_CONNECTION_POOL_QUEUE_CONSTRAINT_ATTRIBUTE)

[`HTTP_CONNECTION_POOL_QUEUE_DURATION_BOUNDARIES_MS`](#nemo_gym-telemetry-gym_metrics-HTTP_CONNECTION_POOL_QUEUE_DURATION_BOUNDARIES_MS)

[`HTTP_CONNECTION_POOL_QUEUE_DURATION_INSTRUMENT`](#nemo_gym-telemetry-gym_metrics-HTTP_CONNECTION_POOL_QUEUE_DURATION_INSTRUMENT)

[`HTTP_CONNECTION_POOL_QUEUE_OUTCOME_ATTRIBUTE`](#nemo_gym-telemetry-gym_metrics-HTTP_CONNECTION_POOL_QUEUE_OUTCOME_ATTRIBUTE)

[`HTTP_DESTINATION_SERVER_NAME_ATTRIBUTE`](#nemo_gym-telemetry-gym_metrics-HTTP_DESTINATION_SERVER_NAME_ATTRIBUTE)

[`REWARD_RANGE_ATTRIBUTE`](#nemo_gym-telemetry-gym_metrics-REWARD_RANGE_ATTRIBUTE)

[`ROLLOUT_COMPLETED_INSTRUMENT`](#nemo_gym-telemetry-gym_metrics-ROLLOUT_COMPLETED_INSTRUMENT)

[`ROLLOUT_HEALTH_FLAGGED_INSTRUMENT`](#nemo_gym-telemetry-gym_metrics-ROLLOUT_HEALTH_FLAGGED_INSTRUMENT)

[`ROLLOUT_HEALTH_VERDICT_INSTRUMENT`](#nemo_gym-telemetry-gym_metrics-ROLLOUT_HEALTH_VERDICT_INSTRUMENT)

[`ROLLOUT_MASKED_ATTRIBUTE`](#nemo_gym-telemetry-gym_metrics-ROLLOUT_MASKED_ATTRIBUTE)

[`ROLLOUT_OUTCOME_ATTRIBUTE`](#nemo_gym-telemetry-gym_metrics-ROLLOUT_OUTCOME_ATTRIBUTE)

[`ROLLOUT_REWARDED_INSTRUMENT`](#nemo_gym-telemetry-gym_metrics-ROLLOUT_REWARDED_INSTRUMENT)

[`SANDBOX_ACTIVE_INSTRUMENT`](#nemo_gym-telemetry-gym_metrics-SANDBOX_ACTIVE_INSTRUMENT)

[`SANDBOX_CREATE_RETRY_INSTRUMENT`](#nemo_gym-telemetry-gym_metrics-SANDBOX_CREATE_RETRY_INSTRUMENT)

[`SANDBOX_DURATION_BOUNDARIES_MS`](#nemo_gym-telemetry-gym_metrics-SANDBOX_DURATION_BOUNDARIES_MS)

[`SANDBOX_EXEC_INSTRUMENT`](#nemo_gym-telemetry-gym_metrics-SANDBOX_EXEC_INSTRUMENT)

[`SANDBOX_PROVIDER_ATTRIBUTE`](#nemo_gym-telemetry-gym_metrics-SANDBOX_PROVIDER_ATTRIBUTE)

[`SANDBOX_STARTUP_INSTRUMENT`](#nemo_gym-telemetry-gym_metrics-SANDBOX_STARTUP_INSTRUMENT)

[`SERVER_NAME_ATTRIBUTE`](#nemo_gym-telemetry-gym_metrics-SERVER_NAME_ATTRIBUTE)

[`SERVER_STARTUP_BOUNDARIES_MS`](#nemo_gym-telemetry-gym_metrics-SERVER_STARTUP_BOUNDARIES_MS)

[`SERVER_STARTUP_STAGE_INSTRUMENT`](#nemo_gym-telemetry-gym_metrics-SERVER_STARTUP_STAGE_INSTRUMENT)

[`SERVER_TYPE_ATTRIBUTE`](#nemo_gym-telemetry-gym_metrics-SERVER_TYPE_ATTRIBUTE)

[`STARTUP_STAGE_ATTRIBUTE`](#nemo_gym-telemetry-gym_metrics-STARTUP_STAGE_ATTRIBUTE)

[`UNREGISTERED_FAILURE_KIND`](#nemo_gym-telemetry-gym_metrics-UNREGISTERED_FAILURE_KIND)

[`WORKER_ATTRIBUTE`](#nemo_gym-telemetry-gym_metrics-WORKER_ATTRIBUTE)

[`_INSTRUMENTS`](#nemo_gym-telemetry-gym_metrics-_INSTRUMENTS)

[`_INSTRUMENT_LOCK`](#nemo_gym-telemetry-gym_metrics-_INSTRUMENT_LOCK)

[`logger`](#nemo_gym-telemetry-gym_metrics-logger)

### API

```python
nemo_gym.telemetry.gym_metrics._get_or_create(
    meter: typing.Any,
    name: str,
    factory: collections.abc.Callable[[], typing.Any]
) -> typing.Any
```

```python
nemo_gym.telemetry.gym_metrics._meter() -> typing.Optional[typing.Any]
```

```python
nemo_gym.telemetry.gym_metrics._record_counter(
    name: str,
    description: str,
    attributes: dict[str, typing.Any],
    amount: int = 1,
    unit: str = '1'
) -> None
```

```python
nemo_gym.telemetry.gym_metrics._record_histogram(
    name: str,
    unit: str,
    description: str,
    value: float,
    attributes: dict[str, typing.Any],
    boundaries: collections.abc.Sequence[float] | None = None
) -> None
```

```python
nemo_gym.telemetry.gym_metrics._record_up_down_counter(
    name: str,
    unit: str,
    description: str,
    delta: int,
    attributes: dict[str, typing.Any]
) -> None
```

```python
nemo_gym.telemetry.gym_metrics._reset_for_testing() -> None
```

Drop cached instruments. Test-only.

```python
nemo_gym.telemetry.gym_metrics._reward_range(
    reward: float
) -> str
```

```python
nemo_gym.telemetry.gym_metrics._server_attributes(
    server_name: str,
    server_type: typing.Optional[str]
) -> dict[str, typing.Any]
```

```python
nemo_gym.telemetry.gym_metrics.record_http_connection_pool_connect_duration(
    duration_ms: float,
    outcome: str,
    server_name: str
) -> None
```

Record the setup time of one new pooled connection.

```python
nemo_gym.telemetry.gym_metrics.record_http_connection_pool_queue_duration(
    duration_ms: float,
    queue_constraint: str,
    queue_outcome: str,
    server_name: str
) -> None
```

Record one queued connection acquisition.

```python
nemo_gym.telemetry.gym_metrics.record_rollout_completed(
    outcome: str,
    dispatch_name: str,
    failure_class: str | None = None,
    failure_kind: str | None = None,
    failure_type: str | None = None
) -> None
```

Count one finished rollout attempt in `gym.rollout.completed_total`.

```python
nemo_gym.telemetry.gym_metrics.record_rollout_health(
    verdict: str,
    outcome: str,
    flagged_checks: collections.abc.Iterable[str]
) -> None
```

Count one health-evaluated rollout and each distinct check that flagged it.

```python
nemo_gym.telemetry.gym_metrics.record_rollout_reward(
    reward: float,
    dispatch_name: str,
    masked: bool
) -> None
```

Count one scored attempt in `gym.rollout.rewarded_total` by the range its reward falls in.

```python
nemo_gym.telemetry.gym_metrics.record_sandbox_active(
    delta: int,
    provider: str
) -> None
```

Add `delta` (`+1` on start, `-1` on stop) to `gym.sandbox.active` for `provider`.

```python
nemo_gym.telemetry.gym_metrics.record_sandbox_create_retry(
    provider: str
) -> None
```

Count one retried sandbox-create attempt, by provider.

```python
nemo_gym.telemetry.gym_metrics.record_sandbox_exec_duration(
    duration_ms: float,
    provider: str
) -> None
```

Record one command's wall-clock inside an already-provisioned sandbox, by provider.

```python
nemo_gym.telemetry.gym_metrics.record_sandbox_startup(
    duration_ms: float,
    provider: str
) -> None
```

Record one sandbox's provisioning wall-clock, by provider.

```python
nemo_gym.telemetry.gym_metrics.record_server_startup_stage(
    duration_ms: float,
    stage: str,
    server_name: str,
    server_type: typing.Optional[str],
    worker: bool = False
) -> None
```

Record one stage of one server's own startup, from the process that ran it.

`worker` is true for a Uvicorn worker of a multi-worker server.
A worker's stages are timed separately from its main process's.

```python
nemo_gym.telemetry.gym_metrics.register_http_connection_pool_connect_counter(
    snapshot: collections.abc.Callable[[], dict[str, int]]
) -> None
```

Export cumulative connection-attempt counts without an OTel call on each connect.

```python
nemo_gym.telemetry.gym_metrics.DISPATCH_NAME_ATTRIBUTE = 'nemo.gym.rollout.dispatch.name'
```

```python
nemo_gym.telemetry.gym_metrics.FAILURE_CLASS_ATTRIBUTE = 'nemo.gym.failure_class'
```

```python
nemo_gym.telemetry.gym_metrics.FAILURE_KIND_ATTRIBUTE = 'nemo.gym.failure_kind'
```

```python
nemo_gym.telemetry.gym_metrics.FAILURE_TYPE_ATTRIBUTE = 'nemo.gym.failure_type'
```

```python
nemo_gym.telemetry.gym_metrics.HEALTH_CHECK_ATTRIBUTE = 'nemo.gym.health.check'
```

```python
nemo_gym.telemetry.gym_metrics.HEALTH_VERDICT_ATTRIBUTE = 'nemo.gym.health.verdict'
```

```python
nemo_gym.telemetry.gym_metrics.HTTP_CONNECTION_POOL_CONNECT_DURATION_INSTRUMENT = 'gym.http.connection_pool.connect_duration_ms'
```

```python
nemo_gym.telemetry.gym_metrics.HTTP_CONNECTION_POOL_CONNECT_INSTRUMENT = 'gym.http.connection_pool.connect_total'
```

```python
nemo_gym.telemetry.gym_metrics.HTTP_CONNECTION_POOL_CONNECT_OUTCOME_ATTRIBUTE = 'nemo.gym.http.connection_pool.connect_outcome'
```

```python
nemo_gym.telemetry.gym_metrics.HTTP_CONNECTION_POOL_QUEUE_CONSTRAINT_ATTRIBUTE = 'nemo.gym.http.connection_pool.queue_constraint'
```

```python
nemo_gym.telemetry.gym_metrics.HTTP_CONNECTION_POOL_QUEUE_DURATION_BOUNDARIES_MS: tuple[float, ...] = (0.01, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10, 25, 50, 100, 250, 500, 1000, 5000, 3...
```

```python
nemo_gym.telemetry.gym_metrics.HTTP_CONNECTION_POOL_QUEUE_DURATION_INSTRUMENT = 'gym.http.connection_pool.queue_duration_ms'
```

```python
nemo_gym.telemetry.gym_metrics.HTTP_CONNECTION_POOL_QUEUE_OUTCOME_ATTRIBUTE = 'nemo.gym.http.connection_pool.queue_outcome'
```

```python
nemo_gym.telemetry.gym_metrics.HTTP_DESTINATION_SERVER_NAME_ATTRIBUTE = 'nemo.gym.http.destination.server.name'
```

```python
nemo_gym.telemetry.gym_metrics.REWARD_RANGE_ATTRIBUTE = 'nemo.gym.rollout.reward.range'
```

```python
nemo_gym.telemetry.gym_metrics.ROLLOUT_COMPLETED_INSTRUMENT = 'gym.rollout.completed_total'
```

```python
nemo_gym.telemetry.gym_metrics.ROLLOUT_HEALTH_FLAGGED_INSTRUMENT = 'gym.rollout.health_flagged_total'
```

```python
nemo_gym.telemetry.gym_metrics.ROLLOUT_HEALTH_VERDICT_INSTRUMENT = 'gym.rollout.health_verdict_total'
```

```python
nemo_gym.telemetry.gym_metrics.ROLLOUT_MASKED_ATTRIBUTE = 'nemo.gym.rollout.masked'
```

```python
nemo_gym.telemetry.gym_metrics.ROLLOUT_OUTCOME_ATTRIBUTE = 'nemo.gym.rollout.outcome'
```

```python
nemo_gym.telemetry.gym_metrics.ROLLOUT_REWARDED_INSTRUMENT = 'gym.rollout.rewarded_total'
```

```python
nemo_gym.telemetry.gym_metrics.SANDBOX_ACTIVE_INSTRUMENT = 'gym.sandbox.active'
```

```python
nemo_gym.telemetry.gym_metrics.SANDBOX_CREATE_RETRY_INSTRUMENT = 'gym.sandbox.create_retry_total'
```

```python
nemo_gym.telemetry.gym_metrics.SANDBOX_DURATION_BOUNDARIES_MS: tuple[float, ...] = (250, 500, 1000, 2000, 5000, 10000, 30000, 60000, 120000, 300000, 600000, 180000...
```

```python
nemo_gym.telemetry.gym_metrics.SANDBOX_EXEC_INSTRUMENT = 'gym.sandbox.exec_duration_ms'
```

```python
nemo_gym.telemetry.gym_metrics.SANDBOX_PROVIDER_ATTRIBUTE = 'nemo.gym.sandbox.provider'
```

```python
nemo_gym.telemetry.gym_metrics.SANDBOX_STARTUP_INSTRUMENT = 'gym.sandbox.startup_duration_ms'
```

```python
nemo_gym.telemetry.gym_metrics.SERVER_NAME_ATTRIBUTE = 'nemo.gym.server.name'
```

```python
nemo_gym.telemetry.gym_metrics.SERVER_STARTUP_BOUNDARIES_MS: tuple[float, ...] = (100, 250, 500, 1000, 2500, 5000, 10000, 30000, 60000, 120000, 300000, 600000, 1...
```

```python
nemo_gym.telemetry.gym_metrics.SERVER_STARTUP_STAGE_INSTRUMENT = 'gym.server.startup_stage_duration_ms'
```

```python
nemo_gym.telemetry.gym_metrics.SERVER_TYPE_ATTRIBUTE = 'nemo.gym.server.type'
```

```python
nemo_gym.telemetry.gym_metrics.STARTUP_STAGE_ATTRIBUTE = 'nemo.gym.startup.stage'
```

```python
nemo_gym.telemetry.gym_metrics.UNREGISTERED_FAILURE_KIND = 'unregistered'
```

```python
nemo_gym.telemetry.gym_metrics.WORKER_ATTRIBUTE = 'nemo.gym.server.worker'
```

```python
nemo_gym.telemetry.gym_metrics._INSTRUMENTS: dict[int, dict[str, Any]] = {}
```

```python
nemo_gym.telemetry.gym_metrics._INSTRUMENT_LOCK = threading.Lock()
```

```python
nemo_gym.telemetry.gym_metrics.logger = logging.getLogger(__name__)
```