> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/gym/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/gym/_mcp/server.

# nemo_gym.telemetry.gym_metrics

Attributed OTel instruments that `nemo.lens.instruments.gym.record_gym_metrics` cannot express.

`nemo_gym.telemetry.metrics` forwards to the five fixed, undimensioned instruments nemo-lens
declares. Anything that needs an attribute (a provider, a site, a class) is created here,
directly on the lens meter, and cached per meter so a re-initialised telemetry handle gets
fresh instruments. Every recorder is a no-op unless telemetry is initialised and exporting,
and never raises into its caller; call sites still sit under a span-group gate so they cost
nothing when disabled.

## Sandbox lifecycle

`gym.sandbox.active` (up-down counter): sandboxes this process currently holds, `+1` when a
provider hands back a handle, `-1` when Gym releases it. Summed by the backend across the
processes that hold sandboxes; a gauge would show whichever process wrote last.

`gym.sandbox.startup_duration_ms` / `gym.sandbox.exec_duration_ms` (histograms): wall-clock
of one provisioning and one command. Explicit bucket boundaries up to thirty minutes: the SDK
default stops at ten seconds and a sandbox start routinely takes a minute.

`gym.sandbox.create_retry_total` (counter): one per create attempt a provider retried.
Retrying is provider-internal, so each provider that retries records it from its own loop.

All four carry `nemo.gym.sandbox.provider`.

## Module Contents

### Functions

| Name                                                                                           | Description                                                                         |
| ---------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------- |
| [`_get_or_create`](#nemo_gym-telemetry-gym_metrics-_get_or_create)                             | -                                                                                   |
| [`_meter`](#nemo_gym-telemetry-gym_metrics-_meter)                                             | -                                                                                   |
| [`_record_counter`](#nemo_gym-telemetry-gym_metrics-_record_counter)                           | -                                                                                   |
| [`_record_histogram`](#nemo_gym-telemetry-gym_metrics-_record_histogram)                       | -                                                                                   |
| [`_record_up_down_counter`](#nemo_gym-telemetry-gym_metrics-_record_up_down_counter)           | -                                                                                   |
| [`_reset_for_testing`](#nemo_gym-telemetry-gym_metrics-_reset_for_testing)                     | Drop cached instruments. Test-only.                                                 |
| [`record_sandbox_active`](#nemo_gym-telemetry-gym_metrics-record_sandbox_active)               | Add `delta` (`+1` on start, `-1` on stop) to `gym.sandbox.active` for `provider`.   |
| [`record_sandbox_create_retry`](#nemo_gym-telemetry-gym_metrics-record_sandbox_create_retry)   | Count one retried sandbox-create attempt, by provider.                              |
| [`record_sandbox_exec_duration`](#nemo_gym-telemetry-gym_metrics-record_sandbox_exec_duration) | Record one command's wall-clock inside an already-provisioned sandbox, by provider. |
| [`record_sandbox_startup`](#nemo_gym-telemetry-gym_metrics-record_sandbox_startup)             | Record one sandbox's provisioning wall-clock, by provider.                          |

### Data

[`SANDBOX_ACTIVE_INSTRUMENT`](#nemo_gym-telemetry-gym_metrics-SANDBOX_ACTIVE_INSTRUMENT)

[`SANDBOX_CREATE_RETRY_INSTRUMENT`](#nemo_gym-telemetry-gym_metrics-SANDBOX_CREATE_RETRY_INSTRUMENT)

[`SANDBOX_DURATION_BOUNDARIES_MS`](#nemo_gym-telemetry-gym_metrics-SANDBOX_DURATION_BOUNDARIES_MS)

[`SANDBOX_EXEC_INSTRUMENT`](#nemo_gym-telemetry-gym_metrics-SANDBOX_EXEC_INSTRUMENT)

[`SANDBOX_PROVIDER_ATTRIBUTE`](#nemo_gym-telemetry-gym_metrics-SANDBOX_PROVIDER_ATTRIBUTE)

[`SANDBOX_STARTUP_INSTRUMENT`](#nemo_gym-telemetry-gym_metrics-SANDBOX_STARTUP_INSTRUMENT)

[`_INSTRUMENTS`](#nemo_gym-telemetry-gym_metrics-_INSTRUMENTS)

[`_INSTRUMENT_LOCK`](#nemo_gym-telemetry-gym_metrics-_INSTRUMENT_LOCK)

[`logger`](#nemo_gym-telemetry-gym_metrics-logger)

### API

```python
nemo_gym.telemetry.gym_metrics._get_or_create(
    meter: typing.Any,
    name: str,
    factory: collections.abc.Callable[[], typing.Any]
) -> typing.Any
```

```python
nemo_gym.telemetry.gym_metrics._meter() -> typing.Optional[typing.Any]
```

```python
nemo_gym.telemetry.gym_metrics._record_counter(
    name: str,
    description: str,
    attributes: dict[str, typing.Any],
    amount: int = 1
) -> None
```

```python
nemo_gym.telemetry.gym_metrics._record_histogram(
    name: str,
    unit: str,
    description: str,
    value: float,
    attributes: dict[str, typing.Any],
    boundaries: collections.abc.Sequence[float] | None = None
) -> None
```

```python
nemo_gym.telemetry.gym_metrics._record_up_down_counter(
    name: str,
    unit: str,
    description: str,
    delta: int,
    attributes: dict[str, typing.Any]
) -> None
```

```python
nemo_gym.telemetry.gym_metrics._reset_for_testing() -> None
```

Drop cached instruments. Test-only.

```python
nemo_gym.telemetry.gym_metrics.record_sandbox_active(
    delta: int,
    provider: str
) -> None
```

Add `delta` (`+1` on start, `-1` on stop) to `gym.sandbox.active` for `provider`.

```python
nemo_gym.telemetry.gym_metrics.record_sandbox_create_retry(
    provider: str
) -> None
```

Count one retried sandbox-create attempt, by provider.

```python
nemo_gym.telemetry.gym_metrics.record_sandbox_exec_duration(
    duration_ms: float,
    provider: str
) -> None
```

Record one command's wall-clock inside an already-provisioned sandbox, by provider.

```python
nemo_gym.telemetry.gym_metrics.record_sandbox_startup(
    duration_ms: float,
    provider: str
) -> None
```

Record one sandbox's provisioning wall-clock, by provider.

```python
nemo_gym.telemetry.gym_metrics.SANDBOX_ACTIVE_INSTRUMENT = 'gym.sandbox.active'
```

```python
nemo_gym.telemetry.gym_metrics.SANDBOX_CREATE_RETRY_INSTRUMENT = 'gym.sandbox.create_retry_total'
```

```python
nemo_gym.telemetry.gym_metrics.SANDBOX_DURATION_BOUNDARIES_MS: tuple[float, ...] = (250, 500, 1000, 2000, 5000, 10000, 30000, 60000, 120000, 300000, 600000, 180000...
```

```python
nemo_gym.telemetry.gym_metrics.SANDBOX_EXEC_INSTRUMENT = 'gym.sandbox.exec_duration_ms'
```

```python
nemo_gym.telemetry.gym_metrics.SANDBOX_PROVIDER_ATTRIBUTE = 'nemo.gym.sandbox.provider'
```

```python
nemo_gym.telemetry.gym_metrics.SANDBOX_STARTUP_INSTRUMENT = 'gym.sandbox.startup_duration_ms'
```

```python
nemo_gym.telemetry.gym_metrics._INSTRUMENTS: dict[int, dict[str, Any]] = {}
```

```python
nemo_gym.telemetry.gym_metrics._INSTRUMENT_LOCK = threading.Lock()
```

```python
nemo_gym.telemetry.gym_metrics.logger = logging.getLogger(__name__)
```