nemo_gym.telemetry.gym_metrics

View as Markdown

Attributed OTel instruments that nemo.lens.instruments.gym.record_gym_metrics cannot express.

nemo_gym.telemetry.metrics forwards to the five fixed, undimensioned instruments nemo-lens declares. Anything that needs an attribute (a provider, a site, a class) is created here, directly on the lens meter, and cached per meter so a re-initialised telemetry handle gets fresh instruments. Every recorder is a no-op unless telemetry is initialised and exporting, and never raises into its caller; call sites still sit under a span-group gate so they cost nothing when disabled.

Sandbox lifecycle

gym.sandbox.active (up-down counter): sandboxes this process currently holds, +1 when a provider hands back a handle, -1 when Gym releases it. Summed by the backend across the processes that hold sandboxes; a gauge would show whichever process wrote last.

gym.sandbox.startup_duration_ms / gym.sandbox.exec_duration_ms (histograms): wall-clock of one provisioning and one command. Explicit bucket boundaries up to thirty minutes: the SDK default stops at ten seconds and a sandbox start routinely takes a minute.

gym.sandbox.create_retry_total (counter): one per create attempt a provider retried. Retrying is provider-internal, so each provider that retries records it from its own loop.

All four carry nemo.gym.sandbox.provider.

Module Contents

Functions

NameDescription
_get_or_create-
_meter-
_record_counter-
_record_histogram-
_record_up_down_counter-
_reset_for_testingDrop cached instruments. Test-only.
record_sandbox_activeAdd delta (+1 on start, -1 on stop) to gym.sandbox.active for provider.
record_sandbox_create_retryCount one retried sandbox-create attempt, by provider.
record_sandbox_exec_durationRecord one command’s wall-clock inside an already-provisioned sandbox, by provider.
record_sandbox_startupRecord one sandbox’s provisioning wall-clock, by provider.

Data

SANDBOX_ACTIVE_INSTRUMENT

SANDBOX_CREATE_RETRY_INSTRUMENT

SANDBOX_DURATION_BOUNDARIES_MS

SANDBOX_EXEC_INSTRUMENT

SANDBOX_PROVIDER_ATTRIBUTE

SANDBOX_STARTUP_INSTRUMENT

_INSTRUMENTS

_INSTRUMENT_LOCK

logger

API

nemo_gym.telemetry.gym_metrics._get_or_create(
meter: typing.Any,
name: str,
factory: collections.abc.Callable[[], typing.Any]
) -> typing.Any
nemo_gym.telemetry.gym_metrics._meter() -> typing.Optional[typing.Any]
nemo_gym.telemetry.gym_metrics._record_counter(
name: str,
description: str,
attributes: dict[str, typing.Any],
amount: int = 1
) -> None
nemo_gym.telemetry.gym_metrics._record_histogram(
name: str,
unit: str,
description: str,
value: float,
attributes: dict[str, typing.Any],
boundaries: collections.abc.Sequence[float] | None = None
) -> None
nemo_gym.telemetry.gym_metrics._record_up_down_counter(
name: str,
unit: str,
description: str,
delta: int,
attributes: dict[str, typing.Any]
) -> None
nemo_gym.telemetry.gym_metrics._reset_for_testing() -> None

Drop cached instruments. Test-only.

nemo_gym.telemetry.gym_metrics.record_sandbox_active(
delta: int,
provider: str
) -> None

Add delta (+1 on start, -1 on stop) to gym.sandbox.active for provider.

nemo_gym.telemetry.gym_metrics.record_sandbox_create_retry(
provider: str
) -> None

Count one retried sandbox-create attempt, by provider.

nemo_gym.telemetry.gym_metrics.record_sandbox_exec_duration(
duration_ms: float,
provider: str
) -> None

Record one command’s wall-clock inside an already-provisioned sandbox, by provider.

nemo_gym.telemetry.gym_metrics.record_sandbox_startup(
duration_ms: float,
provider: str
) -> None

Record one sandbox’s provisioning wall-clock, by provider.

nemo_gym.telemetry.gym_metrics.SANDBOX_ACTIVE_INSTRUMENT = 'gym.sandbox.active'
nemo_gym.telemetry.gym_metrics.SANDBOX_CREATE_RETRY_INSTRUMENT = 'gym.sandbox.create_retry_total'
nemo_gym.telemetry.gym_metrics.SANDBOX_DURATION_BOUNDARIES_MS: tuple[float, ...] = (250, 500, 1000, 2000, 5000, 10000, 30000, 60000, 120000, 300000, 600000, 180000...
nemo_gym.telemetry.gym_metrics.SANDBOX_EXEC_INSTRUMENT = 'gym.sandbox.exec_duration_ms'
nemo_gym.telemetry.gym_metrics.SANDBOX_PROVIDER_ATTRIBUTE = 'nemo.gym.sandbox.provider'
nemo_gym.telemetry.gym_metrics.SANDBOX_STARTUP_INSTRUMENT = 'gym.sandbox.startup_duration_ms'
nemo_gym.telemetry.gym_metrics._INSTRUMENTS: dict[int, dict[str, Any]] = {}
nemo_gym.telemetry.gym_metrics._INSTRUMENT_LOCK = threading.Lock()
nemo_gym.telemetry.gym_metrics.logger = logging.getLogger(__name__)