nemo_gym.orchestration.executors.otel

The OpenTelemetry collector that runs beside every benchmark job.

One collector per benchmark job, in its own srun --overlap step like any other service. It scrapes each model service’s Prometheus /metrics on localhost, accepts OTLP from the job’s own processes, stamps the resource attributes the shared dashboards filter on (user, run_id, slurm_job_id, plus benchmark, cluster, model), and exports to the configured OTLP/HTTP endpoint (when remote export is enabled) and to <job dir>/otel/*.jsonl.

The ingest token travels twice, as the HTTP Authorization header and as an Authorization resource attribute: a routing proxy in front of the backend resolves the tenant from the attribute and silently drops payloads without it while still answering 200.

Module Contents

Functions

Name	Description
`_policy_model_name`	-
`collector_config_path`	-
`driver_telemetry_env`	Environment that switches on Gym’s Lens instrumentation and points it at the collector.
`gym_telemetry_active`	Whether the driver is instrumented with nemo-lens and pointed at the collector.
`otel_active`	Enabled is enough: Gym’s own servers push telemetry even when there is no local model to scrape.
`render_collector_config`	The collector’s YAML for one benchmark job.
`resolve_token`	Resolve the remote ingest token, switching to local collection with a warning if absent.
`scrape_targets`	Service name to serving port for every service that exposes a model (and so `/metrics`).
`validate_destination`	Remote export needs a destination; local collection does not.
`validate_gym_telemetry`	-
`vllm_traces_env`	Environment for a traced vLLM service.
`vllm_traces_flag`	The `vllm serve` flag that sends one span per request to the collector’s OTLP/HTTP receiver.

Data

COLLECTOR_CONFIG_NAME

COLLECTOR_DIR

COLLECTOR_HEALTH_PORT

COLLECTOR_SERVICE_NAME

FINAL_SCRAPE_GRACE_SECONDS

GYM_TELEMETRY_EXTRA

OTLP_GRPC_PORT

OTLP_HTTP_PORT

SHUTDOWN_WAIT_SECONDS

VLLM_TRACES_FLAG

logger

API

nemo_gym.orchestration.executors.otel._policy_model_name(
    config: nemo_gym.orchestration.api.SubmitConfig
) -> str | None

nemo_gym.orchestration.executors.otel.collector_config_path(
    remote_bench_dir: pathlib.Path
) -> pathlib.Path

nemo_gym.orchestration.executors.otel.driver_telemetry_env(
    gym_job_id: str,
    span_groups: str,
    logs: bool = True
) -> dict[str, str]

Environment that switches on Gym’s Lens instrumentation and points it at the collector.

Gym reads these in every server process (NEMO_GYM_OTEL_* are Gym’s own, the OTEL_* ones are the SDK’s); an explicit value in driver.env wins over these.

nemo_gym.orchestration.executors.otel.gym_telemetry_active(
    config: nemo_gym.orchestration.api.SubmitConfig
) -> bool

Whether the driver is instrumented with nemo-lens and pointed at the collector.

nemo_gym.orchestration.executors.otel.otel_active(
    config: nemo_gym.orchestration.api.SubmitConfig
) -> bool

Enabled is enough: Gym’s own servers push telemetry even when there is no local model to scrape.

nemo_gym.orchestration.executors.otel.render_collector_config(
    config: nemo_gym.orchestration.api.SubmitConfig,
    benchmark_name: str,
    remote_bench_dir: pathlib.Path
) -> str

The collector’s YAML for one benchmark job.

run_id is the submission’s gym job id, which is the job directory’s parent by construction (<output_path>/<gym_job_id>/<benchmark>); cluster is the sole compute key, the same value SubmissionRecord.cluster records. Values only known inside the job (SLURM_JOB_ID, the token) are left as ${env:...} for the collector to expand at startup.

nemo_gym.orchestration.executors.otel.resolve_token(
    config: nemo_gym.orchestration.api.SubmitConfig
) -> str | None

Resolve the remote ingest token, switching to local collection with a warning if absent.

nemo_gym.orchestration.executors.otel.scrape_targets(
    config: nemo_gym.orchestration.api.SubmitConfig
) -> dict[str, int]

Service name to serving port for every service that exposes a model (and so /metrics).

nemo_gym.orchestration.executors.otel.validate_destination(
    config: nemo_gym.orchestration.api.SubmitConfig
) -> None

Remote export needs a destination; local collection does not.

nemo_gym.orchestration.executors.otel.validate_gym_telemetry(
    config: nemo_gym.orchestration.api.SubmitConfig
) -> None

nemo_gym.orchestration.executors.otel.vllm_traces_env(
    component: str,
    service_name: str
) -> dict[str, str]

Environment for a traced vLLM service.

vLLM picks the exporter by OTEL_EXPORTER_OTLP_TRACES_PROTOCOL and defaults to gRPC, so the protocol has to match the HTTP endpoint in the flag. vLLM sets no service.name of its own; the collector copies the one given here into service.name.override, so the spans carry the same display identity as the service’s scraped metrics (<component>/<service>).

nemo_gym.orchestration.executors.otel.vllm_traces_flag(
    host: str = 'localhost'
) -> str

The vllm serve flag that sends one span per request to the collector’s OTLP/HTTP receiver.

nemo_gym.orchestration.executors.otel.COLLECTOR_CONFIG_NAME = 'collector.yaml'

nemo_gym.orchestration.executors.otel.COLLECTOR_DIR = 'otel'

nemo_gym.orchestration.executors.otel.COLLECTOR_HEALTH_PORT = 13133

nemo_gym.orchestration.executors.otel.COLLECTOR_SERVICE_NAME = 'otel_collector'

nemo_gym.orchestration.executors.otel.FINAL_SCRAPE_GRACE_SECONDS = 20

nemo_gym.orchestration.executors.otel.GYM_TELEMETRY_EXTRA = 'telemetry'

nemo_gym.orchestration.executors.otel.OTLP_GRPC_PORT = 4317

nemo_gym.orchestration.executors.otel.OTLP_HTTP_PORT = 4318

nemo_gym.orchestration.executors.otel.SHUTDOWN_WAIT_SECONDS = 30

nemo_gym.orchestration.executors.otel.VLLM_TRACES_FLAG = '--otlp-traces-endpoint'

nemo_gym.orchestration.executors.otel.logger = logging.getLogger(__name__)