> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/gym/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/gym/_mcp/server.

# nemo_gym.orchestration.executors.otel

The OpenTelemetry collector that runs beside every benchmark job.

One collector per benchmark job, in its own `srun --overlap` step like any other service. It
scrapes each model service's Prometheus `/metrics` on localhost, accepts OTLP from the job's own
processes, stamps the resource attributes the shared dashboards filter on (`user`, `run_id`,
`slurm_job_id`, plus `benchmark`, `cluster`, `model`), and exports to the configured OTLP/HTTP
endpoint and to `<job dir>/otel/*.jsonl` at the same time.

The ingest token travels twice, as the HTTP `Authorization` header and as an `Authorization`
resource attribute: a routing proxy in front of the backend resolves the tenant from the
attribute and silently drops payloads without it while still answering 200.

## Module Contents

### Functions

| Name                                                                                        | Description                                                                               |
| ------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------- |
| [`_policy_model_name`](#nemo_gym-orchestration-executors-otel-_policy_model_name)           | -                                                                                         |
| [`collector_config_path`](#nemo_gym-orchestration-executors-otel-collector_config_path)     | -                                                                                         |
| [`otel_active`](#nemo_gym-orchestration-executors-otel-otel_active)                         | Whether a collector step is added to this job: enabled, and there is something to scrape. |
| [`render_collector_config`](#nemo_gym-orchestration-executors-otel-render_collector_config) | The collector's YAML for one benchmark job.                                               |
| [`resolve_token`](#nemo_gym-orchestration-executors-otel-resolve_token)                     | The ingest token from the submitting environment; a missing one fails the submit.         |
| [`scrape_targets`](#nemo_gym-orchestration-executors-otel-scrape_targets)                   | Service name to serving port for every service that exposes a model (and so `/metrics`).  |
| [`validate_destination`](#nemo_gym-orchestration-executors-otel-validate_destination)       | An enabled collector needs somewhere to send to; a bare default config has none.          |

### Data

[`COLLECTOR_CONFIG_NAME`](#nemo_gym-orchestration-executors-otel-COLLECTOR_CONFIG_NAME)

[`COLLECTOR_DIR`](#nemo_gym-orchestration-executors-otel-COLLECTOR_DIR)

[`COLLECTOR_HEALTH_PORT`](#nemo_gym-orchestration-executors-otel-COLLECTOR_HEALTH_PORT)

[`COLLECTOR_SERVICE_NAME`](#nemo_gym-orchestration-executors-otel-COLLECTOR_SERVICE_NAME)

[`FINAL_SCRAPE_GRACE_SECONDS`](#nemo_gym-orchestration-executors-otel-FINAL_SCRAPE_GRACE_SECONDS)

[`OTLP_GRPC_PORT`](#nemo_gym-orchestration-executors-otel-OTLP_GRPC_PORT)

[`OTLP_HTTP_PORT`](#nemo_gym-orchestration-executors-otel-OTLP_HTTP_PORT)

[`SHUTDOWN_WAIT_SECONDS`](#nemo_gym-orchestration-executors-otel-SHUTDOWN_WAIT_SECONDS)

### API

```python
nemo_gym.orchestration.executors.otel._policy_model_name(
    config: nemo_gym.orchestration.api.SubmitConfig
) -> str | None
```

```python
nemo_gym.orchestration.executors.otel.collector_config_path(
    remote_bench_dir: pathlib.Path
) -> pathlib.Path
```

```python
nemo_gym.orchestration.executors.otel.otel_active(
    config: nemo_gym.orchestration.api.SubmitConfig
) -> bool
```

Whether a collector step is added to this job: enabled, and there is something to scrape.

```python
nemo_gym.orchestration.executors.otel.render_collector_config(
    config: nemo_gym.orchestration.api.SubmitConfig,
    benchmark_name: str,
    remote_bench_dir: pathlib.Path
) -> str
```

The collector's YAML for one benchmark job.

`run_id` is the submission's gym job id, which is the job directory's parent by construction
(`<output_path>/<gym_job_id>/<benchmark>`); `cluster` is the sole compute key, the same value
`SubmissionRecord.cluster` records. Values only known inside the job (`SLURM_JOB_ID`, the
token) are left as `${env:...}` for the collector to expand at startup.

```python
nemo_gym.orchestration.executors.otel.resolve_token(
    config: nemo_gym.orchestration.api.SubmitConfig
) -> str
```

The ingest token from the submitting environment; a missing one fails the submit.

```python
nemo_gym.orchestration.executors.otel.scrape_targets(
    config: nemo_gym.orchestration.api.SubmitConfig
) -> dict[str, int]
```

Service name to serving port for every service that exposes a model (and so `/metrics`).

```python
nemo_gym.orchestration.executors.otel.validate_destination(
    config: nemo_gym.orchestration.api.SubmitConfig
) -> None
```

An enabled collector needs somewhere to send to; a bare default config has none.

```python
nemo_gym.orchestration.executors.otel.COLLECTOR_CONFIG_NAME = 'collector.yaml'
```

```python
nemo_gym.orchestration.executors.otel.COLLECTOR_DIR = 'otel'
```

```python
nemo_gym.orchestration.executors.otel.COLLECTOR_HEALTH_PORT = 13133
```

```python
nemo_gym.orchestration.executors.otel.COLLECTOR_SERVICE_NAME = 'otel_collector'
```

```python
nemo_gym.orchestration.executors.otel.FINAL_SCRAPE_GRACE_SECONDS = 20
```

```python
nemo_gym.orchestration.executors.otel.OTLP_GRPC_PORT = 4317
```

```python
nemo_gym.orchestration.executors.otel.OTLP_HTTP_PORT = 4318
```

```python
nemo_gym.orchestration.executors.otel.SHUTDOWN_WAIT_SECONDS = 30
```