> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/gym/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nemo/gym/_mcp/server.

# Collector for Submitted Jobs

> gym eval submit runs an OpenTelemetry Collector beside every benchmark job.

`gym eval submit` adds an OpenTelemetry Collector to every benchmark job it renders. The collector
scrapes the model services' Prometheus metrics, receives OTLP from anything in the job, stamps a
per-run identity on the data, and ships it to the OTLP/HTTP endpoint the deployment configures.
It is on by default and needs three site-specific values.

## Configure

```yaml
otel:
  enabled: true                       # default
  endpoint: https://otlp.example.com  # OTLP/HTTP ingest; required when enabled
  service_name: my-eval-service       # service.name resource attribute the backend routes by; required
  token_env: OTEL_TOKEN      # env var in the submitting shell holding the bearer token
  binary: /shared/otelcol-contrib     # static collector binary reachable on the compute nodes
  container: null                     # optional image instead of the host binary
  component: gym-vllm                 # display name for the scraped engine metrics
  gpu_metrics_port: 9400              # dcgm-exporter on the node; null to skip
  node_metrics_port: 9100             # node_exporter on the node; null to skip
  scrape_interval_seconds: 15
  health_check_timeout_seconds: 300
```

The block is named `otel` because it configures the OpenTelemetry Collector only; Gym's rollout
evidence (`observability_enabled`) and the in-process SDK (`telemetry.enabled`, `NEMO_GYM_OTEL_*`)
are separate settings. A deployment usually puts `endpoint`, `service_name`, `token_env` and `binary` in a shared cluster
fragment; user configs rarely need to mention the block. Set `enabled: false` to opt a run out. A job
without a model service gets no collector.

The submit fails before staging anything when `enabled` is true and `endpoint` or `service_name` is
unset, or when the token variable is missing from the submitting shell. The error names the key to set.

## What runs

The collector is the first `srun --overlap` step of the sbatch script and must answer its health
check before the services start. By default it runs as a host process from `binary`; the upstream
collector image is distroless and cannot start under enroot, so use `container` only with an image
that has a shell.

After the driver exits, the script waits one scrape interval so the final counters are collected,
signals the collector process (not its `srun`), waits for a clean shutdown, and exits with the
driver's return code.

## What is exported

| Source                             | How                                                                | Notes                                                                                          |
| ---------------------------------- | ------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------- |
| Model services (`type: vllm`, ...) | Prometheus scrape of `localhost:<port>/metrics`                    | `vllm:<name>` is renamed to `vllm_<name>` after parsing so histogram and counter types survive |
| `dcgm-exporter`, `node_exporter`   | Prometheus scrape on the node when the ports are set               | per-GPU activity, memory and power; CPU, memory, network                                       |
| Anything in the job                | OTLP traces and logs on `localhost:4317` (gRPC) and `:4318` (HTTP) | the on-ramp for Gym's own Lens telemetry                                                       |

Every batch carries resource attributes the dashboards filter on: `user`, `run_id` (the gym job id),
`slurm_job_id`, `benchmark`, `cluster` and `model`, plus `service.name` set to the routing identity
and `service.name.override` to the producer's display name.

Everything exported is also written to `<job dir>/otel/{metrics,traces,logs}.jsonl`, so a run can be
inspected without backend access. The rendered collector configuration is at
`<job dir>/otel/collector.yaml`; `gym eval submit --resolve-only` prints it without submitting.

## The token

The bearer token is read from the submitting shell, exported into the remote shell that runs
`sbatch`, and referenced from the collector configuration as `${env:<token_env>}`. It is not written
to the job directory, the sbatch script or the resolved config.