Collector for Submitted Jobs

gym eval submit adds an OpenTelemetry Collector to every benchmark job it renders. The collector scrapes the model services’ Prometheus metrics, receives OTLP from anything in the job, stamps a per-run identity on the data, and ships it to the OTLP/HTTP endpoint the deployment configures. It is on by default and needs three site-specific values.

Configure

otel:
  enabled: true                       # default
  endpoint: https://otlp.example.com  # OTLP/HTTP ingest; required when enabled
  service_name: my-eval-service       # service.name resource attribute the backend routes by; required
  token_env: OTEL_TOKEN      # env var in the submitting shell holding the bearer token
  binary: /shared/otelcol-contrib     # static collector binary reachable on the compute nodes
  container: null                     # optional image instead of the host binary
  component: gym-vllm                 # display name for the scraped engine metrics
  gpu_metrics_port: 9400              # dcgm-exporter on the node; null to skip
  node_metrics_port: 9100             # node_exporter on the node; null to skip
  scrape_interval_seconds: 15
  health_check_timeout_seconds: 300

The block is named otel because it configures the OpenTelemetry Collector only; Gym’s rollout evidence (observability_enabled) and the in-process SDK (telemetry.enabled, NEMO_GYM_OTEL_*) are separate settings. A deployment usually puts endpoint, service_name, token_env and binary in a shared cluster fragment; user configs rarely need to mention the block. Set enabled: false to opt a run out. A job without a model service gets no collector.

The submit fails before staging anything when enabled is true and endpoint or service_name is unset, or when the token variable is missing from the submitting shell. The error names the key to set.

What runs

The collector is the first srun --overlap step of the sbatch script and must answer its health check before the services start. By default it runs as a host process from binary; the upstream collector image is distroless and cannot start under enroot, so use container only with an image that has a shell.

After the driver exits, the script waits one scrape interval so the final counters are collected, signals the collector process (not its srun), waits for a clean shutdown, and exits with the driver’s return code.

What is exported

Every batch carries resource attributes the dashboards filter on: user, run_id (the gym job id), slurm_job_id, benchmark, cluster and model, plus service.name set to the routing identity and service.name.override to the producer’s display name.

Everything exported is also written to <job dir>/otel/{metrics,traces,logs}.jsonl, so a run can be inspected without backend access. The rendered collector configuration is at <job dir>/otel/collector.yaml; gym eval submit --resolve-only prints it without submitting.

The token

The bearer token is read from the submitting shell, exported into the remote shell that runs sbatch, and referenced from the collector configuration as ${env:<token_env>}. It is not written to the job directory, the sbatch script or the resolved config.