Configuration Reference

A submit config is a YAML file with the top-level sections services, compute, driver, job, and optionally otel. All models are strict: an unknown key anywhere is an error, which is how typos surface. Top-level keys that start with _ are the exception; they are scratch namespaces for interpolation and are dropped before validation.

Fields without a default are required.

env values

env appears on every service and on the driver. Every value must start with one of three prefixes; a missing or misspelled prefix is an error, not a guess.

env:
  HF_TOKEN: host:HF_TOKEN
  TRUST_REMOTE_CODE: lit:1
  CUDA_VISIBLE_DEVICES: runtime:CUDA_VISIBLE_DEVICES

VAR must be a valid environment variable name. host: values are resolved when the config is validated, so they must be set for --resolve-only too. In the persisted resolved-config.yaml and gym-job.json the original host:VAR reference is written back, not the secret.

services

A mapping from service name to a service definition. The type field selects the kind: vllm, vllm_pd, or ray. Service names are free-form, but they become part of log file names and shell variable names.

Fields common to every service

type: vllm

A vLLM server, started with vllm serve.

If health_check is not set, it defaults to /health on port. If it is set without a port, the service’s port is used.

type: vllm_pd

Prefill/decode disaggregated vLLM: two vLLM tiers behind a vllm-router. A single vllm_pd entry named X deploys three services: X-prefill, X-decode, and the router X. Set driver.policy_model to X. The router’s container must provide vllm-router. See the how-to.

type: ray

A Ray cluster. The head starts on the driver’s node and every other node of the spanned pools joins it as a worker.

A ray service is not a model service: it cannot be driver.policy_model for URL injection, and it has no default health check. The head’s address is exported to the job as GYM_RAY_ADDRESS_<SERVICE> (service name upper-cased, non-alphanumerics replaced by _).

compute

A mapping with exactly one entry; its key is a name you choose (the examples use cluster). More than one is an error.

type: slurm

Use forward_env with runtime: for credentials needed by a remote job:

compute:
  cluster:
    type: slurm
    hostname: login.example.com
    account: my-account
    forward_env: [HF_TOKEN, INFERENCE_API_KEY]
driver:
  env:
    HF_TOKEN: runtime:HF_TOKEN
    INFERENCE_API_KEY: runtime:INFERENCE_API_KEY

Only listed names are forwarded. Leave scheduler-provided values such as CUDA_VISIBLE_DEVICES out of this list. --resolve-only and dry runs do not require forwarded values to be set. Slurm must export the submission environment to the job; an explicit --export=NONE prevents that inheritance.

Node pools

A benchmark is one Slurm job. Pools do not create separate Slurm requests; they divide the one allocation between services. The job requests the sum of every pool’s nodes, and pool i owns the contiguous node range after all pools before it, in declaration order. Because Slurm takes one value per job, pools must agree on partition, ntasks_per_node, and gpus_per_node, and may not set the same extra_args key to different values. Disagreement is an error.

driver

The driver runs the benchmarks against the services.

Policy wiring

When policy_model names a vllm or vllm_pd service, each benchmark’s run config receives:

• policy_base_url: http://<policy_host>:<service port>/v1
• policy_model_name: the service’s served_model_name, or its model
• policy_api_key: a dummy value (vLLM does not require authentication)

If a benchmark’s run already sets any of these keys, validation fails: remove one or the other. policy_model must name an existing service.

benchmarks

Each key under driver.benchmarks is a benchmark name. It must match ^[A-Za-z0-9._-]+$, because it is used in job names and in the line each sbatch reports back.

prepare and run are nested mappings that are flattened to Hydra overrides: {split: benchmark, config_paths: [a.yaml, b.yaml]} becomes +split=benchmark and +config_paths=[a.yaml,b.yaml], and nested keys become dotted paths.

Submit sets a few defaults on run that you can override:

resumable

Asking for resumable on an executor that does not support it is an error rather than a silent no-op.

job

otel

The OpenTelemetry collector that runs beside every benchmark job. It is enabled by default, and an enabled collector requires endpoint, service_name, and the token variable (token_env, default OTEL_TOKEN) to be set in the submitting shell. Set enabled: false to opt out. All fields, and what is exported, are described in Collector for Submitted Jobs.

Cross-field validation

Schema validation runs when the config is composed, before anything is rendered or contacted. --resolve-only performs these checks and stops. The pool-agreement checks listed at the end are made later, while the script is rendered, so use --dry-run to verify them:

• One compute resource. More than one is rejected. A service’s placement must name it.
• Node pool references. A service’s node_pool, and every pool in a ray service’s node_pools, must exist in the compute’s node_pools.
• Tier name collisions. A vllm_pd service X deploys X-prefill and X-decode; services must not already contain those names.
• Prefill/decode tiers each need a node_pool, must use different data_parallel_rpc_port values, and with server_per_node must have number_of_instances: 1. The router, if pinned, must be pinned to the first pool, since the driver reaches it over localhost on the first node.
• vLLM GPU footprint (when pools declare gpus_per_node). The GPUs a service needs are compared with what it can use. Too few is an error that names the footprint; too many is a warning that GPUs would idle. The calculation depends on the deployment shape:
  • single node: tensor_parallel_size x pipeline_parallel_size x number_of_instances against one node’s GPUs;
  • one instance across nodes: tensor_parallel_size x pipeline_parallel_size against all GPUs of the nodes;
  • multi-node data parallel: the per-node share of replicas against one node’s GPUs; and
  • Ray Serve gateway: the total footprint against all GPUs.
• Multi-node data parallelism requires number_of_instances to be evenly divisible by the number of nodes, unless the Ray Serve gateway is in use.
• Policy wiring conflicts, as described above.
• Benchmark command combined with run or resumable is rejected.
• Pool agreement (checked at render time, not by --resolve-only). Pools used together by one job must agree on partition, tasks, GPUs, and extra directives. A mismatch fails --dry-run and the real submit.

Mount source existence, SSH reachability, and OTel destination checks happen later, at submit time.