Orchestration

The orchestration layer turns one YAML file into batch jobs on a cluster. You describe the model services to deploy (for example vLLM), the compute to run them on, and the benchmarks to evaluate. gym eval submit validates the file, renders one job script per benchmark, copies the scripts to the cluster, queues them, and writes a record you can use to find the run again.

Use it when a benchmark needs more than one machine can offer: a model that spans GPUs or nodes, many data-parallel replicas for rollout throughput, or a prefill/decode split. For a single-machine run, use gym eval run directly.

How it fits together

A submit config has four required sections and one optional one:

Submission works in these steps:

1. Compose and validate. Hydra composes the YAML, resolves interpolations and env prefixes, and validates it against a strict schema. Unknown keys are errors, so typos fail immediately.
2. Render. One job script is rendered for each entry in driver.benchmarks.
3. Stage and copy. The scripts are staged in a temporary directory and copied to a new run directory on the cluster, <job.output_path>/gym-job-<timestamp>-<id>/.
4. Queue. One sbatch is issued per benchmark. Each benchmark is its own Slurm job and deploys its own copy of the services.
5. Record. A manifest (gym-job.json) and the resolved config are written to the run directory, and a copy of the manifest to a local index on your machine.

See Slurm Executor for what each job script does and Job Records for the files left behind.

The command

gym eval submit --config path/to/submit.yaml [overrides...]

Anything after the flags is passed to Hydra as an override, so job.output_path=/lustre/me/evals changes a value without editing the file.

gym eval submit -c submit.yaml job.output_path=/lustre/me/evals compute.cluster.walltime=04:00:00

Scratch keys

A top-level key that starts with _ is a scratch namespace. It is not part of the schema and is dropped after interpolation, so other parts of the file can reference it:

_images:
  vllm: vllm/vllm-openai:v0.9.0
services:
  policy:
    type: vllm
    container: ${_images.vllm}
    # ...

You can override an existing scratch value (_images.vllm=...), but +_images.new=... and ++_images.new=... are rejected, so a typo cannot silently create a value that nothing reads. Any other unrecognized top-level key is a validation error.

Output and exit status

On success the command prints the run directory and one line per benchmark:

Run directory: /lustre/me/evals/gym-job-20260101T120000Z-a1b2c3
submitted gpqa → Slurm job 1234567

If sbatch rejects a benchmark, that benchmark prints failed <name>: <error>. The other benchmarks in the same submission are still queued and recorded. The command then prints Error: N benchmark(s) failed to submit to stderr and exits with status 1.

Pages in this section