Custom Harness

Some benchmarks do not run through Gym’s rollout collection: the harness may provision external machines and drive Gym from there, for example. For those, set command on the benchmark. The driver runs your script instead of gym eval run, in the same container and with the same services running.

driver:
  policy_model: policy
  gym_install:
    ref: main
  mounts:
    - /lustre/my-project/harness:/harness:ro
  benchmarks:
    my-harness:
      prepare:
        split: benchmark
      command: |
        python /harness/run_my_harness.py \
          --base-url "$NEMO_GYM_POLICY_BASE_URL" \
          --model "$NEMO_GYM_POLICY_MODEL_NAME" \
          --api-key "$NEMO_GYM_POLICY_API_KEY" \
          --output "$NEMO_GYM_BENCH_DIR/artifacts/results.json"

What your script gets

The driver clones Gym into a fresh temporary checkout when gym_install is set, so a script that sits next to your submit config is not copied there. Put the harness on a shared cluster path, mount it with driver.mounts, and call it by absolute path, as /harness/run_my_harness.py above.

The variables from driver.env are present too, and driver.container and driver.mounts apply as usual.

What changes compared with gym eval run

• The command is emitted verbatim as the last statement of the driver’s shell, under set -euo pipefail. A multi-line script is fine, and its exit status is the job’s.
• If gym_install is set, the install step still runs first, and so does prepare. Your script starts in the Gym checkout.
• Nothing from run is applied, which is why run and command cannot be combined: run only configures gym eval run. Put those settings in your script.
• --model-type, require_complete, observability capture, and the default rollouts.jsonl output are properties of gym eval run. They do not apply; your script decides where results go and what counts as complete.
• resumable cannot be combined with command. A resumed job only passes resume_from_cache to gym eval run, so a custom command would restart from zero.