Config Composition

NEL supports Hydra-style config composition: reusable fragments, base-config inheritance, and self-referencing interpolation — with zero external dependencies and full backward compatibility.

Existing flat YAML configs continue to work unchanged. Composition is opt-in via the defaults: key.

Quick example

Instead of duplicating 100+ lines across variant configs, inherit from a base:

1
# 24a_miniswe.yaml — 15 lines instead of 106
2
defaults:
3
  - _base_: 24_slurm_swebench_verified_super
4
5
benchmarks:
6
  - name: harbor://swebench-verified@1.0
7
    max_concurrent: 30
8
    solver:
9
      agent: mini-swe-agent
10
      agent_kwargs: { max_iterations: 30 }
11
    sandbox: { concurrency: 30 }
12
13
output:
14
  dir: ./eval_results/swebench-verified-miniswe

defaults: list

Add a defaults: key at the top of your config. Each entry loads and merges a config fragment or base config:

1
defaults:
2
  - services/nemotron_fp8_vllm     # loads conf/services/nemotron_fp8_vllm.yaml
3
  - clusters/slurm_oci_8gpu        # loads conf/clusters/slurm_oci_8gpu.yaml
4
  - sandboxes/ecs_swebench         # loads conf/sandboxes/ecs_swebench.yaml
5
  - _self_                         # where *this* config sits in merge order
6
7
benchmarks:
8
  - name: harbor://swebench-verified@1.0
9
    solver: { type: harbor, service: nemotron }

_self_ marks where the main config body sits. Default (when omitted): last, meaning your config has the highest priority.

Config groups

The directory name in a fragment path maps to the top-level config key:

Note: clusters/ (plural) maps to cluster: (singular). All other group names match their key directly.

Fragment file structure

A fragment file contains the content for its section, not the wrapping key:

1
# conf/clusters/slurm_oci_8gpu.yaml
2
type: slurm
3
hostname: slurm-login.example.com
4
account: <your-slurm-account>
5
walltime: "04:00:00"
6
node_pools:
7
  gpu:
8
    partition: batch
9
    gres: "gpu:8"

For services/, include the service name as a key (since multiple services can coexist):

1
# conf/services/nemotron_fp8_vllm.yaml
2
nemotron:
3
  type: vllm
4
  model: nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-FP8
5
  # ... full config ...

Base-config inheritance

Use _base_: to inherit from another complete config:

1
defaults:
2
  - _base_: 24_slurm_swebench_verified_super
3
4
# Only specify what differs
5
benchmarks:
6
  - name: harbor://swebench-verified@1.0
7
    solver:
8
      agent: mini-swe-agent
9
output:
10
  dir: /path/to/different/output

The base config is loaded and recursively composed (including its own defaults:), then the current config’s body is deep-merged on top.

Chains work too: grandparent → parent → child.

Merge semantics

• Dicts: recursive deep merge (later values win on conflict)
• Lists: replace entirely (no concatenation)
• Scalars: later value wins
• null: explicitly deletes a key from the base

1
defaults:
2
  - _base_: base_config
3
4
output:
5
  extra_field: null     # removes extra_field inherited from base
6
  dir: /new/output      # overrides base's dir

Self-referencing interpolation

Reference other values in the merged config with ${.path.to.value}:

1
services:
2
  model:
3
    type: vllm
4
    model: nvidia/Llama-3.1-70B
5
6
output:
7
  dir: ./results/${.services.model.model}    # → ./results/nvidia/Llama-3.1-70B

The leading dot distinguishes self-references from environment variables:

• ${.services.model.model} → self-reference (resolved during composition)
• ${HF_TOKEN} → env var (resolved later by _expand_env)

Resolution order

1. Composition — defaults: fragments are merged and _base_: inheritance applied
2. Self-refs — ${.path} references resolved against the merged dict
3. CLI overrides — -O key=value applied
4. Pydantic validation — schema checked, types coerced
5. Env-var expansion — ${HF_TOKEN} resolved at runtime (not during composition)

This means self-refs see the fully-merged config but not CLI overrides. Env vars are intentionally deferred so secrets stay out of serialized configs.

Search path

Fragments are resolved in order (first match wins):

1. conf/ relative to the config file
2. ~/.config/nemo-evaluator/conf/
3. Package built-in defaults (shipped with NEL)

For _base_: references, the directory containing the referencing config is searched first.

File extensions .yaml and .yml are tried automatically.

CLI overrides

-O key=value overrides are applied after composition, so they always win:

$
nel eval run config.yaml -O output.dir=/tmp/test -O benchmarks.0.max_problems=5

Creating your own fragment library

my-project/
  configs/
    conf/
      services/
        nemotron_fp8.yaml
        llama_70b.yaml
      clusters/
        slurm_8gpu.yaml
        slurm_4gpu.yaml
      sandboxes/
        ecs_swebench.yaml
    swebench_openhands.yaml       # uses defaults: to compose
    swebench_miniswe.yaml         # _base_: swebench_openhands

Or share team-wide fragments via ~/.config/nemo-evaluator/conf/.

Error handling

• Missing fragment: FileNotFoundError with the fragment name and all search paths that were tried
• Circular reference: ValueError if config A references B which references A
• Invalid entry: ValueError for unrecognized defaults: entries