> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nemo/gym/llms.txt.
> For full documentation content, see https://docs.nvidia.com/nemo/gym/llms-full.txt.

# nemo_gym.benchmarks

Benchmark discovery and preparation utilities.

## Module Contents

### Classes

| Name                                                                    | Description                                                          |
| ----------------------------------------------------------------------- | -------------------------------------------------------------------- |
| [`BenchmarkConfig`](#nemo_gym-benchmarks-BenchmarkConfig)               | -                                                                    |
| [`PrepareBenchmarkConfig`](#nemo_gym-benchmarks-PrepareBenchmarkConfig) | Prepare benchmark data by running the benchmark's prepare.py script. |

### Functions

| Name                                                                                            | Description                             |
| ----------------------------------------------------------------------------------------------- | --------------------------------------- |
| [`_load_benchmarks_from_config_paths`](#nemo_gym-benchmarks-_load_benchmarks_from_config_paths) | -                                       |
| [`_multiprocess_benchmark_prepare_fn`](#nemo_gym-benchmarks-_multiprocess_benchmark_prepare_fn) | -                                       |
| [`list_benchmarks`](#nemo_gym-benchmarks-list_benchmarks)                                       | CLI command: list available benchmarks. |
| [`prepare_benchmark`](#nemo_gym-benchmarks-prepare_benchmark)                                   | CLI command: prepare benchmark data.    |

### Data

[`BENCHMARKS_DIR`](#nemo_gym-benchmarks-BENCHMARKS_DIR)

### API

<Anchor id="nemo_gym-benchmarks-BenchmarkConfig">
  <CodeBlock showLineNumbers={false} wordWrap={true}>
    ```python
    class nemo_gym.benchmarks.BenchmarkConfig()
    ```
  </CodeBlock>
</Anchor>

<Indent>
  **Bases:** `BaseModel`

  <ParamField path="agent_name" type="str" />

  <ParamField path="dataset" type="BenchmarkDatasetConfig" />

  <ParamField path="name" type="str" />

  <ParamField path="num_repeats" type="int" />

  <ParamField path="path" type="Path" />

  <Anchor id="nemo_gym-benchmarks-BenchmarkConfig-from_config_path">
    <CodeBlock links={{"nemo_gym.benchmarks.BenchmarkConfig":"#nemo_gym-benchmarks-BenchmarkConfig"}} showLineNumbers={false} wordWrap={true}>
      ```python
      nemo_gym.benchmarks.BenchmarkConfig.from_config_path(
          config_path: pathlib.Path
      ) -> typing.Optional[nemo_gym.benchmarks.BenchmarkConfig]
      ```
    </CodeBlock>
  </Anchor>

  <Indent>
    <Badge>
      classmethod
    </Badge>
  </Indent>

  <Anchor id="nemo_gym-benchmarks-BenchmarkConfig-from_initial_config_dict">
    <CodeBlock links={{"nemo_gym.benchmarks.BenchmarkConfig":"#nemo_gym-benchmarks-BenchmarkConfig"}} showLineNumbers={false} wordWrap={true}>
      ```python
      nemo_gym.benchmarks.BenchmarkConfig.from_initial_config_dict(
          path: pathlib.Path,
          initial_config_dict: omegaconf.DictConfig
      ) -> typing.Optional[nemo_gym.benchmarks.BenchmarkConfig]
      ```
    </CodeBlock>
  </Anchor>

  <Indent>
    <Badge>
      classmethod
    </Badge>
  </Indent>
</Indent>

<Anchor id="nemo_gym-benchmarks-PrepareBenchmarkConfig">
  <CodeBlock showLineNumbers={false} wordWrap={true}>
    ```python
    class nemo_gym.benchmarks.PrepareBenchmarkConfig()
    ```
  </CodeBlock>
</Anchor>

<Indent>
  **Bases:** [BaseNeMoGymCLIConfig](/nemo-gym/nemo_gym/config_types#nemo_gym-config_types-BaseNeMoGymCLIConfig)

  Prepare benchmark data by running the benchmark's prepare.py script.

  The benchmark is identified from a config\_paths entry pointing to a
  benchmarks/\*/config.yaml file.

  Examples:

  <CodeBlock showLineNumbers={false}>
    ```python
    ng_prepare_benchmark "+config_paths=[benchmarks/aime24/config.yaml]"
    ```
  </CodeBlock>

  <ParamField path="num_prepare_benchmark_processes" type="int" />

  <ParamField path="use_cached_prepared_benchmarks" type="bool" />
</Indent>

<Anchor id="nemo_gym-benchmarks-_load_benchmarks_from_config_paths">
  <CodeBlock links={{"nemo_gym.benchmarks.BenchmarkConfig":"#nemo_gym-benchmarks-BenchmarkConfig"}} showLineNumbers={false} wordWrap={true}>
    ```python
    nemo_gym.benchmarks._load_benchmarks_from_config_paths(
        config_paths: typing.List[pathlib.Path]
    ) -> typing.Dict[str, nemo_gym.benchmarks.BenchmarkConfig]
    ```
  </CodeBlock>
</Anchor>

<Indent />

<Anchor id="nemo_gym-benchmarks-_multiprocess_benchmark_prepare_fn">
  <CodeBlock showLineNumbers={false} wordWrap={true}>
    ```python
    nemo_gym.benchmarks._multiprocess_benchmark_prepare_fn(
        args
    )
    ```
  </CodeBlock>
</Anchor>

<Indent />

<Anchor id="nemo_gym-benchmarks-list_benchmarks">
  <CodeBlock showLineNumbers={false} wordWrap={true}>
    ```python
    nemo_gym.benchmarks.list_benchmarks() -> None
    ```
  </CodeBlock>
</Anchor>

<Indent>
  CLI command: list available benchmarks.
</Indent>

<Anchor id="nemo_gym-benchmarks-prepare_benchmark">
  <CodeBlock showLineNumbers={false} wordWrap={true}>
    ```python
    nemo_gym.benchmarks.prepare_benchmark() -> None
    ```
  </CodeBlock>
</Anchor>

<Indent>
  CLI command: prepare benchmark data.
</Indent>

<Anchor id="nemo_gym-benchmarks-BENCHMARKS_DIR">
  <CodeBlock showLineNumbers={false} wordWrap={true}>
    ```python
    nemo_gym.benchmarks.BENCHMARKS_DIR = PARENT_DIR / 'benchmarks'
    ```
  </CodeBlock>
</Anchor>