Add On-Demand Evaluation Packages#

This guide explains how to extend the NeMo evaluation environment by adding optional NVIDIA Eval Factory packages. It walks through installation, setup, and execution steps for various packages such as BFCL, garak, BigCode, simple-evals, and safety-harness, each enabling specialized model assessments.

The following diagram illustrates the architecture of the Eval ecosystem, showing how different evaluation harnesses integrate with NeMo Eval:

┌──────────────────────┐
│                      │
│lm-evaluation-harness ◄─────────┐
│                      │         │      ┌─────────────────────┐
└──────────────────────┘         │      │                     │
                                 │      │ NVIDIA Eval Factory │
┌──────────────────────┐         │      │ =================== │
│                      ◄─────────┼──────┼                     ◄───┐      ┌────────────────────┐
│     simple-evals     │    packaging   │ unified interface   │ model    │                    │
│                      │    of eval     │ for LLM evaluation  │ querying │     NeMo Eval      │
└──────────────────────┘    harnesses   │                     │   └──────┼ ================== │
                                 │      └─────────────────────┘          │                    │
  .                              │                                       │   LLM evaluation   │
  .                   ◄──────────┼                                ┌──────┼ with server-client │
  .                              │      ┌────────────────────┐  model    │      approach      │
                                 │      │                    │  serving  │                    │
                                 │      │ NeMo Export-Deploy │    │      └────────────────────┘
┌──────────────────────┐         │      │ ================== │    │
|                      │         │      │                    │    │
│        garak         ◄─────────┘      │  model deployment  ◄────┘
│                      │                │  for NeMo and HF   │
└──────────────────────┘                │                    │
                                        └────────────────────┘

Add New Evaluation Frameworks#

The NeMo Framework Docker image comes with nvidia-lm-eval pre-installed. However, you can add more evaluation methods by installing additional NVIDIA Eval Factory packages.

For each package, follow these steps:

  1. Install the required package.

  2. Deploy your model:

 1# File deploy.py
 2
 3from nemo_eval.api import deploy
 4
 5CHECKPOINT_PATH = "/checkpoints/llama-3_2-1b-instruct_v2.0"
 6
 7if __name__ == "__main__":
 8    deploy(
 9        nemo_checkpoint=CHECKPOINT_PATH,
10        max_input_len=8192,
11    )

Run the deployment in the background:

python deploy.py

Make sure to open two separate terminals within the same container for executing the deployment and evaluation.

  1. (Optional) Export the required environment variables.

  2. Run the evalution of your choice.

Below you can find examples for enabling and launching evaluations for different packages. Note that all example use only a subset of samples. To run the evaluation on the whole dataset, remove the "limit_samples" parameter.

Enable BFCL#

First, install the nvidia-bfcl package:

pip install nvidia-bfcl==25.6

  1. Run the evaluation:

 1from nemo_eval.api import evaluate
 2from nemo_eval.utils.api import EvaluationConfig, EvaluationTarget
 3
 4model_name = "megatron_model"
 5chat_url = "http://0.0.0.0:8080/v1/chat/completions/"
 6
 7
 8target_config = EvaluationTarget(
 9    api_endpoint={
10        "url": chat_url,
11        "type": "chat",
12    }
13)
14eval_config = EvaluationConfig(type="bfclv3_ast", output_dir="/results/", params={"limit_samples": 10})
15
16
17results = evaluate(target_cfg=target_config, eval_cfg=eval_config)
18
19
20print(results)

Enable garak#

  1. Install the nvidia-eval-factory-garak package:

pip install nvidia-eval-factory-garak==25.6

  1. Run the evaluation:

 1from nemo_eval.api import evaluate
 2from nemo_eval.utils.api import EvaluationConfig, EvaluationTarget
 3
 4model_name = "megatron_model"
 5chat_url = "http://0.0.0.0:8080/v1/chat/completions/"
 6
 7
 8target_config = EvaluationTarget(
 9    api_endpoint={
10        "url": chat_url,
11        "type": "chat",
12    }
13)
14eval_config = EvaluationConfig(
15    type="garak",
16    output_dir="/results/",
17    params={"extra": {"probes": "ansiescape.AnsiEscaped"}},
18)
19
20
21results = evaluate(target_cfg=target_config, eval_cfg=eval_config)
22
23
24print(results)

Enable BigCode#

  1. Install the nvidia-bigcode-eval package:

pip install nvidia-bigcode-eval==25.6

  1. Run the evaluation:

 1from nemo_eval.api import evaluate
 2from nemo_eval.utils.api import EvaluationConfig, EvaluationTarget
 3
 4model_name = "megatron_model"
 5chat_url = "http://0.0.0.0:8080/v1/chat/completions/"
 6
 7
 8target_config = EvaluationTarget(
 9    api_endpoint={
10        "url": chat_url,
11        "type": "chat",
12    }
13)
14eval_config = EvaluationConfig(type="mbpp", output_dir="/results/", params={"limit_samples": 10})
15
16
17results = evaluate(target_cfg=target_config, eval_cfg=eval_config)
18
19
20print(results)

Enable simple-evals#

  1. Install the nvidia-simple-evals package:

pip install nvidia-simple-evals==25.6

In the example below, we use the AIME_2025 task, which follows the llm-as-a-judge approach for checking the output correctness. By default, Llama 3.3 70B NVIDIA NIM is used for judging.

  1. To run evaluation, set your build.nvidia.com API key as the JUDGE_API_KEY variable:

export JUDGE_API_KEY=...

To customize the judge setting, see the instructions for NVIDIA Eval Factory package.

  1. Run the evaluation:

 1from nemo_eval.api import evaluate
 2from nemo_eval.utils.api import EvaluationConfig, EvaluationTarget
 3
 4model_name = "megatron_model"
 5chat_url = "http://0.0.0.0:8080/v1/chat/completions/"
 6
 7
 8target_config = EvaluationTarget(
 9    api_endpoint={
10        "url": chat_url,
11        "type": "chat",
12    }
13)
14eval_config = EvaluationConfig(
15    type="AIME_2025",
16    output_dir="/results/",
17    params={"limit_samples": 10},
18)
19
20
21results = evaluate(target_cfg=target_config, eval_cfg=eval_config)
22
23
24print(results)

Enable safety-harness#

  1. Install the nvidia-safety-harness package:

pip install nvidia-safety-harness==25.6

  1. Deploy the judge model

In the example below, we use the aegis_v2 task, which requires the Llama 3.1 NemoGuard 8B ContentSafety model to assess your model’s responses.

The model is available through NVIDIA NIM. See the instructions on deploying the judge model.

If you set a gated judge endpoint up, you must export your API key as the JUDGE_API_KEY variable:

export JUDGE_API_KEY=...

  1. To access the evaluation dataset, you must authenticate with the Hugging Face Hub.

  2. Run the evaluation:

 1from nemo_eval.api import evaluate
 2from nemo_eval.utils.api import EvaluationConfig, EvaluationTarget
 3
 4model_name = "megatron_model"
 5chat_url = "http://0.0.0.0:8080/v1/chat/completions/"
 6
 7
 8target_config = EvaluationTarget(
 9    api_endpoint={
10        "url": chat_url,
11        "type": "chat",
12    }
13)
14eval_config = EvaluationConfig(
15    type="aegis_v2",
16    output_dir="/results/",
17    params={
18        "limit_samples": 10,
19        "extra": {
20            "judge": {
21                "model_id": "llama-nemotron-safety-guard-v2",
22                "url": "http://0.0.0.0:9000/v1/completions",
23            }
24        },
25    },
26)
27
28
29results = evaluate(target_cfg=target_config, eval_cfg=eval_config)
30
31
32print(results)

Make sure to modify the judge configuration in the provided snippet to match your Llama 3.1 NemoGuard 8B ContentSafety endoint:

    params={
        "extra": {
            "judge": {
                "model_id": "my-llama-3.1-nemoguard-8b-content-safety-endpoint",
                "url": "http://my-hostname:1234/v1/completions",
            }
        }
    }