Create and Manage Evaluation Configurations

When you run an evaluation in NVIDIA NeMo Evaluator, you create a separate target and configuration for the evaluation.

Tip

Because NeMo Evaluator separates the target and the configuration, you can create a configuration once, and reuse it multiple times with different targets (for example, to compare models). To see what targets and configurations are supported together, refer to Job Target and Configuration Matrix.

---

Task Guides

Perform common evaluation configuration tasks.

Create Config

Create and submit a new evaluation configuration

Create an Evaluation Configuration

Delete Config

Delete an existing evaluation configuration

Delete a Config

References

Review detailed specifications for configuring evaluation configurations for different evaluation types and scenarios.

Passing System Prompts

Reference documentation for the JSON schema used to define evaluation configurations

Passing System Prompts for Advanced Reasoning

Config Schema

Reference documentation for the JSON schema used to define evaluation configurations

Evaluation Config Schema