Create and Manage Evaluation Configurations

When you run an evaluation in NVIDIA NeMo Evaluator, you create a separate target and configuration for the evaluation.

Tip

Because NeMo Evaluator separates the target and the configuration, you can create a configuration once, and reuse it multiple times with different targets (for example, to compare models). To see what targets and configurations are supported together, refer to Job Target and Configuration Matrix.

---

Task Guides

Perform common evaluation configuration tasks.

Tip

The tutorials r/eference an EVALUATOR_BASE_URL whose value will depend on the ingress in your particular cluster. If you are using the minikube demo installation, it will be http://nemo.test. Otherwise, you will need to consult with your own cluster administrator for the ingress values.

Create Config

Create and submit a new evaluation configuration

Create an Evaluation Configuration

Delete Config

Delete an existing evaluation configuration

Delete a Config

References

Review detailed specifications for configuring evaluation configurations for different evaluation types and scenarios.

Config Schema

Reference documentation for the JSON schema used to define evaluation configurations

Evaluation Config Schema