Data for Custom Evaluations#

This section describes how to prepare and format datasets for custom evaluation jobs.

Input Dataset Requirements#

The input dataset for the task is configured at the task level. This dataset can be directly specified within the configuration of the evaluation job.

You must ensure all fields that you intend to use for prompt and scoring templates have been added to your dataset before using it in a custom evaluation.

Supported Formats#

The input dataset can be formatted in the following ways:

  • CSV: Comma-separated values format (not supported for custom tool calling evaluation).

  • JSON: An array of JSON objects (default format).

  • JSONL: JSON Lines format, where each line is a separate JSON object.

Example CSV Dataset#

"question","answer","reference_answer"
"What is 2+2?","4","The answer is 4"
"Square root of 256?","16","The answer is 16"
"What power of 2 is 1024?","10","The answer is 10"

After you have formatted your file and verified that it includes all the fields you intend to use, save the file and upload it to your dataset into NeMo Data Store. Follow the steps at Using Custom Data.