Job Target and Configuration Matrix

Because NeMo Evaluator separates the target and the configuration, you can create a configuration once, and reuse it multiple times with different targets. You can also create a target once, and reuse it multiple times with different evaluations. The following table gives examples of how to combine targets and configurations for an evaluation job.

Evaluation Job Combinations

Evaluation Type	Example Configurations	Example Targets	Custom Data Options
BigCode Evaluation Harness	Config	LLM Model	—
LM Evaluation Harness	Config	LLM Model	—
Similarity Metrics	Config	LLM Model Pre-generated Answers	Custom Data
LLM As A Judge	Config	LLM Model Pre-generated Answers	—
Retriever Pipeline	Config (Embedding + Reranking) Config (Custom Data)	Target (Embedding Only) Target (Embedding + Reranking)	BEIR, SQuAD
RAG Pipeline	Config (Standard Data) Config (Custom Data) Config (Pre-generated Answers) Config (OpenAI-compatible Judge LLM)	Target (Embedding only) Target (Embedding + Reranking) Target Answer Evaluation Target Answer Generation + Answer Evaluation	BEIR, SQuAD, Ragas

Caution

Before you can run a retriever evaluation, or a RAG evaluation that has a retriever step, you must set up a Milvus document store. For more information, refer to Milvus.