RAG Pipeline Targets#
Retrieval Augmented Generation (RAG) pipelines are built by combining NeMo Retriever and LLM. A retriever pipeline is used to retrieve relevant documents based on a query, and the LLM is used to generate answers based on the query and the retrieved documents. For more information, refer to RAG Evaluation Type.
Answer Evaluation#
NeMo Evaluator supports Answer Evaluation RAG pipelines. The rag pipeline is replaced by a cached_outputs field that contains pre-generated retrieved documents and pre-generated answers. For more information, refer to Using Custom Data.
curl -X "POST" "${EVALUATOR_SERVICE_URL}/v1/evaluation/targets" \
-H 'accept: application/json' \
-H 'Content-Type: application/json' \
-d '
{
"type": "rag",
"name": "my-target-rag-1",
"namespace": "my-organization",
"rag": {
"cached_outputs": {
"files_url": "hf://datasets/<my-dataset-namespace>/<my-dataset-name>/<my-dataset-file-path>"
}
}
}'
data = {
"type": "rag",
"name": "my-target-rag-1",
"namespace": "my-organization",
"rag": {
"cached_outputs": {
"files_url": "hf://datasets/<my-dataset-namespace>/<my-dataset-name>/<my-dataset-file-path>"
}
}
}
endpoint = f"{EVALUATOR_SERVICE_URL}/v1/evaluation/targets"
response = requests.post(endpoint, json=data).json()
Answer Generation and Evaluation#
NeMo Evaluator supports Answer Generation + Answer Evaluation RAG pipelines. The retriever pipeline is replaced by a cached_outputs field that contains pre-generated retrieved documents. For more information, refer to Using Custom Data.
curl -X "POST" "${EVALUATOR_SERVICE_URL}/v1/evaluation/targets" \
-H 'accept: application/json' \
-H 'Content-Type: application/json' \
-d '
{
"type": "rag",
"name": "my-target-rag-2",
"namespace": "my-organization",
"rag": {
"pipeline": {
"retriever": {
"cached_outputs": {
"files_url": "hf://datasets/<my-dataset-namespace>/<my-dataset-name>/<my-dataset-file-path>"
}
},
"model": {
"api_endpoint": {
"url": "<my-nim-deployment-base-url>/chat/completions",
"model_id": "<my-model>"
}
}
}
}
}'
data = {
"type": "rag",
"name": "my-target-rag-2",
"namespace": "my-organization",
"rag": {
"pipeline": {
"retriever": {
"cached_outputs": {
"files_url": "hf://datasets/<my-dataset-namespace>/<my-dataset-name>/<my-dataset-file-path>"
}
},
"model": {
"api_endpoint": {
"url": "<my-nim-deployment-base-url>/chat/completions",
"model_id": "<my-model>"
}
}
}
}
}
endpoint = f"{EVALUATOR_SERVICE_URL}/v1/evaluation/targets"
response = requests.post(endpoint, json=data).json()
Retrieval#
Embedding + Answer Generation and Evaluation#
NeMo Evaluator supports Retrieval + Answer Generation + Answer Evaluation RAG pipelines. The rag pipeline field contains a retriever pipeline and a model.
curl -X "POST" "${EVALUATOR_SERVICE_URL}/v1/evaluation/targets" \
-H 'accept: application/json' \
-H 'Content-Type: application/json' \
-d '
{
"type": "rag",
"name": "my-target-rag-3",
"namespace": "my-organization",
"rag": {
"pipeline": {
"retriever": {
"pipeline": {
"query_embedding_model": {
"api_endpoint": {
"url": "<my-query-embedding-url>",
"model_id": "<my-query-embedding-model>"
}
},
"index_embedding_model": {
"api_endpoint": {
"url": "<my-index-embedding-url>",
"model_id": "<my-index-embedding-model>"
}
},
"top_k": 3
}
},
"model": {
"api_endpoint": {
"url": "<my-nim-deployment-base-url>/chat/completions",
"model_id": "<my-model>"
}
}
}
}
}'
data = {
"type": "rag",
"name": "my-target-rag-3",
"namespace": "my-organization",
"rag": {
"pipeline": {
"retriever": {
"pipeline": {
"query_embedding_model": {
"api_endpoint": {
"url": "<my-query-embedding-url>",
"model_id": "<my-query-embedding-model>"
}
},
"index_embedding_model": {
"api_endpoint": {
"url": "<my-index-embedding-url>",
"model_id": "<my-index-embedding-model>"
}
},
"top_k": 3
}
},
"model": {
"api_endpoint": {
"url": "<my-nim-deployment-base-url>/chat/completions",
"model_id": "<my-model>"
}
}
}
}
}
endpoint = f"{EVALUATOR_SERVICE_URL}/v1/evaluation/targets"
response = requests.post(endpoint, json=data).json()
Embedding, Reranking + Answer Generation and Evaluation#
NeMo Evaluator supports Retrieval + Answer Generation + Answer Evaluation RAG pipelines. The rag pipeline field contains a retriever pipeline and a model.
curl -X "POST" "${EVALUATOR_SERVICE_URL}/v1/evaluation/targets" \
-H 'accept: application/json' \
-H 'Content-Type: application/json' \
-d '
{
"type": "rag",
"name": "my-target-rag-4",
"namespace": "my-organization",
"rag": {
"pipeline": {
"retriever": {
"pipeline": {
"query_embedding_model": {
"api_endpoint": {
"url": "<my-query-embedding-url>",
"model_id": "<my-query-embedding-model>"
}
},
"index_embedding_model": {
"api_endpoint": {
"url": "<my-index-embedding-url>",
"model_id": "<my-index-embedding-model>"
}
},
"reranker_model": {
"api_endpoint": {
"url": "<my-ranker-url>",
"model_id": "<my-ranker-model>"
}
},
"top_k": 3
}
},
"model": {
"api_endpoint": {
"url": "<my-nim-deployment-base-url>/chat/completions",
"model_id": "<my-model>"
}
}
}
}
}'
data = {
"type": "rag",
"name": "my-target-rag-4",
"namespace": "my-organization",
"rag": {
"pipeline": {
"retriever": {
"pipeline": {
"query_embedding_model": {
"api_endpoint": {
"url": "<my-query-embedding-url>",
"model_id": "<my-query-embedding-model>"
}
},
"index_embedding_model": {
"api_endpoint": {
"url": "<my-index-embedding-url>",
"model_id": "<my-index-embedding-model>"
}
},
"reranker_model": {
"api_endpoint": {
"url": "<my-ranker-url>",
"model_id": "<my-ranker-model>"
}
},
"top_k": 3
}
},
"model": {
"api_endpoint": {
"url": "<my-nim-deployment-base-url>/chat/completions",
"model_id": "<my-model>"
}
}
}
}
}
endpoint = f"{EVALUATOR_SERVICE_URL}/v1/evaluation/targets"
response = requests.post(endpoint, json=data).json()