Templating for Tasks

This section explains how to use Jinja2 templates for prompts and tasks in custom evaluation jobs.

Available Template Objects

When rendering templates, two default objects are available:

• item: Represents the current item from the dataset.

• sample: Contains data related to the output from the model. The sample.output_text represents the completion text for completion models and the content of the first message for chat models.

The properties on the item object are derived from the dataset’s column names (for CSVs) or keys (for JSONs):

• All non-alphanumeric characters are replaced with underscores.

• Column names are converted to lowercase.

• In case of conflicts, suffixes (_1, _2, etc.), are appended to the property names.

Templates for Chat Models

Prompt templates are used to structure tasks for evaluating the performance of models, specifically following the NIM/OpenAI format for chat-completion tasks. Templates use the Jinja2 templating syntax. Variables are represented using double-curly brackets, for example, {{item.review}}.

Example Template for Chat-Completion Task

{
    "messages": [{
        "role": "system",
        "content": "You are an expert in analyzing the sentiment of movie reviews."
    }, {
        "role": "user",
        "content": "Determine if the following review is positive or negative: {{item.review}}"
    }]
}

Simple Chat Templating

If your custom data is structured as prompt and ideal_response, you can structure this as a single-turn chat.

{ 
    "messages": [{
        "role": "system", 
        "content": "You are an expert in analyzing the sentiment of movie reviews."
    }, { 
        "role": "user", 
        "content": "Determine if the following review is positive or negative: {{item.prompt}}"
    }] 
} 

You can include this in a call to a /chat/completion endpoint.

   {
    "config": {
      "type": "custom",
      "tasks": {
        "qa": {
          "type": "completion",
          "params": {
            "template": {
              "messages": [{
                "role": "system",
                "content": "You are a helpful, respectful and honest assistant. \nExtract from the following context the minimal span word for word that best answers the question.\n."
              }, { 
                "role": "user",
                "content": "Context: {{item.prompt}}"
              }] 
            }
          },
          "metrics": {
            "accuracy": {
              "type": "string-check",
              "params": {
                "check": [
                  "{{sample.output_text}}",
                  "contains",
                  "{{item.ideal_response}}"
                ]
              }
            }
          },
          "dataset": {
            "files_url": "hf://datasets/<my-dataset-namespace>/<my-dataset-name>"
          }
        }
      }
    },
    "target": {
      "type": "model",
      "model": {
        "api_endpoint": {
          "url": "<my-nim-url>/chat/completions",
          "model_id": "<my-model-id>"
        }
      }
    }
  }

Messages Data Template

If your custom data is already formatted as JSON, you can configure your template similar to the following:

{
    "messages": "{{ item.messages | tojson }}"
}