Framework Definition File (FDF)#

Framework Definition Files are YAML configuration files that integrate evaluation frameworks into NeMo Evaluator. They define framework metadata, execution commands, and evaluation tasks.

New to FDFs? Learn about the concepts and architecture before creating one.

Prerequisites#

Before creating an FDF, you should:

Understand YAML syntax and structure
Be familiar with your evaluation framework’s CLI interface
Have basic knowledge of Jinja2 templating
Know the API endpoint types your framework supports

Getting Started#

Creating your first FDF? Follow this sequence:

Framework Section - Define framework metadata
Defaults Section - Configure command templates and parameters
Evaluations Section - Define evaluation tasks
Integration with Eval Factory - Integrate with Eval Factory

Need help? Refer to Troubleshooting for debugging common issues.

Complete Example#

The FDF follows a hierarchical structure with three main sections. Here’s a minimal but complete example:

# 1. Framework Identification
framework:
  name: my-custom-eval
  pkg_name: my_custom_eval
  full_name: My Custom Evaluation Framework
  description: Evaluates domain-specific capabilities
  url: https://github.com/example/my-eval

# 2. Default Command and Configuration
defaults:
  command: >-
    {% if target.api_endpoint.api_key is not none %}export API_KEY=${{target.api_endpoint.api_key}} && {% endif %}
    my-eval-cli --model {{target.api_endpoint.model_id}} 
                --task {{config.params.task}}
                --output {{config.output_dir}}
  
  config:
    params:
      temperature: 0.0
      max_new_tokens: 1024
  
  target:
    api_endpoint:
      type: chat
      supported_endpoint_types:
        - chat
        - completions

# 3. Evaluation Types
evaluations:
  - name: my_task_1
    description: First evaluation task
    defaults:
      config:
        type: my_task_1
        supported_endpoint_types:
          - chat
        params:
          task: my_task_1