Is this page helpful?

Job JSON Schema Reference#

When you create a job for an evaluation, you send a JSON data structure that contains the information for your configuration. Each job is uniquely identified by a job_id that the Evaluator service creates. For example, eval-1234ABCD5678EFGH.

Important

API Version Note: The job schema structure differs between v1 and v2 APIs. Select the appropriate tab below for your API version.

v1 (Current)

The following table contains selected field reference for the v1 API. For the full API reference, refer to Evaluator API.

Name	Description	Type
`config`	The evaluation configuration, either as a URN reference (string) or a full configuration object. For more information, refer to Create and Manage Evaluation Configurations.	String or Object
`id`	The ID of the job. The ID is returned in the response when you create a job.	String
`namespace`	An arbitrary organization name, a vendor name, or any other text. If you don’t specify a namespace, the default is `default`.	String
`progress`	The completion percentage of the evaluation job, calculated as samples processed divided by limit_samples. Only available for custom evaluation jobs or when limit_samples is set in the configuration.	Float
`samples_processed`	The number of dataset samples that have been processed during the evaluation. Updated in real-time as the job progresses.	Integer
`status_details`	Object containing detailed status information including progress tracking and task-level status.	Object
`target`	The evaluation target, either as a URN reference (string) or a full target object. For more information, refer to Create and Manage Evaluation Targets.	String or Object
`task_status`	A mapping of task names to their current status. Each task can be PENDING, RUNNING, COMPLETED, or FAILED.	Object

URNs are formatted as "<namespace>/<name>". E.g. "my-namespace/my-target-name"

Example v1 Job Structure (with URN references):

{
  "namespace": "my-organization",
  "target": "my-namespace/my-target-name",
  "config": "my-namespace/my-config-name"
}

Example v1 Job Structure (with full object definitions):

{
  "namespace": "my-organization",
  "target": {
    "name": "my-target-name",
    "namespace": "my-namespace",
    "type": "model",
    "model": {
      "api_endpoint": {
        "url": "http://localhost:8000/v1/chat/completions",
        "model_id": "meta/llama-3.1-8b-instruct"
      }
    }
  },
  "config": {
    "name": "my-config-name",
    "namespace": "my-namespace",
    "type": "bfclv3",
    "params": {
      "limit_samples": 10
    },
    "tasks": {
      "task1": {
        "type": "simple"
      }
    }
  }
}

You can also mix both styles (e.g., URN for config and full object for target).

v2 (Preview)

Warning

v2 API Preview: The v2 API is available for testing and feedback but is not yet recommended for production use. Breaking changes may occur before the stable release.

The v2 API introduces a spec envelope that wraps the target and config.

The following table contains selected field reference for the v2 API:

Name	Description	Type
`spec`	Required wrapper object containing the job specification (target and config).	Object
`spec.config`	The evaluation configuration, either as a URN reference (string) or a full configuration object. For more information, refer to Create and Manage Evaluation Configurations.	String or Object
`spec.target`	The evaluation target, either as a URN reference (string) or a full target object. For more information, refer to Create and Manage Evaluation Targets.	String or Object
`id`	The ID of the job. The ID is returned in the response when you create a job.	String
`namespace`	An arbitrary organization name, a vendor name, or any other text. If you don’t specify a namespace, the default is `default`.	String
`progress`	The completion percentage of the evaluation job, calculated as samples processed divided by limit_samples. Only available for custom evaluation jobs or when limit_samples is set in the configuration.	Float
`samples_processed`	The number of dataset samples that have been processed during the evaluation. Updated in real-time as the job progresses.	Integer
`status_details`	Object containing detailed status information including progress tracking and task-level status.	Object
`task_status`	A mapping of task names to their current status. Each task can be PENDING, RUNNING, COMPLETED, or FAILED.	Object

URNs are formatted as "<namespace>/<name>". E.g. "my-namespace/my-target-name"

Example v2 Job Structure (with URN references):

{
  "spec": {
    "target": "my-namespace/my-target-name",
    "config": "my-namespace/my-config-name"
  }
}

Example v2 Job Structure (with full object definitions):

{
  "spec": {
    "target": {
      "name": "my-target-name",
      "namespace": "my-namespace",
      "type": "model",
      "model": {
        "api_endpoint": {
          "url": "http://localhost:8000/v1/chat/completions",
          "model_id": "meta/llama-3.1-8b-instruct"
        }
      }
    },
    "config": {
      "name": "my-config-name",
      "namespace": "my-namespace",
      "type": "bfclv3",
      "params": {
        "limit_samples": 10
      },
      "tasks": {
        "task1": {
          "type": "simple"
        }
      }
    }
  }
}

You can also mix both styles (e.g., URN for config and full object for target).

For complete v2 examples and migration guidance, refer to: