nat.data_models.evaluate_config

YAML-backed evaluation configuration models (eval.*) for workflow configs.

Classes

JobEvictionPolicy	Policy for evicting old jobs when max_jobs is exceeded.
EvalCustomScriptConfig	Configuration for a custom script to run after the workflow and evaluation results are saved.
JobManagementConfig	Configuration for the job management of the evaluation.
EvalOutputConfig	Configuration for the output of the evaluation. This is specifiied in the eval.general.output
EvalGeneralConfig	Configuration for the general evaluation options. This is specifiied in the eval.general section
EvalConfig	Configuration for the evaluation options. This is specifiied in the eval section

Module Contents

class JobEvictionPolicy

Bases: enum.StrEnum

Policy for evicting old jobs when max_jobs is exceeded.

Initialize self. See help(type(self)) for accurate signature.

TIME_CREATED = 'time_created'

TIME_MODIFIED = 'time_modified'

class EvalCustomScriptConfig(/, **data: Any)

Bases: pydantic.BaseModel

Configuration for a custom script to run after the workflow and evaluation results are saved. This is specifiied in the eval.general.output.custom_scripts section of the evaluation configuration yaml file.

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

script: pathlib.Path = None

kwargs: dict[str, str] = None

class JobManagementConfig(/, **data: Any)

Bases: pydantic.BaseModel

Configuration for the job management of the evaluation. This is specifiied in the eval.general.output.job_management section of the evaluation configuration yaml file.

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

append_job_id_to_output_dir: bool = None

max_jobs: int = None

eviction_policy: JobEvictionPolicy = None

class EvalOutputConfig(/, **data: Any)

Bases: pydantic.BaseModel

Configuration for the output of the evaluation. This is specifiied in the eval.general.output section of the evaluation configuration yaml file.

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

dir: pathlib.Path = None

remote_dir: str | None = None

custom_pre_eval_process_function: str | None = None

custom_scripts: dict[str, EvalCustomScriptConfig] = None

s3: nat.data_models.dataset_handler.EvalS3Config | None = None

cleanup: bool = None

job_management: JobManagementConfig = None

workflow_output_step_filter: list[nat.data_models.intermediate_step.IntermediateStepType] | None = None

class EvalGeneralConfig(/, **data: Any)

Bases: pydantic.BaseModel

Configuration for the general evaluation options. This is specifiied in the eval.general section of the evaluation configuration yaml file.

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

max_concurrency: int = None

workflow_alias: str | None = None

output_dir: pathlib.Path = None

output: EvalOutputConfig | None = None

dataset: nat.data_models.dataset_handler.EvalDatasetBaseConfig | None = None

profiler: nat.data_models.profiler.ProfilerConfig | None = None

validate_llm_endpoints: bool = None

per_input_user_id: bool = None

classmethod override_output_dir(values)

classmethod rebuild_annotations()

class EvalConfig(/, **data: Any)

Bases: pydantic.BaseModel

Configuration for the evaluation options. This is specifiied in the eval section of the evaluation configuration yaml file.

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

general: EvalGeneralConfig = None

evaluators: dict[str, nat.data_models.evaluator.EvaluatorBaseConfig] = None

classmethod rebuild_annotations()