nat.data_models.dataset_handler

Attributes

EvalDatasetConfig

Classes

EvalS3Config
EvalFilterEntryConfig
EvalFilterConfig
EvalDatasetStructureConfig
EvalDatasetBaseConfig	Subclass of Pydantic BaseModel that allows for specifying the object type. Use in Pydantic discriminated unions.
EvalDatasetJsonConfig	Subclass of Pydantic BaseModel that allows for specifying the object type. Use in Pydantic discriminated unions.
EvalDatasetJsonlConfig	Subclass of Pydantic BaseModel that allows for specifying the object type. Use in Pydantic discriminated unions.
EvalDatasetCsvConfig	Subclass of Pydantic BaseModel that allows for specifying the object type. Use in Pydantic discriminated unions.
EvalDatasetParquetConfig	Subclass of Pydantic BaseModel that allows for specifying the object type. Use in Pydantic discriminated unions.
EvalDatasetXlsConfig	Subclass of Pydantic BaseModel that allows for specifying the object type. Use in Pydantic discriminated unions.
EvalDatasetCustomConfig	Configuration for custom dataset type that allows users to specify

Functions

read_jsonl(file_path)

Module Contents

class EvalS3Config(/, **data: Any)

Bases: pydantic.BaseModel

endpoint_url: str | None = None

region_name: str | None = None

bucket: str

access_key: str

secret_key: str

class EvalFilterEntryConfig(/, **data: Any)

Bases: pydantic.BaseModel

field: dict[str, list[str | int | float]]

class EvalFilterConfig(/, **data: Any)

Bases: pydantic.BaseModel

allowlist: EvalFilterEntryConfig | None = None

denylist: EvalFilterEntryConfig | None = None

class EvalDatasetStructureConfig(/, **data: Any)

Bases: pydantic.BaseModel

disable: bool = False

question_key: str = 'question'

answer_key: str = 'answer'

generated_answer_key: str = 'generated_answer'

trajectory_key: str = 'intermediate_steps'

expected_trajectory_key: str = 'expected_intermediate_steps'

class EvalDatasetBaseConfig(/, **data: Any)

Bases: nat.data_models.common.TypedBaseModel, nat.data_models.common.BaseModelRegistryTag

Subclass of Pydantic BaseModel that allows for specifying the object type. Use in Pydantic discriminated unions.

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

id_key: str = 'id'

structure: EvalDatasetStructureConfig

filter: EvalFilterConfig | None

s3: EvalS3Config | None = None

remote_file_path: str | None = None

file_path: pathlib.Path | str

class EvalDatasetJsonConfig(/, **data: Any)

Bases: EvalDatasetBaseConfig

Subclass of Pydantic BaseModel that allows for specifying the object type. Use in Pydantic discriminated unions.

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

static parser() → tuple[collections.abc.Callable, dict]

read_jsonl(file_path: pydantic.FilePath)

class EvalDatasetJsonlConfig(/, **data: Any)

Bases: EvalDatasetBaseConfig

Subclass of Pydantic BaseModel that allows for specifying the object type. Use in Pydantic discriminated unions.

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

static parser() → tuple[collections.abc.Callable, dict]

class EvalDatasetCsvConfig(/, **data: Any)

Bases: EvalDatasetBaseConfig

Subclass of Pydantic BaseModel that allows for specifying the object type. Use in Pydantic discriminated unions.

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

static parser() → tuple[collections.abc.Callable, dict]

class EvalDatasetParquetConfig(/, **data: Any)

Bases: EvalDatasetBaseConfig

Subclass of Pydantic BaseModel that allows for specifying the object type. Use in Pydantic discriminated unions.

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

static parser() → tuple[collections.abc.Callable, dict]

class EvalDatasetXlsConfig(/, **data: Any)

Bases: EvalDatasetBaseConfig

Subclass of Pydantic BaseModel that allows for specifying the object type. Use in Pydantic discriminated unions.

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

static parser() → tuple[collections.abc.Callable, dict]

class EvalDatasetCustomConfig(/, **data: Any)

Bases: EvalDatasetBaseConfig

Configuration for custom dataset type that allows users to specify a custom Python function to transform their dataset into EvalInput format.

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

function: str

kwargs: dict[str, Any]

parser() → tuple[collections.abc.Callable, dict]

Load and return the custom function for dataset transformation.

Returns:

Tuple of (custom_function, kwargs) where custom_function transforms a dataset file into an EvalInput object.

_load_custom_function() → collections.abc.Callable

Import and return the custom function using standard Python import path.

EvalDatasetConfig