`nemo_evaluator.api.api_dataclasses`#

Module Contents#

Classes#

`ApiEndpoint`	API endpoint configuration.
`ConfigParams`	Parameters for evaluation execution.
`EndpointType`	str(object=’’) -> str str(bytes_or_buffer[, encoding[, errors]]) -> str
`Evaluation`	!!! abstract “Usage Documentation” Models
`EvaluationConfig`	Configuration for evaluation runs.
`EvaluationResult`	!!! abstract “Usage Documentation” Models
`EvaluationTarget`	Target configuration for API endpoints.
`GroupResult`	The evaluation results for a group.
`MetricResult`	!!! abstract “Usage Documentation” Models
`Score`	!!! abstract “Usage Documentation” Models
`ScoreStats`	Stats for a score.
`TaskResult`	!!! abstract “Usage Documentation” Models

API#

class nemo_evaluator.api.api_dataclasses.ApiEndpoint(/, **data: Any)[source]#

Bases: pydantic.BaseModel

API endpoint configuration.

Initialization

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

adapter_config: nemo_evaluator.adapters.adapter_config.AdapterConfig | None#: ‘Field(…)’

api_key: str | None#: ‘Field(…)’

model_config#: ‘ConfigDict(…)’

model_id: str | None#: ‘Field(…)’

stream: bool | None#: ‘Field(…)’

type: nemo_evaluator.api.api_dataclasses.EndpointType | None#: ‘Field(…)’

url: str | None#: ‘Field(…)’

class nemo_evaluator.api.api_dataclasses.ConfigParams(/, **data: Any)[source]#

Bases: pydantic.BaseModel

Parameters for evaluation execution.

Initialization

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

extra: Dict[str, Any] | None#: ‘Field(…)’

limit_samples: int | float | None#: ‘Field(…)’

max_new_tokens: int | None#: ‘Field(…)’

max_retries: int | None#: ‘Field(…)’

parallelism: int | None#: ‘Field(…)’

request_timeout: int | None#: ‘Field(…)’

task: str | None#: ‘Field(…)’

temperature: float | None#: ‘Field(…)’

top_p: float | None#: ‘Field(…)’

class nemo_evaluator.api.api_dataclasses.EndpointType[source]#

Bases: str, enum.Enum

str(object=’’) -> str str(bytes_or_buffer[, encoding[, errors]]) -> str

Create a new string object from the given object. If encoding or errors is specified, then the object must expose a data buffer that will be decoded using the given encoding and error handler. Otherwise, returns the result of object.str() (if defined) or repr(object). encoding defaults to sys.getdefaultencoding(). errors defaults to ‘strict’.

Initialization

Initialize self. See help(type(self)) for accurate signature.

CHAT#: ‘chat’

COMPLETIONS#: ‘completions’

EMBEDDING#: ‘embedding’

UNDEFINED#: ‘undefined’

VLM#: ‘vlm’

class nemo_evaluator.api.api_dataclasses.Evaluation(/, **data: Any)[source]#

Bases: pydantic.BaseModel

!!! abstract “Usage Documentation” Models

A base class for creating Pydantic models.

Attributes: class_vars: The names of the class variables defined on the model. private_attributes: Metadata about the private attributes of the model. signature: The synthesized __init__ [Signature][inspect.Signature] of the model.

__pydantic_complete__: Whether model building is completed, or if there are still undefined fields.
__pydantic_core_schema__: The core schema of the model.
__pydantic_custom_init__: Whether the model has a custom `__init__` function.
__pydantic_decorators__: Metadata containing the decorators defined on the model.
    This replaces `Model.__validators__` and `Model.__root_validators__` from Pydantic V1.
__pydantic_generic_metadata__: Metadata for generic models; contains data used for a similar purpose to
    __args__, __origin__, __parameters__ in typing-module generics. May eventually be replaced by these.
__pydantic_parent_namespace__: Parent namespace of the model, used for automatic rebuilding of models.
__pydantic_post_init__: The name of the post-init method for the model, if defined.
__pydantic_root_model__: Whether the model is a [`RootModel`][pydantic.root_model.RootModel].
__pydantic_serializer__: The `pydantic-core` `SchemaSerializer` used to dump instances of the model.
__pydantic_validator__: The `pydantic-core` `SchemaValidator` used to validate instances of the model.

__pydantic_fields__: A dictionary of field names and their corresponding [`FieldInfo`][pydantic.fields.FieldInfo] objects.
__pydantic_computed_fields__: A dictionary of computed field names and their corresponding [`ComputedFieldInfo`][pydantic.fields.ComputedFieldInfo] objects.

__pydantic_extra__: A dictionary containing extra values, if [`extra`][pydantic.config.ConfigDict.extra]
    is set to `'allow'`.
__pydantic_fields_set__: The names of fields explicitly set during instantiation.
__pydantic_private__: Values of private attributes set on the model instance.

Initialization

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

command: str#: ‘Field(…)’

config: nemo_evaluator.api.api_dataclasses.EvaluationConfig#: None

framework_name: str#: ‘Field(…)’

pkg_name: str#: ‘Field(…)’

render_command()[source]#

target: nemo_evaluator.api.api_dataclasses.EvaluationTarget#: None

class nemo_evaluator.api.api_dataclasses.EvaluationConfig(/, **data: Any)[source]#

Bases: pydantic.BaseModel

Configuration for evaluation runs.

Initialization

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

output_dir: str | None#: ‘Field(…)’

params: nemo_evaluator.api.api_dataclasses.ConfigParams | None#: ‘Field(…)’

supported_endpoint_types: list[str] | None#: ‘Field(…)’

type: str | None#: ‘Field(…)’

class nemo_evaluator.api.api_dataclasses.EvaluationResult(/, **data: Any)[source]#

Bases: pydantic.BaseModel

!!! abstract “Usage Documentation” Models

A base class for creating Pydantic models.

Attributes: class_vars: The names of the class variables defined on the model. private_attributes: Metadata about the private attributes of the model. signature: The synthesized __init__ [Signature][inspect.Signature] of the model.

__pydantic_complete__: Whether model building is completed, or if there are still undefined fields.
__pydantic_core_schema__: The core schema of the model.
__pydantic_custom_init__: Whether the model has a custom `__init__` function.
__pydantic_decorators__: Metadata containing the decorators defined on the model.
    This replaces `Model.__validators__` and `Model.__root_validators__` from Pydantic V1.
__pydantic_generic_metadata__: Metadata for generic models; contains data used for a similar purpose to
    __args__, __origin__, __parameters__ in typing-module generics. May eventually be replaced by these.
__pydantic_parent_namespace__: Parent namespace of the model, used for automatic rebuilding of models.
__pydantic_post_init__: The name of the post-init method for the model, if defined.
__pydantic_root_model__: Whether the model is a [`RootModel`][pydantic.root_model.RootModel].
__pydantic_serializer__: The `pydantic-core` `SchemaSerializer` used to dump instances of the model.
__pydantic_validator__: The `pydantic-core` `SchemaValidator` used to validate instances of the model.

__pydantic_fields__: A dictionary of field names and their corresponding [`FieldInfo`][pydantic.fields.FieldInfo] objects.
__pydantic_computed_fields__: A dictionary of computed field names and their corresponding [`ComputedFieldInfo`][pydantic.fields.ComputedFieldInfo] objects.

__pydantic_extra__: A dictionary containing extra values, if [`extra`][pydantic.config.ConfigDict.extra]
    is set to `'allow'`.
__pydantic_fields_set__: The names of fields explicitly set during instantiation.
__pydantic_private__: Values of private attributes set on the model instance.

Initialization

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

groups: Dict[str, nemo_evaluator.api.api_dataclasses.GroupResult] | None#: ‘Field(…)’

tasks: Dict[str, nemo_evaluator.api.api_dataclasses.TaskResult] | None#: ‘Field(…)’

class nemo_evaluator.api.api_dataclasses.EvaluationTarget(/, **data: Any)[source]#

Bases: pydantic.BaseModel

Target configuration for API endpoints.

Initialization

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

api_endpoint: nemo_evaluator.api.api_dataclasses.ApiEndpoint | None#: ‘Field(…)’

class nemo_evaluator.api.api_dataclasses.GroupResult(/, **data: Any)[source]#

Bases: pydantic.BaseModel

The evaluation results for a group.

Initialization

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

groups: Dict[str, nemo_evaluator.api.api_dataclasses.GroupResult] | None#: ‘Field(…)’

metrics: Dict[str, nemo_evaluator.api.api_dataclasses.MetricResult]#: ‘Field(…)’

class nemo_evaluator.api.api_dataclasses.MetricResult(/, **data: Any)[source]#

Bases: pydantic.BaseModel

!!! abstract “Usage Documentation” Models

A base class for creating Pydantic models.

Attributes: class_vars: The names of the class variables defined on the model. private_attributes: Metadata about the private attributes of the model. signature: The synthesized __init__ [Signature][inspect.Signature] of the model.

__pydantic_complete__: Whether model building is completed, or if there are still undefined fields.
__pydantic_core_schema__: The core schema of the model.
__pydantic_custom_init__: Whether the model has a custom `__init__` function.
__pydantic_decorators__: Metadata containing the decorators defined on the model.
    This replaces `Model.__validators__` and `Model.__root_validators__` from Pydantic V1.
__pydantic_generic_metadata__: Metadata for generic models; contains data used for a similar purpose to
    __args__, __origin__, __parameters__ in typing-module generics. May eventually be replaced by these.
__pydantic_parent_namespace__: Parent namespace of the model, used for automatic rebuilding of models.
__pydantic_post_init__: The name of the post-init method for the model, if defined.
__pydantic_root_model__: Whether the model is a [`RootModel`][pydantic.root_model.RootModel].
__pydantic_serializer__: The `pydantic-core` `SchemaSerializer` used to dump instances of the model.
__pydantic_validator__: The `pydantic-core` `SchemaValidator` used to validate instances of the model.

__pydantic_fields__: A dictionary of field names and their corresponding [`FieldInfo`][pydantic.fields.FieldInfo] objects.
__pydantic_computed_fields__: A dictionary of computed field names and their corresponding [`ComputedFieldInfo`][pydantic.fields.ComputedFieldInfo] objects.

__pydantic_extra__: A dictionary containing extra values, if [`extra`][pydantic.config.ConfigDict.extra]
    is set to `'allow'`.
__pydantic_fields_set__: The names of fields explicitly set during instantiation.
__pydantic_private__: Values of private attributes set on the model instance.

Initialization

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

scores: Dict[str, nemo_evaluator.api.api_dataclasses.Score]#: ‘Field(…)’

class nemo_evaluator.api.api_dataclasses.Score(/, **data: Any)[source]#

Bases: pydantic.BaseModel

!!! abstract “Usage Documentation” Models

A base class for creating Pydantic models.

Attributes: class_vars: The names of the class variables defined on the model. private_attributes: Metadata about the private attributes of the model. signature: The synthesized __init__ [Signature][inspect.Signature] of the model.

__pydantic_complete__: Whether model building is completed, or if there are still undefined fields.
__pydantic_core_schema__: The core schema of the model.
__pydantic_custom_init__: Whether the model has a custom `__init__` function.
__pydantic_decorators__: Metadata containing the decorators defined on the model.
    This replaces `Model.__validators__` and `Model.__root_validators__` from Pydantic V1.
__pydantic_generic_metadata__: Metadata for generic models; contains data used for a similar purpose to
    __args__, __origin__, __parameters__ in typing-module generics. May eventually be replaced by these.
__pydantic_parent_namespace__: Parent namespace of the model, used for automatic rebuilding of models.
__pydantic_post_init__: The name of the post-init method for the model, if defined.
__pydantic_root_model__: Whether the model is a [`RootModel`][pydantic.root_model.RootModel].
__pydantic_serializer__: The `pydantic-core` `SchemaSerializer` used to dump instances of the model.
__pydantic_validator__: The `pydantic-core` `SchemaValidator` used to validate instances of the model.

__pydantic_fields__: A dictionary of field names and their corresponding [`FieldInfo`][pydantic.fields.FieldInfo] objects.
__pydantic_computed_fields__: A dictionary of computed field names and their corresponding [`ComputedFieldInfo`][pydantic.fields.ComputedFieldInfo] objects.

__pydantic_extra__: A dictionary containing extra values, if [`extra`][pydantic.config.ConfigDict.extra]
    is set to `'allow'`.
__pydantic_fields_set__: The names of fields explicitly set during instantiation.
__pydantic_private__: Values of private attributes set on the model instance.

Initialization

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

stats: nemo_evaluator.api.api_dataclasses.ScoreStats#: ‘Field(…)’

value: float#: ‘Field(…)’

class nemo_evaluator.api.api_dataclasses.ScoreStats(/, **data: Any)[source]#

Bases: pydantic.BaseModel

Stats for a score.

Initialization

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

count: int | None#: ‘Field(…)’

max: float | None#: ‘Field(…)’

mean: float | None#: ‘Field(…)’

min: float | None#: ‘Field(…)’

stddev: float | None#: ‘Field(…)’

stderr: float | None#: ‘Field(…)’

sum: float | None#: ‘Field(…)’

sum_squared: float | None#: ‘Field(…)’

variance: float | None#: ‘Field(…)’

class nemo_evaluator.api.api_dataclasses.TaskResult(/, **data: Any)[source]#

Bases: pydantic.BaseModel

!!! abstract “Usage Documentation” Models

A base class for creating Pydantic models.

Attributes: class_vars: The names of the class variables defined on the model. private_attributes: Metadata about the private attributes of the model. signature: The synthesized __init__ [Signature][inspect.Signature] of the model.

__pydantic_complete__: Whether model building is completed, or if there are still undefined fields.
__pydantic_core_schema__: The core schema of the model.
__pydantic_custom_init__: Whether the model has a custom `__init__` function.
__pydantic_decorators__: Metadata containing the decorators defined on the model.
    This replaces `Model.__validators__` and `Model.__root_validators__` from Pydantic V1.
__pydantic_generic_metadata__: Metadata for generic models; contains data used for a similar purpose to
    __args__, __origin__, __parameters__ in typing-module generics. May eventually be replaced by these.
__pydantic_parent_namespace__: Parent namespace of the model, used for automatic rebuilding of models.
__pydantic_post_init__: The name of the post-init method for the model, if defined.
__pydantic_root_model__: Whether the model is a [`RootModel`][pydantic.root_model.RootModel].
__pydantic_serializer__: The `pydantic-core` `SchemaSerializer` used to dump instances of the model.
__pydantic_validator__: The `pydantic-core` `SchemaValidator` used to validate instances of the model.

__pydantic_fields__: A dictionary of field names and their corresponding [`FieldInfo`][pydantic.fields.FieldInfo] objects.
__pydantic_computed_fields__: A dictionary of computed field names and their corresponding [`ComputedFieldInfo`][pydantic.fields.ComputedFieldInfo] objects.

__pydantic_extra__: A dictionary containing extra values, if [`extra`][pydantic.config.ConfigDict.extra]
    is set to `'allow'`.
__pydantic_fields_set__: The names of fields explicitly set during instantiation.
__pydantic_private__: Values of private attributes set on the model instance.

Initialization

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

metrics: Dict[str, nemo_evaluator.api.api_dataclasses.MetricResult]#: ‘Field(…)’

nemo_evaluator.api.api_dataclasses#

Module Contents#

Classes#

API#

`nemo_evaluator.api.api_dataclasses`#