nemo_eval.utils.api

Module Contents

Classes

ApiEndpoint	Represents evaluation Standard API target.api_endpoint object
EvaluationTarget	Represents evaluation Standard API target object
ConfigParams	Represents evaluation Standard API config.params object
EvaluationConfig	Represents evaluation Standard API config object
AdapterConfig	Adapter is a mechanism for hooking into the chain of requests/responses btw benchmark and endpoint.

API

class nemo_eval.utils.api.ApiEndpoint(/, **data: typing.Any)

Bases: pydantic.BaseModel

Represents evaluation Standard API target.api_endpoint object

Initialization

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

url: str

‘Field(…)’

model_id: str

‘Field(…)’

type: str

‘Field(…)’

class nemo_eval.utils.api.EvaluationTarget(/, **data: typing.Any)

Bases: pydantic.BaseModel

Represents evaluation Standard API target object

Initialization

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

api_endpoint: nemo_eval.utils.api.ApiEndpoint

‘Field(…)’

class nemo_eval.utils.api.ConfigParams(/, **data: typing.Any)

Bases: pydantic.BaseModel

Represents evaluation Standard API config.params object

Initialization

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

top_p: float

‘Field(…)’

temperature: float

‘Field(…)’

limit_samples: Optional[Union[int, float]]

‘Field(…)’

max_new_tokens: Optional[int]

‘Field(…)’

max_retries: Optional[int]

‘Field(…)’

parallelism: Optional[int]

‘Field(…)’

task: Optional[str]

‘Field(…)’

request_timeout: Optional[int]

‘Field(…)’

extra: Optional[Dict[str, Any]]

‘Field(…)’

class nemo_eval.utils.api.EvaluationConfig(/, **data: typing.Any)

Bases: pydantic.BaseModel

Represents evaluation Standard API config object

Initialization

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

output_dir: str

‘Field(…)’

supported_endpoint_types: Optional[list[str]]

‘Field(…)’

type: str

‘Field(…)’

params: nemo_eval.utils.api.ConfigParams

‘Field(…)’

class nemo_eval.utils.api.AdapterConfig(/, **data: typing.Any)

Bases: pydantic.BaseModel

Adapter is a mechanism for hooking into the chain of requests/responses btw benchmark and endpoint.

Initialization

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

static get_validated_config(

run_config: dict[str, Any],

) → AdapterConfig | None

Factory. Shall return None if the adapter_config is not passed, or validate the schema.

Parameters:

run_config – is the main dict of a configuration run, see api_dataclasses.

api_url: str

‘Field(…)’

local_port: Optional[int]

‘Field(…)’

use_reasoning: bool

‘Field(…)’

end_reasoning_token: str

‘Field(…)’

custom_system_prompt: Optional[str]

‘Field(…)’

max_logged_responses: int | None

‘Field(…)’

max_logged_requests: int | None

‘Field(…)’

exception nemo_eval.utils.api.MisconfigurationError

Bases: Exception

Exception raised when evaluation is not correctly configured.

Initialization

Initialize self. See help(type(self)) for accurate signature.