nat.eval.runtime_evaluator.register#

Classes#

`AverageLLMLatencyConfig`	Mean difference between connected LLM_START and LLM_END events (same UUID).
`AverageWorkflowRuntimeConfig`	Average workflow runtime per item (max timestamp - min timestamp).
`AverageNumberOfLLMCallsConfig`	Average number of LLM calls per item (count of LLM_END).
`AverageTokensPerLLMEndConfig`	Average total tokens per LLM_END event (prompt + completion if available).

`register_avg_llm_latency_evaluator`(config, builder)
`register_avg_workflow_runtime_evaluator`(config, builder)
`register_avg_num_llm_calls_evaluator`(config, builder)
`register_avg_tokens_per_llm_end_evaluator`(config, builder)

class AverageLLMLatencyConfig(/, **data: Any)#

Mean difference between connected LLM_START and LLM_END events (same UUID).

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

class AverageWorkflowRuntimeConfig(/, **data: Any)#

Average workflow runtime per item (max timestamp - min timestamp).

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

class AverageNumberOfLLMCallsConfig(/, **data: Any)#

Average number of LLM calls per item (count of LLM_END).

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

class AverageTokensPerLLMEndConfig(/, **data: Any)#

Average total tokens per LLM_END event (prompt + completion if available).

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

async register_avg_llm_latency_evaluator( config: AverageLLMLatencyConfig, builder: nat.builder.builder.EvalBuilder, )#

async register_avg_workflow_runtime_evaluator( config: AverageWorkflowRuntimeConfig, builder: nat.builder.builder.EvalBuilder, )#

async register_avg_num_llm_calls_evaluator( config: AverageNumberOfLLMCallsConfig, builder: nat.builder.builder.EvalBuilder, )#

async register_avg_tokens_per_llm_end_evaluator( config: AverageTokensPerLLMEndConfig, builder: nat.builder.builder.EvalBuilder, )#