nemo_evaluator.adapters.interceptors.response_stats_interceptor#

Response stats interceptor that collects aggregated statistics from API responses.

Module Contents#

Classes#

ResponseStatsInterceptor

Collects aggregated statistics from API responses for metrics collection.

API#

class nemo_evaluator.adapters.interceptors.response_stats_interceptor.ResponseStatsInterceptor(params: Params)#

Bases: nemo_evaluator.adapters.types.ResponseInterceptor, nemo_evaluator.adapters.types.PostEvalHook

Collects aggregated statistics from API responses for metrics collection.

Tracks the following statistics:

  • Token usage (prompt, completion, total) with averages and maximums

  • Response status codes and counts

  • Finish reasons and stop reasons

  • Tool calls and function calls counts

  • Response latency (average and maximum)

  • Total response count

  • Number of runs, inference times (approximated by processing time from the first to the last response)

Initialization

Initialize the response stats interceptor.

Args: params: Configuration parameters

class Params(/, **data: Any)#

Bases: nemo_evaluator.logging.BaseLoggingParams

Configuration parameters for response stats collection.

Initialization

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

cache_dir: str#

‘Field(…)’

collect_finish_reasons: bool#

‘Field(…)’

collect_token_stats: bool#

‘Field(…)’

collect_tool_calls: bool#

‘Field(…)’

logging_aggregated_stats_interval: int#

‘Field(…)’

save_individuals: bool#

‘Field(…)’

stats_file_saving_interval: int | None#

‘Field(…)’

intercept_response(
resp: nemo_evaluator.adapters.types.AdapterResponse,
context: nemo_evaluator.adapters.types.AdapterGlobalContext,
) → nemo_evaluator.adapters.types.AdapterResponse#

Collect aggregated statistics from the response.

post_eval_hook(
context: nemo_evaluator.adapters.types.AdapterGlobalContext,
) → None#

Write collected response statistics to eval_factory_metrics.json.