aiq.profiler.inference_optimization.llm_metrics#

Classes#

A utility class for computing derived metrics on standardized LLM call logs.

class LLMMetrics#

A utility class for computing derived metrics on standardized LLM call logs.

static compute_profiling_metrics( all_steps: list[list[aiq.data_models.intermediate_step.IntermediateStep]], ) → pandas.DataFrame#

Compute and append the following columns to the provided DataFrame:

NOVA-Event-ID (str):
- The name of the calling function (function_name).
NOVA-Requests-Remaining-In-Event (int):
- For each row, how many future LLM_START events will occur (strictly after this row’s event_timestamp) in the same (example_number, function_name).
NOVA-Time-To-Next-Event (float):
- For each row, the number of milliseconds until the next LLM_START event in the same (example_number, function_name). If no future event, set to -1.
NOVA-Time-To-Event-End (float):
- For each row, the number of milliseconds until the last future LLM_START event in the same (example_number, function_name). If no future event, set to -1.
NOVA-Predicted-OSL (float or int):
- For rows where event_type == ‘LLM_START’, this column will hold the completion_tokens of the corresponding LLM_END (matched by UUID). If no match, set to NaN (or another sentinel).
NOVA-Time-To-Session-End (float):
- For each row, the total milliseconds remaining in the workflow invocation, i.e. until the max event_timestamp within that example_number.

Assumptions:

Columns required in the input df (at minimum):

['example_number', 'event_timestamp', 'event_type', 'function_name', 'UUID', 'completion_tokens']

‘LLM_START’ / ‘LLM_END’ events share the same UUID.
The DataFrame may have additional columns such as ‘llm_text_input’, ‘llm_text_output’,
‘function_id’, ‘parent_function_name’, ‘parent_function_id’, etc.