aiq.profiler.inference_optimization.token_uniqueness#

Functions#

compute_inter_query_token_uniqueness_by_llm(...)

Computes p90, p95, and p99 of 'new words added' between consecutive llm_start events,

Module Contents#

compute_inter_query_token_uniqueness_by_llm(
all_steps: list[list[aiq.data_models.intermediate_step.IntermediateStep]],
) → aiq.profiler.inference_optimization.data_models.LLMUniquenessMetricsByLLM#

Computes p90, p95, and p99 of ‘new words added’ between consecutive llm_start events, grouped by (llm_name, example_number).

Steps:

  1. Filter df to only llm_start events.

  2. Group first by (llm_name, example_number), then sort by event_timestamp in each group.

  3. Compare each llm_text_input to the previous one in the same group to find how many new words appear.

  4. Aggregate all ‘new words count’ across each llm_name, compute p90/p95/p99 for each LLM.

  5. Return a Pydantic RootModel containing a dictionary:

    { llm_name -> LLMUniquenessMetrics(p90, p95, p99) }.